Aws Data Engineer

Responsibilities:

• Design and implement the data modeling, data ingestion and data processing for various datasets
• Design, develop and maintain ETL Framework for various new data source
• Develop data ingestion using AWS Glue/ EMR, data pipeline using PySpark, Python and Databricks.
• Build orchestration workflow using Airflow & databricks Job workflow
• Develop and execute adhoc data ingestion to support business analytics.
• Proactively interact with vendors for any questions and report the status accordingly
• Explore and evaluate the tools/service to support business requirement
• Ability to learn to create a data-driven culture and impactful data strategies.
• Aptitude towards learning new technologies and solving complex problem.

Qualifications:

• Minimum of bachelor s degree. Preferably in Computer Science, Information system, Information technology.
• Minimum 5 years of experience on cloud platforms such as AWS, Azure, GCP.
• Minimum 5 year of experience in Amazon Web Services like VPC, S3, EC2, Redshift, RDS, EMR, Athena, IAM, Glue, DMS, Data pipeline & API, Lambda, etc.
• Minimum of 5 years of experience in ETL and data engineering using Python, AWS Glue, AWS EMR /PySpark and Airflow for orchestration.
• Minimum 2 years of experience in Databricks including unity catalog, data engineering Job workflow orchestration and dashboard generation based on business requirements
• Minimum 5 years of experience in SQL, Python, and source control such as Bitbucket, CICD for code deployment.
• Experience in PostgreSQL, SQL Server, MySQL & Oracle databases.
• Experience in MPP such as AWS Redshift, AWS EMR, Databricks SQL warehouse & compute cluster.
• Experience in distributed programming with Python, Unix Scripting, MPP, RDBMS databases for data integration
• Experience building distributed high-performance systems using Spark/PySpark, AWS Glue and developing applications for loading/streaming data into Databricks SQL warehouse & Redshift.
• Experience in Agile methodology
• Proven skills to write technical specifications for data extraction and good quality code.
• Experience with big data processing techniques using Sqoop, Spark, hive is additional plus
• Experience in data visualization tools including PowerBI, Tableau.
• Nice to have experience in UI using Python Flask framework and /or React/Angular JS.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.