Data engineer

Job Summary

We are looking for a Python Data Engineer to build a data-processing platform that handles Excel/CSV files, data mapping, cleansing, validation, transformation and standardized output generation.

Responsibilities

• 4+ years of Python/Data Engineering experience.
• Strong Python, Pandas, SQL and ETL experience.
• Hands-on Excel/CSV data processing.
• Strong data cleansing, transformation and validation experience.
• FastAPI/Flask and REST API experience.
• PostgreSQL/MySQL or similar database experience.
• Experience handling large datasets and data-quality issues.
• Git and good software engineering practices.
• Good to Have Polars Airflow Celery/Redis AWS S3 Docker PySpark/Spark Schema validation/mapping Address/data standardization or geospatial data.
• Someone who has built real-world data processing/ETL systems, rather than only traditional Django/CRUD applications.
• Build Python-based ETL/data processing pipelines.
• Process and transform Excel/CSV datasets using Pandas.
• Implement source-column standard-template mapping.
• Develop data cleansing, normalization and validation rules.
• Handle missing, duplicate, invalid and inconsistent data.
• Implement business rules and data-quality checks.
• Build FastAPI REST APIs for file upload, processing and output.
• Implement background/asynchronous processing for larger files.
• Maintain processing status, history and error handling.
• Work with PostgreSQL and cloud storage such as AWS S3.
• Write clean, modular and testable Python code.

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.