Data Engineer

Key Responsibilities

• Design and implement ETL/ELT pipelines using Spark SQL and Python within Databricks Medallion architecture
• Develop dimensional data models following star schema methodology with proper fact and dimension table design, SCD implementation, and optimization for analytical workloads
• Optimize Spark SQL and DataFrame operations through appropriate partitioning strategies, clustering and join optimizations to maximize performance and minimize costs
• Build comprehensive data quality frameworks with automated validation checks, statistical profiling, exception handling, and data reconciliation processes
• Establish CI/CD pipelines incorporating version control, automated testing including but not limited to unit test, integration test, smoke test, etc.
• Implement data governance standards including row-level and column-level security policies for access controls and compliance requirements
• Create and maintain technical documentation including ERDs, schema specifications, data lineage diagrams, and metadata repositories

Key Requirements

• Min. 4 years’ fulltime experience with SparkSQL, Python and PySpark for data engineering workflow
• Strong proficiency in dimensional modeling and star schema design for analytical workloads
• Experience implementing automated testing and CI/CD pipelines for data workflows
• Familiarity with GitHub operations and collaborative development practices
• Demonstrated ability to optimize engineering workflow jobs for performance and cost efficiency
• Experience with cloud data services and infrastructure (AWS, Azure, or GCP)
• Proficiency with IDE tools such as Visual Studio Code for efficient development
• Experience with Databricks platform will be a plus