Data Engineer- Databricks

Overview

The Data Engineer builds and tests data pipelines in Databricks, covering ingestion, transformation, and Gold-layer modeling, against tasks assigned by the Lead Engineer at the start of each sprint and refined daily. This role suits an early-to-mid career engineer seeking rapid growth. The Data Engineer is expected to ask questions when requirements are unclear and to raise blockers promptly rather than delay progress.

Responsibilities

• Build and test Databricks pipelines (PySpark/SQL, Delta Lake) for ingestion, transformation, and modeling tasks assigned during sprint planning.
• Write and maintain data quality checks and tests for pipelines built.
• Participate in daily standups, reporting progress accurately, asking questions when requirements are unclear, and flagging blockers promptly.
• Follow the platform's data security model (RLS/CLS) and coding/documentation standards set by the Lead and Senior Engineers.
• Support integration testing and reconciliation checks for Boomi-based data flows (e.g., JDE-Concur, JDE-UKG) as assigned.
• Apply feedback from code reviews to subsequent work.
• Document pipelines clearly enough for another engineer to assume ownership.

Qualifications

Required Qualifications

• 2–4 years of data engineering experience, including hands-on work with Spark/PySpark and SQL.
• Hands-on Databricks experience (Delta Lake, notebooks, jobs/workflows).
• Working understanding of data warehousing/lakehouse concepts (Bronze/Silver/Gold or comparable layered modeling).
• Experience working in an agile team with daily standups and sprint-based delivery.
• Strong written and verbal English communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate.
• Exposure to financial or ERP source data — JD Edwards (JDE), EcoSys, or Microsoft Dynamics — or general ledger/cost data.
• Exposure to an EPC, construction, energy, or other asset-heavy industry.
• Familiarity with Unity Catalog-based security (RLS/CLS) or Power BI/Genie reporting on governed data.
• Exposure to Boomi or another iPaaS/integration middleware platform.

Professional Attributes:

• Asks questions early when a task is unclear rather than proceeding on assumptions.
• Demonstrates visible progress daily rather than only at sprint boundaries.
• Applies feedback from the Lead and Senior Engineers quickly and consistently.
• Seeks increasing ownership and more complex problems over time.