Senior Data Engineer (SQL, Databricks, CDC)

Senior Data Engineer

Responsibilities:

• Analyze the current Provider subject area architecture, data flows, and processing patterns.
• Assess the feasibility of converting the Provider subject area from full-refresh processing to incremental processing.
• Identify source-system capabilities and determine availability of change data capture (CDC), timestamps, or other incremental processing mechanisms.
• Evaluate impacts to downstream applications, reporting, and analytics consumers.
• Develop recommended incremental processing design patterns and implementation approaches.
• Create reconciliation and validation strategies to compare incremental and full-refresh outputs.
• Collaborate with data architecture and data modeling teams to align future-state designs with enterprise standards.
• Identify technical debt, performance bottlenecks, and opportunities for optimization within the Provider data pipelines.
• Produce technical documentation, architecture diagrams, findings, and implementation recommendations.
• Participate in Agile ceremonies, working sessions, backlog refinement, and technical design reviews.

Required Skills:

• Advanced SQL development and data analysis skills.
• Experience designing and implementing incremental processing and change data capture (CDC) solutions.
• Experience performing data lineage analysis and documenting source-to-target mappings.
• Strong understanding of ETL pipeline development and optimization.
• Experience working with large-scale healthcare datasets.
• Familiarity with provider, claims, enrollment, or healthcare master data domains.
• Experience with cloud data platforms such as Databricks, Spark, or equivalent distributed processing technologies.
• Strong troubleshooting, data profiling, and data quality analysis skills.
• Experience creating technical documentation, architecture diagrams, and engineering recommendations.
• Experience supporting healthcare analytics and reporting platforms.
• Experience working within Agile teams and utilizing Jira for delivery management.