Data Engineer

Key Responsibilities

• Billing Data Engineering: Design, build, test, deploy, and maintain reliable ETL/ELT pipelines and reusable data components in Azure and Databricks using Python, PySpark, and SQL.
• Data Hub & Curation: Transform source data into trusted, billing-ready curated datasets. Maintain consistent business rules, source-to-target traceability, and accurate delivery across data hub and curation layers.
• Data Quality & Reconciliation: Implement rigorous automated validations and reconciliations for completeness, accuracy, consistency, duplicates, referential integrity, and expected volumes. Treat data quality defects as production-impacting issues and resolve them with urgency.
• Orchestration & Scheduling: Develop and support ActiveBatch schedules, dependencies, execution chains, reruns, and recovery procedures. Ensure pipelines complete within narrow billing windows and downstream delivery commitments.
• Third-Party & Internal Integrations: Build and operate integrations with third-party providers and internal systems using APIs, databases, secure file transfers, and other appropriate interfaces. Design for retries, recoverability, idempotency, and clear failure handling.
• Production Reliability & Incident Response: Monitor pipeline execution and data delivery; investigate failures, data anomalies, and missed dependencies immediately. Apply safe in-window fixes or workarounds when necessary, communicate impact and status, and follow through with root-cause and permanent corrective actions.
• Performance & Timeliness: Optimize jobs and workflows to meet strict processing deadlines. Identify bottlenecks early, manage competing priorities, and escalate risks before they threaten billing timelines.
• Problem Solving & Alternatives: Break down ambiguous technical and data issues methodically, validate assumptions, and propose alternative implementation paths when constraints or dependencies prevent the preferred solution.
• Business & Technical Collaboration: Partner with Billing, Operations, Product, engineering, data, and other stakeholders to clarify requirements, understand business rules, communicate tradeoffs, and translate needs into dependable technical solutions.
• Security & Sensitive Data Handling: Follow secure engineering and data-handling practices for sensitive financial and billing data, including least-privilege access, appropriate controls, auditability, and protection of confidential information.
• Engineering Practices: Use Git-based workflows, code reviews, automated testing, deployment standards, documentation, and maintainable design patterns. Build solutions that can be supported under production time pressure without sacrificing quality.
• Continuous Improvement: Improve pipeline reliability, operational runbooks, test coverage, observability, recovery procedures, and team knowledge sharing based on production learnings and recurring failure patterns.

Knowledge, Skills, Abilities

• Strong hands-on experience with Databricks for production data engineering workloads.
• Strong Python and PySpark skills for data transformation, automation, validation, troubleshooting, and operational support.
• Strong SQL skills with the ability to analyze, transform, reconcile, and validate complex datasets.
• Practical experience designing, building, and supporting ETL/ELT pipelines in Microsoft Azure.
• Experience with job orchestration, scheduling, dependencies, and recovery patterns; ActiveBatch experience is preferred, or equivalent experience with enterprise batch/orchestration tools.
• Experience integrating data from third-party providers and internal systems through APIs, databases, and file-transfer patterns.
• Demonstrated ability to implement high-confidence data quality checks, source-to-target validation, reconciliation controls, and production-ready error handling.
• Strong understanding of curated data layers, data modeling, dependency management, schema evolution, and reliable downstream delivery.
• Ability to troubleshoot production data issues quickly, isolate root causes, make sound decisions under time pressure, and implement safe corrective actions.
• Strong time management and prioritization skills, including the ability to deliver within narrow, non-negotiable processing windows and manage multiple urgent dependencies.
• Excellent written and verbal communication skills with the ability to provide concise status, explain technical issues to non-technical partners, and raise risks early.
• Strong problem-solving mindset with a willingness to challenge assumptions and propose practical alternative solutions when needed.
• Experience with version control, collaborative Git workflows, automated testing, and CI/CD practices.
• Understanding of secure handling of sensitive or regulated financial data, access controls, privacy, and auditability.
• Exposure to Snowflake is a plus for the team's future-state evolution, but Snowflake experience is not required for the current environment.
• Experience in billing, payments, financial technology, wealth management, asset management, or another high-control financial domain is strongly preferred.

Qualifications & Experience

• 3-6 years of hands-on experience in data engineering, data integration, software engineering, or a closely related technical role, including production support responsibilities.
• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, Statistics, or a related technical field, or equivalent practical experience.
• Demonstrated experience supporting business-critical data pipelines with strict quality and delivery expectations.
• Experience working with sensitive or regulated financial data and following secure data-handling practices is required.
• Ability to work effectively in a hybrid, globally distributed engineering organization and collaborate across technical and business teams.