Senior Databricks Engineer

Role description

Senior Databricks Engineer

Associate Data Architect I

Who We Are:

Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world.

UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live.

With us, you’ll create a boundless impact that transforms your career—and the lives of people across the world.

Visit us at UST.com.

You Are:

UST is searching for a Senior Databricks Engineer with deep hands-on experience designing, developing, securing, and deploying complex enterprise data pipelines on AWS Databricks within the US healthcare industry. The engineer will build scalable batch and streaming data pipelines supporting clinical, claims, analytics, and machine learning use cases.

The opportunity:

• Design, develop, test, deploy, and support complex batch and streaming data pipelines using AWS Databricks

• Design scalable lakehouse solutions using Delta Lake and Apache Iceberg table formats

• Implement enterprise data governance using Databricks Unity Catalog, including catalogs, schemas, tables, views, volumes, external locations, and governed data access

• Design and implement complex Databricks RBAC and access-control models across users, groups, service principals, workspaces, catalogs, schemas, tables, views, external locations, and other governed resources

• Implement secure data-access patterns following least privilege, separation of duties, and role-based access control principles

• Design and implement appropriate data tokenization, masking, encryption, and de-identification techniques for sensitive healthcare data, including PHI/PII

• Ensure encryption is appropriately implemented for data at rest and data in transit, including integration with enterprise key-management and security controls where applicable

• Develop high-volume data ingestion and transformation pipelines using PySpark, Spark SQL and Databricks-native capabilities

• Develop and optimize both batch and real-time/streaming pipelines

• Work with SMEs to understand healthcare data structures, business rules, data lineage, quality requirements, and transformation logic

• Partner with Machine Learning Engineers to create secure, governed, high-quality datasets for ML/AI workloads

• Collaborate with AWS DevOps engineers on CI/CD, infrastructure automation, deployment, monitoring, logging, and production operations

This position description identifies the responsibilities and tasks typically associated with the performance of the position. Other relevant essential functions may be required.

What you need:

• 5+ years of professional data engineering experience, with substantial recent hands-on experience developing production solutions on Databricks

• Strong hands-on expertise with Apache Spark / PySpark and advanced SQL

• Strong experience designing and implementing enterprise-scale batch and streaming data pipelines

• Deep understanding of distributed data processing, partitioning, schema evolution, fault tolerance, performance optimization, and scalability

• Strong experience with Delta Lake and lakehouse architecture

• Hands-on experience with Apache Iceberg table format, including table design, partitioning, schema evolution, interoperability, and production implementation

• Experience working with multiple enterprise source systems and downstream analytical, reporting, and ML platforms

• Strong hands-on experience with Databricks Unity Catalog in an enterprise environment

• Strong understanding of Databricks privilege hierarchies and permission inheritance

• Experience implementing least-privilege access, segregation of duties, and environment-specific access controls across development, test/UAT, and production

• Experience designing secure access patterns for both human users and machine/service identities

• Ability to troubleshoot complex authorization and access-control issues in Databricks

• Hands-on experience implementing data tokenization, masking, anonymization, or de-identification techniques for sensitive enterprise data

• Strong understanding of protecting PHI, PII, and other sensitive healthcare information

• Experience implementing encryption for data exchanged between platforms and services

• Understanding of enterprise key-management concepts and secure key-handling practices

• Experience designing pipelines that maintain appropriate security controls throughout ingestion, processing, storage, and consumption

• Ability to balance data usability with privacy, security, and regulatory requirements

• AWS

• Strong experience working in AWS-based enterprise data environments

• Hands-on experience integrating Databricks with Amazon S3 and other AWS services

• Understanding of AWS IAM, security controls, networking, monitoring, logging, and enterprise cloud architecture

• Experience working with AWS DevOps teams and automated deployment processes

• Understanding of secure connectivity between Databricks and enterprise data sources/consumers

• US Healthcare

• Minimum 3 years of recent professional experience working in US healthcare

• Hands-on experience working with healthcare data in an enterprise environment

• Experience with clinical and/or claims data, with preference for candidates who have worked extensively with both

• Understanding of healthcare data terminology, business processes, data quality challenges, and healthcare-specific business rules

• Experience working with data containing PHI/PII

• Practical understanding of HIPAA and healthcare data security/privacy requirements

• Experience collaborating with Clinical SMEs, Claims SMEs, healthcare business stakeholders, and technology teams

• Ability to translate healthcare business requirements into scalable and secure data engineering solutions

Compensation can differ depending on factors including but not limited to the specific office location, role, skill set, education, and level of experience. UST provides a reasonable range of compensation for roles that may be hired in various U.S. markets as set forth below.

Role Location: Illinois

Compensation Range: $106,000-$159,000

Benefits

Full-time, regular employees accrue a minimum of 10 days of paid vacation per year, receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year), 10 paid holidays, and are eligible for paid bereavement leave and jury duty. They are eligible to participate in the Company’s 401(k) Retirement Plan with employer matching. They and their dependents residing in the US are eligible for medical, dental, and vision insurance, as well as the following Company-paid Employee Only benefits: basic life insurance, accidental death and disability insurance, and short- and long-term disability benefits. Regular employees may purchase additional voluntary short-term disability benefits, and participate in a Health Savings Account (HSA) as well as a Flexible Spending Account (FSA) for healthcare, dependent child care, and/or commuting expenses as allowable under IRS guidelines. Benefits offerings vary in Puerto Rico.

Part-time employees receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year) and are eligible to participate in the Company’s 401(k) Retirement Plan with employer matching.

Full-time temporary employees receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year) and are eligible to participate in the Company’s 401(k) program with employer matching. They and their dependents residing in the US are eligible for medical, dental, and vision insurance.

Part-time temporary employees receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year).

All US employees who work in a state or locality with more generous paid sick leave benefits than specified here will receive the benefit of those sick leave laws.

What we believe:

We proudly embrace the values that have shaped UST since day one. We build our culture of Humility, Humanity, and Integrity. These values inspire us to nurture a people-first, human centric culture that fosters diversity, prioritizes sustainable solutions, and keeps our people and clients at the forefront of all decisions.

Humility:

We will listen, learn, be empathetic and help selflessly in our interactions with everyone.

Humanity:

Through business, we will better the lives of those less fortunate than ourselves.

Integrity:

We honor our commitments and act with responsibility in all our relationships.

Equal Employment Opportunity Statement

UST is an Equal Opportunity Employer.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other applicable characteristics protected by law. We will consider qualified applicants with arrest or conviction records in accordance with state and local laws and “fair chance” ordinances.

UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance.

#UST

#CB

#LI-DU

Skills

Databricks, Data Governance, Streaming, PySpark, Apache Iceberg, AWS, Data Protection, HIPAA, Amazon S3, Delta Lake, CI/CD

Benefits
Compensation range: $106,000.00 to $159,000.00 per year
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.