Data Engineer

Role: Data Engineer

Location: Multiple Locations in USA

Visa Status: Any Visa holder except OPT's

Primary Responsibilities:

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
• Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
• Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
• Ensure data quality, governance, and security across the full data lifecycle
• Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
• Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
• Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
• Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI

Required Qualifications:

• Bachelor's degree in Computer Science, Engineering, or IT related field
• 6+ years of experience in Data Engineering with Python/PySpark
• 6+ years of experience in building ETL/ELT pipelines using Databricks
• 6+ years of experience working in Agile environments
• 5+ years of strong experience in SQL / PL-SQL
• 4+ years of experience with Azure Databricks and Delta Lake architecture
• 4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps)
• 3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)
• 2+ years of experience with Databricks Delta Live Tables (DLT)
• 2+ years of experience with unit testing, validation, and pipeline testing frameworks

Preferred Qualifications:

• Familiarity with medallion architecture and SCD2 implementations
• AI builder: Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
• Experience building enterprise-scale data platforms
• Strong skills in performance tuning and debugging large-scale pipelines
• Experience with real-time/streaming frameworks (Structured Streaming)
• Ability to work in distributed, cross-functional global teams
• Exposure to GenAI tools (e.g., GitHub Copilot) for engineering productivity
• Strong understanding of secure coding practices and vulnerability remediation
• Proven ability to analyze logs, troubleshoot production issues, and optimize performance
• Demonstrated capability to design and deploy AI-powered solutions responsibly.