Data / AI Native Engineer

Job Title: Data / AI Native Engineer

Location : St. Louis, MO / Arlington, VA / NYC, NY / Jersey City, NJ/ Atlanta, GA (Hybrid Onsite Role- 3 Days Onsite Per Week)

Duration : 12+ Months of Contract

Teams Meeting Interview

Job Description:

Day-to-Day Job Duties

• Design, build, optimize, and support enterprise-scale Databricks and AWS Lakehouse data pipelines using Spark and PySpark.
• Develop Databricks solutions leveraging Delta Lake, Medallion Architecture (Bronze/Silver/Gold), and Unity Catalog.
• Develop high-volume batch and streaming pipelines using PySpark, ensuring scalability, performance, reliability, data quality, and maintainability.
• Design and implement modern lakehouse engineering patterns including SCD Type 2, Change Data Capture (CDC), idempotent writes, schema evolution, and data contracts.
• Work across the AWS data ecosystem, including technologies such as S3, Glue, Lambda, EMR, Redshift, Kinesis, and related AWS services.
• Use GitHub Copilot and/or Claude Code CLI as part of the daily engineering workflow to accelerate application code, data-pipeline development, testing, debugging, optimization, and documentation.
• Rapidly develop proofs of concept and MVPs using AI-assisted development, while establishing clear paths to scalable production implementation.
• Apply AI capabilities directly within the data-engineering lifecycle, including use cases such as:
• AI-assisted schema inference
• Automated data-quality rule generation
• LLM-based anomaly detection
• AI-generated PySpark transformations
• Intelligent data validation and enrichment
• Build data foundations for AI-enabled products, including feature stores, vector stores, RAG data layers, embedding pipelines, and other AI-ready data architectures.
• Design and develop high-volume streaming pipelines using technologies such as Spark Structured Streaming, Kafka, Kinesis, and Databricks Auto Loader.
• Develop and orchestrate enterprise data pipelines using technologies such as Databricks Workflows, Delta Live Tables, Airflow, and dbt.
• Implement secure and governed data-engineering practices using Unity Catalog, including catalog/schema organization, access control, lineage, governance, and data discovery.
• Lead technically complex data-engineering initiatives, drive solution design, coordinate engineering activities, and help resolve complex platform and pipeline issues.
• Partner with architects, software engineers, product owners, source-system teams, governance teams, analytics teams, AI teams, and business stakeholders.
• Remain actively hands-on with PySpark, Databricks, AWS, and software-engineering activities, rather than operating solely as a people manager or high-level data architect.

Basic Qualifications

• Minimum 8+ years of overall Data Engineering / Software Engineering experience delivering enterprise-scale data solutions.
• Minimum 5+ years of strong hands-on Data Engineering experience, including design, development, optimization, and support of production data pipelines.
• Minimum 3+ years of hands-on Databricks experience, including production experience with:
• Delta Lake
• Medallion Architecture – Bronze / Silver / Gold
• Unity Catalog
• Minimum 5+ years of hands-on Spark experience, with strong expertise in PySpark specifically, rather than Scala-only Spark development.
• Minimum 3+ years of AWS data-platform experience, using services such as S3, Glue, Lambda, EMR, Redshift, Kinesis, or related AWS technologies.
• Minimum 2+ years of experience leading complex data-engineering initiatives, technical workstreams, or engineering teams while remaining hands-on technically.
• Proven experience combining data engineering and software-engineering practices, including modular development, testing, source control, CI/CD, observability, and production support.
• Demonstrated daily hands-on use of GitHub Copilot and/or Claude Code CLI, with the ability to describe concrete examples of how these tools are used to accelerate engineering work.
• Demonstrated ability to build POCs and MVPs rapidly using AI-assisted development.
• Strong understanding of distributed data-processing principles, large-scale pipeline design, performance optimization, data quality, and production operations.
• Strong SQL skills and experience working with large enterprise datasets.

Critical Screening Requirements

Candidates should be screened out if they do not demonstrate:

• Proven experience spanning Data Engineering and Software Engineering.
• Proven experience leading complex data-engineering initiatives or teams.
• Hands-on Databricks experience including Delta Lake, Bronze/Silver/Gold Medallion Architecture, and Unity Catalog.
• High-volume Spark and PySpark experience; Scala-only Spark experience is not sufficient.
• Strong experience with the AWS data ecosystem.
• Daily hands-on use of GitHub Copilot and/or Claude Code CLI with specific examples.
• Ability to use AI-assisted development to rapidly create POCs and MVPs.

Travel

Travel as required based on project and client needs to Atlanta, GA

Degree

Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical discipline, or equivalent professional work experience.

Nice to Have (But Not a Must)

• 2+ years of experience applying AI/Generative AI within data-engineering solutions, particularly:
• AI-assisted schema inference
• Automated data-quality rules
• LLM-based anomaly detection
• AI-generated PySpark transformations
• Experience building AI-enabled data products, including feature stores, vector stores, RAG data layers, and embedding pipelines.
• 2+ years of high-volume streaming experience using Spark Structured Streaming, Kafka, Kinesis, Auto Loader, or equivalent technologies.
• Strong implementation experience with modern lakehouse patterns including:
• SCD Type 2
• CDC
• Idempotent writes
• Schema evolution
• Data contracts
• Hands-on experience with dbt, Airflow, Databricks Workflows, and/or Delta Live Tables.
• Experience designing data platforms and pipelines supporting enterprise-scale GenAI, RAG, machine-learning, or agentic-AI solutions.

Profiles That Typically Align Well

Senior Data Engineer · Lead Data Engineer · Principal Data Engineer · Data Platform Engineer · Senior Analytics Engineer · Data Engineering Manager with active individual-contributor responsibilities

Profiles That Typically Do Not Align

Traditional ETL Developer focused primarily on Informatica / Ab Initio / DataStage without cloud-native engineering · BI / Report Developer · Diagram-only Data Architect · DBA · Hadoop Administrator