Data Architect

Senior Data Engineer – AI, Automation & Databricks

Location: Houston, TX

Work Arrangement: Houston-based / Onsite as required

Experience: 8+ years preferred

Employment Type: Contract / Full-Time, based on project requirement

Job Summary

We are looking for a highly experienced and innovative Senior Data Engineer to assess, modernize, and optimize enterprise Data Engineering processes while driving the adoption of automation, Artificial Intelligence, and Agentic AI.

The ideal candidate will combine strong hands-on Data Engineering expertise with a proactive problem-solving mindset. This individual will evaluate existing engineering processes, identify opportunities for AI and automation, develop Proofs of Concept (POCs), and help transition successful ideas into production-ready solutions.

The role requires deep experience with Databricks, AWS Redshift, modern Data Engineering architectures, AI-enabled engineering practices, and AI solutions across the Software Development Life Cycle (SDLC). The successful candidate should be comfortable working independently, challenging existing approaches, proposing innovative solutions, and collaborating with both technical and business stakeholders.

Key Responsibilities

• Assess existing Data Engineering platforms, workflows, pipelines, architecture, and operational processes and identify opportunities for optimization.
• Design and implement improvements to enhance data pipeline performance, scalability, reliability, maintainability, and operational efficiency.
• Identify opportunities to introduce automation and AI capabilities across Data Engineering processes.
• Develop Proofs of Concept (POCs) for AI- and automation-driven ideas and evaluate their technical and business feasibility.
• Drive successful POCs through implementation and production adoption in collaboration with engineering, architecture, security, and business teams.
• Design and implement solutions leveraging Agentic AI, Generative AI, and AI-assisted Data Engineering.
• Identify opportunities to apply AI throughout the Software Development Life Cycle, including:
• Requirements analysis
• Solution design
• Code generation and optimization
• Data pipeline development
• Automated testing
• Data quality validation
• Documentation
• Monitoring and observability
• Incident analysis and troubleshooting
• Build and optimize scalable Data Engineering solutions using Databricks.
• Design, develop, and optimize enterprise data workloads using AWS Redshift.
• Develop and optimize ETL/ELT pipelines supporting large-scale structured and unstructured data environments.
• Evaluate existing engineering practices and proactively recommend improvements rather than relying solely on predefined requirements.
• Identify repetitive or manual engineering activities that can be automated through scripts, workflows, AI agents, or orchestration platforms.
• Partner with Data Engineers, Data Architects, Data Scientists, AI/ML teams, application teams, and business stakeholders.
• Translate complex technical opportunities into clear business value and build stakeholder alignment around proposed solutions.
• Establish reusable engineering patterns, frameworks, accelerators, and best practices.
• Conduct technical evaluations of emerging AI and Data Engineering technologies and recommend appropriate adoption strategies.
• Support troubleshooting and root-cause analysis of complex Data Engineering and platform-related issues.
• Document technical designs, architecture decisions, POCs, implementation approaches, and operational procedures.

Required Technical Skills

• Strong hands-on experience in Data Engineering within large-scale enterprise environments.
• Advanced experience with Databricks.
• Strong experience with AWS Redshift.
• Strong understanding of:
• Data lakes and lakehouse architectures
• Data warehouses
• ETL/ELT architecture
• Distributed data processing
• Data pipeline orchestration
• Data quality and governance
• Performance optimization
• Hands-on experience with Apache Spark / PySpark.
• Strong programming experience with Python.
• Strong SQL development and query optimization skills.
• Experience working with AWS Data and Analytics services.
• Experience designing and supporting scalable cloud-based data platforms.
• Understanding of CI/CD and DevOps principles within Data Engineering environments.

AI & Automation Experience

Candidates should have practical experience applying AI beyond traditional analytics or machine learning use cases.

Preferred experience includes:

• Agentic AI / AI Agents
• Generative AI applications within Data Engineering
• LLM-based engineering solutions
• AI-assisted code generation and review
• Automated pipeline troubleshooting
• Intelligent Data Quality validation
• AI-assisted data transformation and mapping
• AI-driven monitoring and anomaly detection
• Automated documentation and metadata generation
• AI-enabled SDLC processes
• Prompt engineering and LLM integration
• Development of AI/automation POCs
• Integration of AI solutions into production engineering workflows

Experience with AI development frameworks such as LangChain, LangGraph, Semantic Kernel, Amazon Bedrock, OpenAI/Azure OpenAI, or similar technologies would be advantageous.

Required Professional Skills

• Strong analytical and problem-solving capabilities.
• Demonstrated ability to independently identify technical and process improvement opportunities.
• Strong innovation mindset with a track record of proposing and implementing new ideas.
• Ability to operate effectively in ambiguous environments without requiring detailed direction.
• Strong communication and stakeholder-management skills.
• Ability to collaborate across technical teams, business stakeholders, leadership, and architecture groups.
• Ability to explain complex technical concepts to non-technical stakeholders.
• Strong ownership mentality with the ability to take ideas from concept → POC → validation → implementation.
• Ability to work effectively as part of a collaborative engineering team.

Preferred Qualifications

• 8+ years of overall Data Engineering, Data Platform, or related engineering experience.
• Significant hands-on experience with enterprise Data Engineering architectures.
• Experience leading Data Engineering modernization or optimization initiatives.
• Demonstrated experience building AI- or automation-enabled engineering solutions.
• Experience working within large, complex enterprise environments.
• Experience influencing architecture or engineering standards across multiple teams.
• Exposure to airline, transportation, travel, or other large-scale transactional environments would be advantageous.

Ideal Candidate Profile

The ideal candidate is not simply someone who can build pipelines from predefined requirements. We are looking for an engineer who can evaluate the existing environment, identify what can be improved, and proactively develop solutions.

This person should be able to:

• Understand the current Data Engineering landscape quickly.
• Identify inefficiencies and improvement opportunities.
• Determine where AI or automation can create measurable value.
• Build working POCs rather than limiting recommendations to conceptual presentations.
• Demonstrate the value of proposed solutions to technical and business stakeholders.
• Convert successful concepts into scalable production implementations.
• Challenge existing engineering practices constructively.
• Work collaboratively and fit effectively within a multidisciplinary team.

Core Skills

Databricks | AWS Redshift | Data Engineering | Python | PySpark | SQL | AWS | ETL/ELT | Data Lakehouse | Agentic AI | Generative AI | LLMs | AI Automation | AI across SDLC | POC Development | Data Platform Optimization | Stakeholder Management | Innovation

Pay: $90,000.00 - $100,000.00 per year

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Flexible schedule
• Paid time off
• Referral program

Experience:

• Databricks: 5 years (Preferred)
• Apache Spark: 5 years (Preferred)
• data engineer: 5 years (Preferred)
• Pyspark: 5 years (Preferred)

Work Location: In person