Senior AI & Data Engineer (Azure Databricks, Microsoft Fabric & Microsoft Foundry)

Pay $145,000 to $185,000 per year. Full-time, remote (US).

About ArchitectNow

For 18 years, ArchitectNow has helped businesses ship software that does the work. We are a Microsoft Solutions Partner with active designations in Digital & App Innovation (Azure), Data & AI (Azure), and Infrastructure (Azure). Across cloud-native applications, data platforms, AI, the Power Platform, and the managed cloud services that keep production running, our focus has always been getting hard things built, well, and on time.

What has changed is how we build them. We are in the middle of a fast, deliberate move into an AI-first practice. Our engineers use Claude, GitHub Copilot, and a growing stack of internal AI tooling to put their experience to work on harder problems and ship faster than the industry standard.

We are fully remote across 14 states. We are small enough that your work is visible to leadership the same week you do it, and there are no layers between an engineer and the client by design.

The role

Our clients have stopped asking whether they should use AI. They are asking why their numbers don't match, why their data isn't ready for Copilot, and how to get an agent into production without putting protected data at risk. Those are data problems first and AI problems second. This role exists to solve both.

The Senior AI & Data Engineer is a new role at ArchitectNow. You will be the technical lead on our Azure data and AI engagements, running discovery and solution design, then building the platform alongside one of our agile delivery teams. Depending on the client, that platform may be Microsoft Fabric, end-to-end Azure Databricks, or Azure Data Factory with Azure SQL. Our current work includes consolidating a health system's departmental databases into a governed Microsoft Fabric warehouse with a certified Direct Lake model and a data agent for leadership, and standing up Microsoft Fabric and Microsoft Foundry across four environments for a national insurer, including RAG over millions of claims documents and a production intake automation workload.

These are joint builds. Client data teams work alongside us, and every engagement ends with them able to run and extend what we built. Teaching is part of the job.

What you will actually do

You will lead the technical side of data and AI engagements from discovery through production handover. You will sit on a delivery team with a Delivery Lead and developers, doing time-and-materials project work across one or two clients at a time.

A typical week. Monday you are in a discovery session with a client's analytics director, mapping where their metric definitions diverge and which tables need change data capture before replication will work. Tuesday you are in a notebook moving business logic upstream into the Gold layer so the semantic model stays simple and fast. Wednesday you are tuning a data agent's grounding instructions against a set of known business questions and writing down where it still gets things wrong. Thursday you are reviewing a teammate's pipeline PR and walking the client's new data engineer through your medallion conventions. Friday you are helping our sales team scope the next engagement.

Specifically, you will

• Lead discovery and solution design for Azure data and AI engagements, and turn findings into architecture, phased plans, and estimates our team can deliver against
• Lead technical execution on AI and data engagements, guiding the delivery team through planning, build, and handover to successful delivery
• Design and build lakehouse and warehouse platforms on Microsoft Fabric, Azure Databricks, or Azure Data Factory with Azure SQL, including ingestion, medallion architecture, pipelines, notebooks, and certified semantic models
• Build production AI solutions on Microsoft Foundry, Azure AI Search, and a range of foundation models, including RAG, document processing, evaluation, content safety, and human-in-the-loop review
• Ground Microsoft Copilot, Copilot Studio agents, and data agents on governed enterprise data
• Implement security and governance that holds up in regulated environments, including Entra groups, row- and object-level security, Purview, private networking, and PHI controls
• Set up CI/CD and environment promotion for data and AI workloads
• Reconcile, validate, and cut over from legacy systems
• Write the documentation, runbooks, and reusable patterns that let clients and our own team repeat what works
• Mentor and upskill our engineers on the Azure data platform and Azure AI, and help build our internal data and AI practice through reference implementations, delivery accelerators, and keeping the team current as the stack moves

Who we are looking for

To be clear on scope first, this is a hands-on engineering role with architect-level responsibility. You will design the solution and you will also write a meaningful share of the code. If you have moved fully into diagrams and slides, this is not the right fit. If you have only ever executed someone else's design, it probably isn't either.

We know the data and AI stack is moving faster than anyone's resume. Microsoft Fabric is only a few years old and some of the newest capabilities are still in preview. We care less about how many years you have with a specific product and more about whether you have shipped real data and AI systems to production, and whether you stay current because you are genuinely curious.

And one trait matters as much as any of the above. We are looking for someone who communicates clearly, takes real ownership, and follows through on what they commit to. Our clients stay with us through hard projects because the people on our teams are people they want to work with. This one is non-negotiable.

Required

• 7+ years of hands-on data engineering experience, including leading the design of data platforms for clients or internal business stakeholders
• Proven ability to design AI and data architectures, from platform and integration patterns through security, governance, and the path to production
• Experience leading technical execution on AI and data projects, guiding delivery teams through planning, build, and handover to successful delivery
• Strong hands-on experience with both Microsoft Fabric and Azure Databricks (required), and comfort building with Azure Data Factory and Azure SQL when that is the right fit. Lakehouses, warehouses, pipelines, notebooks, OneLake, Delta Lake, Unity Catalog, and semantic models
• Strong SQL and Python for data transformation and AI integration, with working Spark / PySpark
• Hands-on experience building AI solutions that reached production or real users, on Microsoft Foundry (formerly Azure AI Foundry), Azure AI Search, Copilot Studio, or comparable, using a range of foundation models such as OpenAI and Anthropic. You can walk us through how you chose a model, how you evaluated quality, and what you did when it was wrong
• Experience leading client-facing discovery, requirements, and solution design conversations, including explaining trade-offs to non-technical stakeholders
• Active, daily use of AI tools in your own engineering work, such as Claude, GitHub Copilot, or comparable
• Strong soft skills for customer communication. You build trust with client stakeholders, run productive working sessions, and keep them informed when plans change. Excellent written and verbal communication, with ownership and follow-through as a defining trait

Data engineering experience

We will explore these areas in the design case and live build. Strong candidates are deep in most and conversant in all.

• Data modeling. Relational and dimensional modeling, with a working command of Inmon, Kimball, and Data Vault approaches and when each fits
• Lakehouse design. Medallion (bronze, silver, gold) data lake designs and the data curation practices that move data between layers
• Data engines. A clear grasp of transactional vs. analytic engines, and hands-on querying across SQL, Spark, Kusto (KQL), and graph engines
• Azure data services. Production experience with Azure SQL, Azure Cosmos DB, and Azure Data Factory
• Ingestion frameworks. Framework-driven, metadata-based data ingestion and data validation, not one-off pipelines
• Pipeline patterns. Building and deploying delta (incremental) load, streaming, and IoT pipelines, and managing data drift and schema drift as sources change
• Quality and testing. Data quality management, reference data management, and automated testing and validation built into the pipeline
• Observability. Data observability with anomaly detection and alerting, so issues surface before users find them
• Discovery and security. Data discovery and cataloging, plus data security management using security zoning
• AI-ready data. Semantic modeling and AI-ready dataset generation for analytics, Microsoft Copilot, and agents
• AI-accelerated builds and migrations. Using AI tooling to accelerate data engineering builds and platform migrations, from code generation and code conversion to tests and documentation

AI engineering experience

• Unstructured knowledge. Preparing documents and other unstructured content for agents, including parsing, chunking, enrichment, indexing, and retrieval tuning
• Structured knowledge. Preparing structured data for agents through semantic models, curated tables, and ontologies that agents can query reliably
• Prototype to enterprise. Elevating AI prototype agents into enterprise agents with security, scale, monitoring, and lifecycle management
• Foundation models. Working across foundation models from OpenAI, Anthropic, and others, choosing by quality, cost, latency, and risk, and designing solutions that can switch models as the field moves
• Machine learning. Building and training ML models across problem types, such as forecasting, optimization, classification, and anomaly detection
• Agent governance. Governance, testing, evaluation, and confidence scoring for agents, including when to hand off to a human
• Microsoft AI stack. Microsoft Copilot, Microsoft Foundry, Azure AI services, Azure Machine Learning, and Microsoft Agent 365
• AI-accelerated delivery. Using agentic tools such as Claude, Claude Code, and GitHub Copilot to accelerate agent outcomes

Preferred

• Consulting, agency, or professional services experience
• Data governance and security in regulated industries, especially healthcare (PHI / HIPAA) or insurance
• Experience with AWS AI services (such as Amazon Bedrock or SageMaker) or Google AI (such as Vertex AI or Gemini)
• CI/CD for data and AI workloads using Microsoft Fabric deployment pipelines, Databricks Asset Bundles, Git integration, Azure DevOps, or GitHub
• Microsoft Fabric data agents, Copilot in Power BI, and semantic model design for AI grounding
• Agile delivery experience working in sprints with a product owner or delivery lead

What will set you apart

• Hands-on time with Microsoft Fabric IQ, Microsoft Foundry IQ, or Work IQ, Microsoft Agent Framework, or MCP
• Full-stack development experience in .NET / C# with React or Angular. Many of our data and AI projects include an application layer
• Microsoft certifications such as DP-700, DP-600, or AI-102, or Databricks certifications such as Data Engineer Professional (or their current equivalents)
• A blog, talk, open-source contribution, or side project that shows how you think about data and AI in practice

The AI stack you will have

• A Claude Max account
• GitHub Copilot
• Microsoft Copilot across the M365 suite
• Our internal library of Claude skills, MCP servers, and agents, and a real opportunity to extend them
• Time and budget to stay current with Microsoft Fabric, Microsoft Foundry, Azure Databricks, and whatever comes next

Our hiring process

Every step below is required for any candidate moving forward. There are no exceptions.

Eligibility and direct hire. This is a US-based remote position. Candidates must be legally authorized to work in the United States indefinitely, without current or future sponsorship. This is a direct-hire position. We are not working with third-party recruiters, staffing firms, or corp-to-corp arrangements.

Identity verification. Candidates selected to interview will complete an identity verification check through Checkr before continuing. You can read more at checkr.com/background-check/identity-verification.

Skills assessment. Candidates may be asked to complete an online assessment focused on data engineering, AI solution design, and applying AI tools to real work.

Live technical interview. We will work through a structured design and build challenge together, with AI tools available and in use. It mirrors how work actually gets done here.

Background check (post-offer). Any candidate who accepts an offer will complete a full background check through Checkr before their first day.

Benefits

401(k), dental insurance, flexible spending account, health insurance, health savings account, paid time off, parental leave, retirement plan, vision insurance.

Pay: $145,000.00 - $185,000.00 per year

Benefits:

• 401(k)
• Dental insurance
• Flexible spending account
• Health insurance
• Health savings account
• Paid time off
• Parental leave
• Retirement plan
• Vision insurance

Work Location: Remote