Data Engineer

Job highlights

• Build the LCA library / reuse store (employer EIN,worksite,SOC,wage level,certification status,expiry) and the indexing required to make reuse checks fast.
• Experience Required . 5 8 years of data engineering experience,including at least 2 years building production ETL on SQL Server and at least 1 year operating a search index in production. . Key Responsibilities
• Good to Have

Job Details:

• Position Title: Data Engineer
• Job Location: Remote
• Experience: 5+ Years of experience.

Position Overview

Own the data plane behind the prevailing wage waterfall. Today, the firm consumes Towers Watson wage survey data as five large Excel spreadsheets (~240 columns, 60K+ rows per file, ~30 MB total) sitting on SharePoint case managers manually filter through them. This role replaces that with an ingested, searchable, validity-filtered, refresh-aware data store, and operationalizes the DOL OFLC data feed.

Experience Required

5 8 years of data engineering experience, including at least 2 years building production ETL on SQL Server and at least 1 year operating a search index in production.

Key Responsibilities

• Design and build the ingestion pipeline for the Towers Watson dataset five Excel files, normalize across them, pick the ~10 columns that matter, and load into a queryable store.
• Implement the validity filter that automatically excludes rows below the threshold ( 3 organizations AND 30 incumbents).
• Stand up a searchable, filterable interface (DB + index) for case managers the existing SharePoint/Excel workflow is being replaced.
• Build the annual mid-September refresh job: idempotent, validated, alerting on schema drift, with rollback to the previous years dataset.
• Build and maintain the DOL OFLC API ingestion (incremental sync, retry, rate-limit handling, key rotation) coordinate the API key procurement with the client.
• Design the Job Library schema (job title, primary SOC, alternate SOC, Occupation Title, wage level) and the import pipelines for each major clients library Amazon, AMD, American Airlines and others have proprietary formats.
• Build the LCA library / reuse store (employer EIN, worksite, SOC, wage level, certification status, expiry) and the indexing required to make reuse checks fast.
• Define data quality checks, monitoring, and reconciliation reports.
• Partner with the AI engineer on golden-set construction from historical LCAs.

Must-Have Skills

• SQL Server schema design, indexing, query tuning, partitioning.
• ETL on at least one of: .NET (SSIS / custom C#) or Python (pandas, SQLAlchemy).
• Excel / CSV ingestion at scale comfortable with messy real-world spreadsheets, mismatched headers, and 60K+ row files.
• Search infrastructure Azure AI Search, OpenSearch, or Elasticsearch including index design, analyzers, and faceted filtering.
• REST API integration with retry, backoff, and pagination.
• Job scheduling Azure Data Factory, Azure Functions on timer triggers, Hangfire, or equivalent.

Good to Have

• Snowflake exposure (the firm has an internal Snowflake initiative for this same data).
• Streamlit or any data-app prototyping experience useful for internal review tooling.
• Experience with US Department of Labor / BLS / OFLC datasets.
• Data versioning (DVC, lakeFS) for the annual refresh lineage.
• Comfort working with PII-adjacent but non-PII data in a regulated environment.