Big Data Engineer (Live Anywhere)

Big Data Engineer Join a dynamic team optimizing a modern, Lakehouse-focused analytics platform. This position blends strategic business insight with hands-on technical execution. You will collaborate with cross-functional partners to gather requirements, design pragmatic architectures, and deploy scalable streaming and ingestion pipelines end-to-end.

Responsibilities

• Design, build, and maintain production ETL/ELT pipelines using Databricks, Apache Spark (PySpark/Spark SQL), and Delta Lake.
• Implement Delta Live Tables (DLT) for streaming/batch workflows and set up data governance via Unity Catalog.
• Integrate high-volume event streams using Apache Kafka and leverage AWS cloud infrastructure (S3, IAM).
• Cleanse, transform, and curate large structured and semi-structured datasets to support BI reporting and machine learning needs.
• Enforce data quality, operational observability, and engineering best practices in an Agile environment.

Qualifications

• 5+ years in Data Engineering; strong proficiency in SQL, Spark, and Databricks.
• Hands-on experience with Delta Lake, Delta Live Tables, Unity Catalog, Kafka, and AWS analytics services.
• Background in Agile delivery and translating complex requirements into reliable models.
• Preferred: Exposure to Snowflake, NoSQL databases (e.g., Cassandra), or enterprise messaging systems.

Location & Work Schedule: This is a 6+ month auto-renewing contract. Open to 100% remote work for non-local candidates willing to work EST hours. Atlanta-based candidates will work a hybrid schedule (3 days in-office).

Through Visionaire Partners, full-time W2 contractors gain access to a full benefits package (for contractors, spouses, domestic partners, and dependents), including a 401(k) with up to 4% match, medical, dental, vision, life insurance, short/long-term disability, critical illness, hospital indemnity, accident coverage, and Medical/Dependent Care FSAs.