Senior Data Engineer

About The Opportunity

We are looking for a Senior Data Engineer to join a team responsible for building and operating large-scale, real-time streaming data platforms that power analytics, reporting, and data-driven products.

In this role, you'll help design and maintain modern streaming architectures capable of processing high volumes of event data with low latency and high reliability. You'll collaborate closely with platform, analytics, and engineering teams to develop scalable data solutions while influencing architecture decisions and improving the overall data ecosystem.

This is an excellent opportunity for engineers who enjoy solving complex distributed systems challenges and building production-grade streaming infrastructure.

Responsibilities

• Design, build, and operate real-time streaming data pipelines using Apache Kafka and Apache Flink.
• Develop production-grade streaming applications capable of processing large volumes of event data with low latency.
• Build and maintain Snowpipe Streaming integrations for continuous ingestion into Snowflake.
• Develop and optimize Snowflake data models using Dynamic Tables, Streams, and Tasks.
• Design and maintain semantic layers and analytical models using Cube.dev and/or MicroStrategy.
• Build highly scalable streaming pipelines while addressing data skew challenges through bucketing, salting, repartitioning, and adaptive partitioning techniques.
• Manage Apache Iceberg tables, including partitioning strategies, compaction tuning, snapshot management, and file optimization.
• Collaborate with platform and analytics teams to design efficient data models for real-time analytics and reporting.
• Optimize streaming jobs, storage layouts, and distributed queries to improve performance while reducing infrastructure costs.
• Troubleshoot ingestion bottlenecks, distributed processing issues, and query performance problems.
• Contribute to shared platform components, internal frameworks, code reviews, and technical documentation.
• Participate in architecture discussions and help define best practices for streaming data platforms.

Required Qualifications

• 5+ years of experience in Data Engineering or Software Engineering.
• Strong production experience building and operating Kafka-based streaming systems.
• Hands-on experience developing streaming applications with Apache Flink.
• Solid experience with Snowflake, including:Snowpipe Streaming Dynamic Tables Streams Tasks
• Experience building semantic layers or analytical cubes using Cube.dev and/or MicroStrategy.
• Strong understanding of distributed data processing and techniques for handling data skew:Bucketing Salting Repartitioning Adaptive partitioning
• Experience working with Apache Iceberg for streaming or near real-time workloads.
• Strong programming skills in Java and/or Python.
• Advanced SQL skills with experience optimizing distributed query execution.
• Experience operating production data pipelines, including monitoring, alerting, troubleshooting, and incident response.
• Excellent communication skills and ability to collaborate across engineering teams.

Nice To Have Skills

• Experience with analytical databases such as:StarRocks Apache Pinot Apache Druid
• Experience with MPP database distribution strategies.
• Familiarity with materialized views and real-time OLAP optimization.
• Experience contributing to open-source data engineering projects.
• Experience building large-scale event-driven architectures.
• Experience with high-throughput distributed systems.

Avenue Code reinforces its commitment to privacy and to all the principles guaranteed by the most accurate global data protection laws, such as GDPR, LGPD, CCPA and CPRA. The Candidate data shared with Avenue Code will be kept confidential and will not be transmitted to disinterested third parties, nor will it be used for purposes other than the application for open positions. As a Consultancy company, Avenue Code may share your information with its clients and other Companies from the CompassUol Group to which Avenue Code’s consultants are allocated to perform its services.