Senior Data Engineer

• We’re seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines that process and transform terabytes of data.
• In this role, you’ll collaborate closely with data scientists and other SWEs to ensure that our data infrastructure is scalable, reliable, and cost-effective
• Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases
• Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts
• Build high-throughput data transformations using Dask or Apache Spark
• Maintain training data clusters across hybrid (on-prem and cloud environments)
• Optimize training jobs for performance, resiliency, and cost
• Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence
• Troubleshoot, debug, and resolve data-processing failures in production
• Work with cross-functional teams to define data contracts, schemas, and quality checks
• Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation
• Design and maintain data models and schemas for AI/ML continuous training use cases
• Load data into cloud storage and lakes, ensuring performance and accessibility

Benefits

• ESOP
• Incentives and Bonuses
• Statutory Benefits: PF, ESI, Maternity Benefits, Leaves & Holidays
• VPF, NPS, Meal Vouchers, Paternity Leaves
• Health: Medical, Accidental, Term Life Insurances
• Relocation Benefits
• Flexi Working
• Reward & Recognition Policies
• Transparent Performance Management Process
• Career Progression
• Global Mobility- Containerization & Orchestration: Expertise with Docker, Kubernetes, and Helm
• ML Pipelines: Exposure to deploying cross-cluster model-training workflows using Ray or similar frameworks
• GCP Proficiency: Hands-on with Google Cloud services (e.g., Pub/Sub, Big Query, Cloud Storage, GKE). Equivalent experience in other public cloud providers is fine
• 3–5 years of professional experience designing and operating production data pipelines at scale
• Compiled Languages: Experience writing data services in Go or Rust
• Cloud Fundamentals: Familiarity with deploying and managing services in a cloud environment
• Security & Compliance: Knowledge of data governance, encryption, and role-based access control
• Programming: Strong proficiency in Python for data engineering tasks
• Distributed Frameworks: Practical experience with Dask or Apache Spark for large-scale data processing
• Workflow Management: Hands-on experience building DAG-based pipelines in Apache Airflow
• Infrastructure as Code: Familiarity with Terraform for deployment