Data Warehouse Engineer

Job Summary

We are seeking an experienced Data Platform & Systems Engineer with 8+ years of experience in cloud data platforms, API development/testing, cluster management, and system integration.

The ideal candidate will have strong expertise in Data Warehouse Cloud Platforms (DWCP) such as Snowflake, Redshift, BigQuery, Synapse, and Databricks, along with hands-on experience in Kubernetes, Docker, Spark, EMR, API testing, and Linux system administration.

The candidate will be responsible for validating end-to-end data platform solutions, developing and testing APIs, managing distributed data processing environments, and ensuring the reliability, scalability, and performance of data systems.

Key Responsibilities:

Data Warehouse Cloud Platforms

• Design, build, test, and support cloud-based data warehouse solutions using Snowflake, Redshift, BigQuery, Synapse, and Databricks.
• Validate data pipelines, data ingestion processes, and platform integrations.
• Perform performance tuning, optimization, and troubleshooting across cloud data platforms.
• Support enterprise-scale analytics and reporting environments.

API Development & Testing

• Develop, test, and validate RESTful APIs and service request interfaces. Create automated API test cases using industry-standard frameworks and tools.
• Perform API integration validation across multiple systems and applications.
• Troubleshoot API performance, authentication, and connectivity issues.

Cluster & Infrastructure Management

• Build, configure, and manage clusters using Kubernetes, Docker, EMR, Databricks, and Apache Spark.
• Monitor cluster health, scalability, availability, and performance.
• Support deployment and orchestration of containerized applications. Implement best practices for resource utilization and cluster optimization.

System Integration & Validation

• Conduct integration testing and end-to-end system validation.
• Analyze system architecture and dependencies to identify integration issues.
• Collaborate with development, DevOps, and platform engineering teams to ensure seamless deployments.
• Support release validation and production readiness testing.

Linux/Unix Administration

• Perform system administration activities in Linux/Unix environments. Develop shell scripts for automation, monitoring, deployment, and troubleshooting.
• Investigate and resolve system-level issues using command-line tools and debugging techniques.
• Support server performance monitoring, log analysis, and incident resolution.

Required Qualifications

• 8+ years of experience in Data Engineering, Platform Engineering, System Testing, or related domains.
• Strong experience with at least one or more cloud data platforms:
• Snowflake, Redshift, BigQuery, Synapse, or Databricks.
• Hands-on experience in API development and API testing.
• Strong Linux/Unix system administration experience.

Technical Skills

Data Platforms

• Snowflake
• Amazon Redshift
• Google BigQuery
• Azure Synapse Analytics
• Databricks

API Technologies

• REST APIs
• Postman
• Swagger/OpenAPI
• API Automation Frameworks
• JSON/XML

Cluster & Container Technologies

• Kubernetes
• Docker
• Apache Spark
• AWS EMR
• Databricks Clusters

System Validation

• Integration Testing
• End-to-End Testing
• System Performance Testing
• Root Cause Analysis

Operating Systems

• Linux/Unix
• Shell Scripting (Bash/Korn Shell)
• System Monitoring & Debugging
• Log Analysis

Preferred Skills

• Experience with CI/CD tools such as Jenkins, GitHub Actions, Azure DevOps, or GitLab.
• Familiarity with cloud platforms (AWS, Azure, GCP).
• Knowledge of DevOps and Site Reliability Engineering (SRE) practices.
• Experience with Infrastructure as Code (Terraform, CloudFormation).
• Exposure to distributed systems and microservices architecture.

Soft Skills

• Strong analytical and problem-solving ability.
• Excellent communication and stakeholder management skills.
• Ability to work in a fast-paced, cross-functional environment.
• Strong collaboration and troubleshooting mindset.