Data Engineering- SSEII

Job Summary

What You'll Do:

• Operationalize our Databricks Lakehouse: Develop and maintain a unified data platform using Azure Databricks, enabling efficient data storage, processing, and analysis across the organization.

• Build Data Pipelines: Design, develop, and maintain scalable data pipelines; establish API integrations for efficient data transfer. Implement ETL processes and ensure data integrity and quality.

• Deploy Data Technology: Utilize technologies such as Spark, Kafka, and Airflow to manage large-scale data processing. Test, monitor and troubleshoot systems to ensure smooth operation.

• Collaborate: Work closely with data engineers, analysts, and scientists to understand their needs and optimize the performance of data workflows

  What You'll Bring

• Development Experience: 5+ years with Python, Scala, Java, or C#. Experience with Python is preferred.

• Data Engineering Tools: 5+ years with Databricks, Snowflake, BigQuery, Apache Spark, HIVE, Hadoop, Cloudera, or RedShift. Experience with PySpark on Databricks is preferred.

• Data Architecture: Proficiency in high-performance data pipeline design and mastery of pipeline guarantees, such as Idempotency, At-Least-Once Processing, Exact-Once Processing, Fault Tolerance, Eventual Consistency, Streaming Consistency, Transactional Consistency, and Observability.

• Containerization: Experience developing in a containerized environment like Docker, Rancher, or Kubernetes.

• Data Orchestration: Experience with Apache Airflow (or similar tool) for orchestrating data processing jobs.

• Education: Bachelors degree in Computer Science (or related field) or equivalent combination of education and experience.

  Educational Qualification and Experience

Minimum of 15 years of formal education - Graduate / Post Graduate in Computer Science / Information Technology

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.