Databricks Data Engineer

Responsibilities

• Understand the business requirements and translate these to data services to solve the business and data problems

• Develop and manage the transports/data pipelines (ETL/ELT jobs) and retrieve applicable datasets for specific use cases using cloud data platforms and tools

• Explore new technologies and tools to design complex data modeling scenarios, transformations and provide optimal data engineering solutions

• Build data integration layers to connect with different heterogeneous sources using various approaches

• Understand data and metadata to support consistency of information retrieval, combination, analysis and reporting

• Troubleshoot and monitor data pipelines to have high availability of reporting layer

• Collaborate with many teams - engineering and business, to build scalable and optimized data solutions

  Qualifications

• Over 1-3 years of experience with building the data pipelines or data ingestion for both Batch/Streaming data from different sources to data warehouse / data lake

• Databricks certified

• Handson experience/knowledge with SQL/Python/Java/Scala/Spark programming, understanding of SQLand Spark is a must

• Experience with any cloud computing platforms like AWS (S3, Lambda functions, RedShift, Athena), GCP (GCS, Dataflow, Dataproc, Pub/Sub, Bigquery), Azure (Blob/Azure, Synapse) along with experience in Databricks

• Experience/knowledge with Big Data tools (Hadoop, Hive, Spark, Presto)

• Data pipelines orchestration tool (Ozzie, Airflow, Nifi)

• Any streaming engines (Kafka, Storm, Spark Streaming)

• Any relational database or data warehousing experience

• Any ETL tool experience (Informatica, Talend, Pentaho, Business Objects Data Services, Hevo)

• Good communication skills

• Experience in working independently and strong analytical skills

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.