Data Engineer (Snowflakes, Databricks)

Related Work Experience

• At least 3 years of experience in data engineering, data analytics, or related fields.
• Proficiency in Python and Java, with experience in other programming languages as an advantage.
• Hands-on experience with data manipulation and processing tools such as pandas, dplyr, and Apache Spark.
• Proficiency in ETL/ELT processes and the development, orchestration, and maintenance of data pipelines.
• Strong proficiency in SQL and experience with NoSQL databases.
• Hands-on experience with Databricks for data engineering, large-scale data processing, and pipeline development.
• Hands-on experience with Snowflake for data warehousing, data transformation, query optimization, and data management.
• Hands-on experience with Google Cloud Platform (GCP) and its data engineering services.
• Experience with big data storage and processing technologies such as Apache Spark, Hadoop, Hive, MongoDB, Snowflake, and similar platforms.
• Experience working with cloud-based or server-based data processing environments, particularly GCP, AWS, or Azure.

Knowledge

• Comprehensive understanding of data manipulation tools such as pandas, dplyr, and Apache Spark.
• In-depth knowledge of big data frameworks and technologies, including Apache Spark, Hadoop, and Databricks.
• Strong understanding of data warehousing concepts, including Snowflake architecture, data modeling, and query performance optimization.
• Knowledge of GCP data services, particularly BigQuery, Cloud Storage, Dataflow, Dataproc, and Cloud Composer.
• Familiarity with AWS data services, particularly AWS Glue and AWS Lake Formation.
• Understanding of ETL/ELT design patterns, data pipeline orchestration, and workflow automation.
• Familiarity with business intelligence tools such as Tableau, Power BI, Amazon QuickSight, or Looker Studio.
• Awareness of data visualization libraries and packages such as Dash, Plotly, Matplotlib, ggplot, and Folium.
• Understanding of machine learning libraries and tools, such as scikit-learn, caret, and MATLAB, is an advantage.
• Knowledge of data governance, data quality management, cloud security, access control, and data protection best practices.

Skills

• Problem-Solving: Ability to address complex technical challenges and deliver efficient, scalable data solutions.
• Technical Proficiency: Ability to develop and optimize data pipelines using Databricks, Snowflake, GCP, Python, SQL, and related technologies.
• Communication: Clear and concise communication skills when collaborating with team members and stakeholders.
• Collaboration: Ability to work effectively with cross-functional teams to achieve shared goals.
• Time Management: Capacity to manage tasks, prioritize deliverables, and meet deadlines.
• Attention to Detail: Careful and accurate handling of data to ensure quality, consistency, and integrity.
• Client Focus: Ability to understand business needs and align data solutions with business objectives and decision-making requirements.
• Adaptability: Willingness to learn and adapt to new tools, technologies, and processes in a rapidly changing environment.