Data Engineer - SQL/PySpark

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure.

Key Responsibilities :

SQL Development :

• Write medium-complexity SQL queries to extract, transform, and load data efficiently.

• Optimize SQL queries for performance and scalability.

• Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.

PySpark/Spark Development :

• Develop and maintain data processing scripts using PySpark or Spark with Scala.

• Implement data transformation and ETL processes using Spark to handle large-scale data.

• Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.

Data Engineering Pipeline Design :

• Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.

• Ensure data pipelines are scalable, reliable, and maintainable.

• Collaborate with the team to understand data flow requirements and implement appropriate solutions.

Cloud Technology Implementation :

• Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.

• Implement cloud-based data storage and processing solutions, ensuring security and compliance.

• Utilize cloud services for data integration, orchestration, and processing.

Issue Resolution and Debugging :

• Identify, diagnose, and resolve data-related issues and bugs independently.

• Perform root cause analysis and implement fixes for any data discrepancies or processing failures.

• Monitor data pipelines and systems to ensure continuous availability and reliability.

CI/CD Integration :

• Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.

• Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.

• Automate testing and deployment processes to improve development efficiency.

Desired Skills :

Data Modeling :

• Experience in designing data models to support analytical and business requirements.

Airflow :

• Experience with Apache Airflow for workflow scheduling and orchestration.

Kafka :

• Knowledge of Kafka for real-time data streaming and integration.

EMR/Databricks :

• Experience with AWS EMR or Databricks for big data processing and analytics.

Snowflake :

• Familiarity with Snowflake for cloud-based data warehousing solutions.

Hive :

• Understanding of Hive for querying and managing large datasets.

**Qualifications :

Experience :** 5+ years of experience in data engineering or related fields.

Technical Skills :

• Strong proficiency in SQL and writing medium-complexity queries.

• Hands-on experience with PySpark or Spark with Scala.

• Understanding of ETL processes and data engineering pipeline design.

• Basic to intermediate knowledge of CI/CD processes.

• Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving :

• Ability to troubleshoot and resolve data issues independently.

Communication :

• Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education :

• Bachelor's degree in Computer Science, Information Technology, or related field.

Location : - Anywhere india ,Multiple Locations