Data Engineer - Scala/PySpark

Contract Duration : 3 months

Skills : - PySpark or Scala with Spark, Spark Architecture, Hadoop, SQL

• Streaming Technologies like Kafka etc.- Proficiency in Advanced SQL (Window functions)- Airflow, S3, and Stream Sets or similar ETL tools.- Basic Knowledge on AWS IAM, AWS EMR and Snowflake.

Responsibilities :

• Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of structured and unstructured data.

• Data Modeling: Develop and maintain data models, schemas, and metadata to support the organization's data initiatives. Ensure data integrity and optimize data storage and retrieval processes.

• Data Integration: Integrate data from various sources, including databases, data warehouses, APIs, and streaming platforms, ensuring compatibility, consistency, and quality.

• Performance Optimization: Optimize data pipelines and processing systems for performance, scalability, and reliability. Identify and resolve bottlenecks and inefficiencies in data workflows.

• Data Quality Assurance: Implement data quality checks, validation rules, and monitoring mechanisms to ensure the accuracy, completeness, and consistency of data across different systems.

• Infrastructure Management: Collaborate with DevOps and IT teams to provision, configure, and manage infrastructure components such as databases, clusters, and cloud services required for data processing and storage.

• Security and Compliance: Implement data security best practices and compliance standards to protect sensitive information and ensure regulatory compliance (e.g., GDPR, HIPAA). Monitor and audit data access and usage to prevent unauthorized activities.

• Documentation and Communication: Document data pipelines, systems architecture, and technical processes. Communicate effectively with cross-functional teams to gather requirements, provide updates, and address issues.

• Continuous Learning: Stay updated on emerging technologies, tools, and techniques in data engineering and related fields. Evaluate and recommend new technologies and approaches to improve data infrastructure and workflows.