Principal Data Engineer - ETL Development

Job Responsibilities :

1. ETL Development and Data Processing :

• Design, develop, and implement ETL processes to extract, transform, and load data from various sources into data warehouses or analytics systems.

• Maintain and enhance ETL pipelines to ensure data quality, integrity, and consistency.

2. DevOps and CI/CD Pipeline Management :

• Implement and support CI/CD pipelines for Enterprise Data Warehouse (EDW) code development and deployment using GitLab.

• Manage and configure workflows for code repositories, ensuring seamless code integration and promotion across environments.

3. Data Analysis and Reporting :

• Apply statistical analysis and data mining techniques to uncover trends, patterns, and insights within large datasets.

• Develop and maintain dashboards, reports, and visualizations to communicate key insights to stakeholders.

4. Programming and Scripting :

• Develop and maintain Linux shell scripting, Python, and PL/SQL code for data extraction, transformation, and automation tasks.

• Create efficient scripts for data replication, migration, and data integration between systems.

5. Quality Assurance and System Optimization :

• Perform quality assurance checks on system installations, configurations, and code to ensure performance and scalability.

• Recommend improvements to optimize performance and reliability of data processes and systems.

6. Business Understanding and Translation :

• Work closely with business stakeholders to understand their goals, processes, and requirements, translating them into technical data solutions.

• Ensure alignment between technical solutions and business objectives to drive strategic decision-making.

7. Data Governance and Security :

• Implement data governance best practices to ensure the security, integrity, and privacy of enterprise data.

• Adhere to data security regulations and policies, ensuring that data is handled with confidentiality and compliance.

8. Collaboration and Stakeholder Communication :

• Collaborate with cross-functional teams, including data analysts, engineers, and business teams, to deliver impactful data solutions.

• Present data findings and actionable insights to stakeholders at various organizational levels, both written and verbally.

**Required Qualifications :

Experience :**

• 8+ years of experience in IT with a focus on Business Intelligence, Data Warehousing, and Analytics systems.

• Proven experience with DevOps methodologies, CI/CD pipelines, and GitLab workflows.

• Proficient in ETL development and creating data workflows in SQL, PL/SQL, Python, and Linux shell scripting.

Technical Skills :

• Strong knowledge of open-source technologies, Linux, and GitLab for codebase management.

• Experience in data replication and data migration processes.

• Hands-on experience with Python and Node.js for programming and automation tasks.

Soft Skills :

• Excellent problem-solving and analytical abilities.

• Strong communication skills with the ability to convey complex technical concepts to non-technical stakeholders.

• Ability to manage multiple projects in a dynamic and fast-paced environment.

Preferred Skills :

• Experience with data governance principles and ensuring data security compliance.

• Familiarity with data quality assurance and optimization techniques.

• Understanding of modern data platforms such as cloud-based data warehouses and data lakes.

Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.