Data Engineer

Job highlights

• Designing data systems / solutions to manage complex data . Strong understanding of database technologies and management systems . Atleast 1 year of exposure to working on AWS systems (Redshift and S3 must,Good to have EC2,lambda) Fluency in Python and Linux shell scripting and small scale server maintenance Strong understanding of cloud-based systems / services .

• we are looking for the skill set the candidate possesses and majorly Python and SQL in Big Data environment

• There were profiles with Python using Pandas and Numpy and these profiles are suited for Data Science or Data Analysis area of work

• Our JD is basically for Data Engineering role so Big Data, ETL and extensive programming using Python/PySpark is required

Responsibilities

• Designs and establishes secure and performant data architectures, enhancements, updates, and programming changes for portions and subsystems of data pipelines, repositories or models for structured/unstructured data
• Analyzes design and determines coding, programming, and integration activities required based on general objectives and knowledge of overall architecture of product or solution
• Writes and executes complete testing plans, protocols, and documentation for assigned portion of data system or component; identifies and debugs, and creates solutions for issues with code and integration into data system architecture
• Ability to manage and execute on an E2E data project from architecture inputs, to laying pipelines, data processing and dashboard building Leads a project team of other data engineers to develop reliable, cost effective and high-quality solutions for assigned data system, model, or component
• Collaborates and communicates with project team regarding project progress and issue resolution
• Represents the data engineering team for all phases of larger and more-complex development projects
• Provides guidance and mentoring to less experienced staff members

Knowledge and Skills

• Using data engineering tools, languages, frameworks to mine, cleanse and explore data
• Fluent in NoSQL & relational based systems
• Fluent in Databricks Fluent in complex, distributed and massively parallel systems
• Strong analytical and problem-solving skills with ability to represent complex algorithms in software
• Designing data systems/solutions to manage complex data
• Strong understanding of database technologies and management systems
• Atleast 1 year of exposure to working on AWS systems (Redshift and S3 must, Good to have EC2,lambda) Fluency in Python and Linux shell scripting and small scale server maintenance Strong understanding of cloud-based systems/services
• Database architecture testing methodology, including execution of test plans, debugging, and testing scripts and tools
• Ability to lay pipelines across multiple systems is a key requirement Familiarity with BI visualization tools, e,g, Looker, PowerBI, Tableau