Sr Data Engineer

RESPONSIBILITIES:

• Work independently to provide solutions and leverage technical experience to align partners to optimal path for delivery
• Partner with team to craft, develop, and implement data pipelines and ETL systems for the data lake
• Perform data validation to ensure data quality and reliability
• Work with BI architect and Data Engineering Lead to build technical requirements and specifications for Business Intelligence products
• Identify and fix data pipeline Production issues
• Build clear documentation of procedures and protocols
• Implement data governance, data security and privacy policies while collaborating multi-functionally with technical and non-technical team members

QUALIFICATIONS:

• Bachelor’s degree in computer science, Information Systems, Engineering, Statistics, or related field
• 7-9 years’ experience as a Data Engineer designing, developing, and implementing batch and real-time data pipelines and ETL systems.
• 4-5 years' proven track record with AWS Data technologies experience with data lake technologies (S3, Glue, Redshift, RDS, DynamoDB, Data Pipeline, EMR) and data lake architectures
• Sophisticated knowledge in data warehousing solutions (Panoply, Redshift, etc.) and hands on experience using Cloud based database systems (Azure SQL, Bigtable, etc.)
• Direct experience using Python, Perl, Scala, Java, SAS, C++, as well as hands on development using real-time platform like Kafka, Kinesis etc.
• Experience working with Infrastructure as Code. Terraform experience is preferred, ability to write new, sophisticated queries with no supervision
• Firm grasp of Agile development methodologies, CI/CD automation, Test Driven Development and knowledge of standard methodologies in database engineering and data security
• Familiarity with machine learning methodologies and high level cloud migration & tools knowledge