Big Data Engineer L2

Key Responsibilities

Defining and developing services and solutions

Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.

Define, design, and run robust data pipelines/batch jobs in a production environment.

Architecting highly scalable, highly concurrent, and low latency systems

Maintain, support, and enhance current systems.

Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.

Contribute feedback to improve the quality, readability, and testability of the code base within your team.

Mentor and train other developers in a non-line management capacity.

Build tools (One of SBT, Gradle, Maven).

Ensure all software built is robust and scalable.

Collaborating with Internal and external stakeholders

Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.

Work with third-party and other internal providers to support a variety of integrations.

As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.

Proactively advise on best practices.

Desired Candidate Profile

To Be The Right Fit, You'll Need

Degree in Computer Science, Software Engineering or related preferred

Minimum of 5 years experience in a Big Data

Follow Clean Code/Solid principles

Adhere and use TDD/BDD.

Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.

Highly Proficient in either Functional Java or Scala, Python

Knowledge of Azure Big Data/Analytics services ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks

Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs

Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs

In-depth knowledge of Hadoop technology ecosystem HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet

Experience debugging a complex multi-server service.

In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)

Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.

Knowledge of relational and non-relational database systems

Understanding continuous integration and delivery.

Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).

IDE Intellij or Eclipse.

Build tools (One of SBT, Gradle, Maven).

Ensure all software built is robust and scalable.

An ability to communicate technical concepts to a non-technical audience.

Working knowledge of unix-like operating systems such as Linux and/or Mac OS X.

Knowledge of the git version control system.

Ability to quickly research and learn new programming tools and techniques.