Staff Platform Engineer - Big Data Platform Management

We are seeking an experience Big Data Platform Staff Engineer to join our team, responsible for managing and optimizing our Big Data Infrastructure platform. The successful candidate will have a deep understanding of Big Data technologies, architectures, and systems, as well as experience with managing large-scale data processing and analytics environments. The role will involve designing, implementing, and maintaining the Big Data Infrastructure platform, ensuring high availability, scalability, and performance.

• Design, implement and manage the Big Data Infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
• Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
• Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
• Ensure the security and governance of the Big Data Infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
• Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data Infrastructure platform.
• Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.