Sr. Data Platform Engineer(AWS)- Bangalore- Hybrid- Immediate Joiner

1. Data Platform Engineering

• Design, develop, and maintain scalable data platform components supporting enterprise data ingestion, movement, processing, storage, and consumption.
• Build and enhance data pipelines for batch processing, streaming, and near-real-time data integration.
• Develop reusable frameworks and automation components for data ingestion, transformation, validation, reconciliation, logging, and error handling.
• Work with data warehouses, data lakes, relational databases, and distributed processing systems to support business-critical data workloads.
• Evaluate suitable technologies and architectural approaches based on scalability, reliability, maintainability, performance, and cost.
• Improve existing data platform architecture and modernize legacy data processing solutions where required.

2. Data Engineering and Distributed Processing

• Develop and optimize data processing solutions using Python, PySpark, Spark SQL, and other suitable programming technologies.
• Design and maintain ETL/ELT pipelines for structured and semi-structured data.
• Implement batch and streaming data processing using Apache Kafka or similar messaging technologies.
• Build efficient data transformation and ingestion workflows that support large data volumes and downstream applications.
• Develop data quality checks, validation mechanisms, reconciliation processes, and exception-handling frameworks.
• Troubleshoot pipeline failures, processing bottlenecks, data inconsistencies, and performance issues.
• Work with cloud data platforms such as Snowflake and data lake technologies to support scalable data storage and processing.

3. Cloud Infrastructure and Platform Automation

• Design and support cloud-based data infrastructure, particularly on AWS.
• Work with cloud storage, compute, networking, and data services to build reliable and cost-efficient platform solutions.
• Use infrastructure-as-code tools such as Terraform or Terragrunt to automate infrastructure provisioning and configuration.
• Work with containerization and orchestration technologies such as Docker and Kubernetes where applicable.
• Develop reusable infrastructure modules, engineering utilities, and automation frameworks to improve developer productivity.
• Implement CI/CD pipelines to automate code validation, testing, deployment, and release management.
• Improve deployment safety through automated validation, version control, rollback mechanisms, and repeatable delivery practices.

4. Software Engineering and Development

• Write clean, modular, testable, maintainable, and production-ready code using Python or another general-purpose programming language.
• Apply sound software engineering principles, including object-oriented design where appropriate, modular architecture, exception handling, and automated testing.
• Participate in code reviews, technical design discussions, and engineering best-practice initiatives.
• Build reusable libraries, frameworks, and utilities that can be adopted across engineering teams.
• Automate repetitive operational tasks and improve the efficiency of development and support workflows.
• Diagnose complex technical issues and implement sustainable solutions rather than relying on temporary fixes.

5. Performance Optimization and Production Operations

• Monitor data platform performance, availability, reliability, and resource utilization.
• Optimize Spark jobs, SQL queries, data models, partitioning strategies, and cloud resource configurations.
• Identify performance bottlenecks and improve processing efficiency and infrastructure cost.
• Implement observability, logging, monitoring, alerting, and operational readiness practices.
• Participate in incident investigation, root-cause analysis, remediation, and recovery planning.
• Design systems with appropriate failure handling, resilience, backup, recovery, and safe deployment practices.
• Take end-to-end ownership of assigned platform components from problem definition and design through implementation, rollout, and ongoing production support.

6. Data Security, Governance, and Compliance

• Incorporate data security, privacy, access control, and governance requirements into platform architecture and implementation.
• Implement appropriate access management, data classification, masking, and protection mechanisms where required.
• Support data lineage, auditability, data quality, and governance automation.
• Work with security, compliance, and business stakeholders to ensure platform solutions meet applicable organizational and regulatory requirements.
• Contribute to the development of reliable data systems operating in regulated or security-sensitive environments.

7. AI-Assisted Engineering and Productivity

• Actively use AI-assisted engineering tools to accelerate technical research, solution design, coding, debugging, testing, and documentation.
• Leverage tools such as GitHub Copilot, ChatGPT, Claude, or equivalent engineering assistants where appropriate.
• Critically validate AI-generated code, recommendations, and documentation before incorporating them into production solutions.
• Use AI responsibly while protecting confidential information, sensitive data, and proprietary code.
• Identify repeatable opportunities to incorporate AI into engineering workflows and improve development speed, quality, and operational effectiveness.

8. Technical Leadership and Collaboration

• Independently drive complex engineering assignments with minimal day-to-day supervision.
• Break ambiguous business or technical problems into clear milestones and deliver actionable solutions.
• Communicate architectural decisions, implementation approaches, technical risks, and delivery status clearly.
• Collaborate with distributed teams across India and US time zones using effective written and asynchronous communication.
• Participate in technical design reviews, documentation, knowledge sharing, and engineering best-practice initiatives.
• Mentor engineers and help improve the technical capability of the wider team where appropriate.

Required Technical Skills

Core Data Engineering

• Strong hands-on experience in data engineering, data platform development, or data infrastructure.
• Data ingestion, ETL/ELT, data transformation, and pipeline orchestration.
• Batch processing and/or streaming data processing.
• Experience with distributed data processing or messaging technologies.
• Production experience in at least one relevant domain, such as data pipelines, data warehouses, data lakes, distributed systems, operational databases, Kubernetes, or cloud infrastructure.

Programming and Software Development

• Strong proficiency in Python or another general-purpose programming language.
• Experience writing, testing, debugging, reviewing, and maintaining production-quality code.
• SQL and data processing fundamentals.
• Automation, version control, code quality, and CI/CD practices.

Cloud and Data Platforms

• Experience with cloud infrastructure, preferably AWS, or equivalent cloud platforms.
• Experience with one or more relevant data technologies such as Snowflake, Apache Kafka, Apache Spark, PySpark, PostgreSQL, or data lake technologies.
• Understanding of distributed systems, scalability, fault tolerance, and system design.

Production Engineering

• Experience operating production systems and resolving technical incidents.
• Knowledge of monitoring, observability, logging, deployment safety, and recovery mechanisms.
• Understanding of performance optimization, resource utilization, and infrastructure cost management.