Lead Engineer (Python)

We are looking for an experienced Python Team Lead with strong expertise in Big Data, statistical data analysis, AWS cloud services, and Agentic AI.

The ideal candidate will combine strong software engineering and team leadership skills with hands-on experience in large-scale data processing, statistical analytics, cloud-native architectures, and modern AI/LLM-based solutions.

The candidate will lead a team of Python engineers and be responsible for designing and delivering scalable data and AI solutions, while working closely with Data Engineering, Product, DevOps, and Business teams.

Key Responsibilities :

Technical & Team Leadership -

• Lead, mentor, and guide a team of Python developers/engineers.
• Own technical architecture, design, development, code quality, and delivery.
• Conduct code reviews and establish engineering best practices.
• Define scalable and maintainable solutions for complex data and analytics problems.
• Troubleshoot complex production, data, performance, and scalability issues.
• Mentor team members on Python, Big Data, AWS, statistics, and AI technologies.
• Collaborate with Product, Data Science, DevOps, and Business teams to translate requirements into technical solutions.

Python & Software Engineering -

• Strong hands-on development experience with Python.

• Design and develop scalable backend services, APIs, data pipelines, and automation frameworks.

• Experience with FastAPI.

• Strong understanding of:

• Object-oriented programming

• Design patterns

• Async programming

• Multiprocessing/concurrency

• Exception handling

• Logging and monitoring

• Unit/integration testing

• Experience with Git, CI/CD, containerization, and production deployment.

Big Data & Data Engineering :

• Strong understanding of Big Data architectures and distributed data processing.

• Experience working with very large datasets and high-volume data pipelines.

• Hands-on experience with technologies such as:

• Apache Spark / PySpark

• Kafka

• Amazon S3

• Amazon Athena

• Parquet, CSV, JSON and other data formats

• Data lakes and analytical databases

• Understanding of distributed computing, partitioning, parallel processing, data serialization, and query optimization.

• Experience designing scalable ETL/ELT pipelines.

• Ability to identify and resolve data-processing and query-performance bottlenecks.

AWS & Cloud :

Strong hands-on experience with AWS services, particularly:

• AWS Lambda serverless applications and data-processing workflows.
• Amazon S3 data lake and object storage.
• Amazon Athena querying large-scale datasets.
• AWS KMS encryption and key management.
• AWS Secrets Manager secure management of credentials and secrets.
• AWS IAM – roles, policies, permissions, and least-privilege access.
• Amazon CloudWatch – logging, monitoring, and alerting.

Additional AWS experience with Glue, EMR, Step Functions, EventBridge, Bedrock, ECS/EKS, or similar services is a strong advantage. The candidate should understand secure AWS architecture, including how IAM, KMS, Secrets Manager, Lambda, S3, and AI services can be integrated securely.

Agentic AI & Gen AI :

The candidate should have a strong understanding of Generative AI and Agentic AI architectures and be able to identify opportunities where AI agents can automate data and engineering workflows.

Key Responsibilities

• Design and develop AI agents and agentic workflows using Python.

• Build AI agents capable of:

• Understanding business/data requirements

• Generating and executing data queries

• Performing data analysis

• Calling APIs and external services

• Orchestrating multi-step workflows

• Validating results

• Taking actions based on defined business rules

• Integrate LLMs with enterprise data and applications.

• Build tool-using AI agents that can interact with databases, APIs, AWS services, data pipelines, and internal systems.

• Implement RAG (Retrieval-Augmented Generation) solutions for accessing enterprise knowledge and documentation.

• Combine semantic/vector search with structured data and metadata to improve AI-agent decision making.

• Develop agent workflows with appropriate guardrails, validation, authorization, and observability.

• Evaluate LLM/agent responses for accuracy, reliability, security, and cost.

• Identify opportunities to automate repetitive data engineering, analytics, troubleshooting, and operational tasks using AI agents.

Summary :

Ideal Candidate Profile

Role: Python Team Lead – Big Data, Statistical Analytics & Agentic AI
Experience: 5+ years
Core Technologies: Python, PySpark/Spark, SQL, Big Data, AWS, Statistics, Generative AI
Agentic AI: AI Agents, LLMs, RAG, Tool Calling, MCP, Agent Workflows
AWS: Lambda, S3, Athena, KMS, Secrets Manager, IAM, CloudWatch, Bedrock
Leadership: Team Management, Architecture, Mentoring, Code Quality, Technical Strategy