Staff Engineer (Platform Engineer)

Must have Skills : AWS Glue Sodacore Lakehouse architecture CDC Medallion design principles 

Key Responsibilities

Data Ingestion Development
o Build and implement AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
o Implement correct loading methods based on source requirements (CDC full load delta snapshot).
o Design and execute historical loading mechanisms to bring legacy data into the Lakehouse.
Performance Optimization
o Optimize Glue job performance (DPU allocation parallelization partitioning) according to best practices.
o Collaborate with platform teams to ensure tooling and optimization alignment.
Migration & Automation
o Aggressively migrate source tables to Bronze layer initially using manual approaches with standards/templates later leveraging AI-enabled acceleration.
o Ensure jobs are version-controlled and production deployment is automated via Git and Terraform.
Governance & Monitoring
o Implement source system connectivity into CDP in collaboration with source system owners.
o Ensure jobs comply with data contracts and are properly monitored.
o Prepare documentation and handover to operational support teams.
Collaboration
o Work closely with Data Architect for ingestion patterns and standards.
o Coordinate with Data Assurance Lead to apply quality checks across all jobs.
o Partner with platform engineers for tooling and optimization. 

Required Skills & Experience

Technical Expertise
o Strong experience in AWS Glue PySpark and ETL pipeline development.
o Knowledge of Lakehouse architecture and Medallion design principles. - imp
o Familiarity with CDC delta loads and historical data ingestion strategies.
Tools & Technologies
o AWS services: Glue S3 Athena Lambda. - Imp
o Git Terraform for CI/CD automation.
o Understanding of data quality frameworks (e.g. Soda Core) is a plus. - imp
o Able to identify ways to automate their work / repetitive tasks.
Soft Skills
o Ability to work in a fast-paced environment and deliver aggressive migration targets.
o Strong collaboration and communication skills.
o Work with Jira and agile way of working.

Remote Work :

No

Employment Type :

Full-time

Key Skills
Computer Science,Docker,Kubernetes,Python,VMware,C/C++,Go,System Architecture,gRPC,OS Kernels,Perl,Distributed Systems
Department / Functional Area: Engineering
Experience: years
Vacancy: 1