Senior Data Engineer in Oakland

Job DescriptionJob Description

Key Responsibilities

• Build and maintain ETL/ELT data pipelines using PySpark and Databricks.
• Work with distributed datasets, ensuring efficiency, scalability, and accuracy.
• Develop and optimize workflows for data ingestion, transformation, and warehousing.
• Write high-quality Python code for data engineering use cases.
• Collaborate with data science and analytics teams to translate requirements into solutions.
• Troubleshoot issues, optimize queries, and manage performance within AWS cloud environments.
• Support CI/CD and automation workflows for reliability and repeatability.

Required Skills & Experience

• Strong hands-on experience in Python and PySpark for large-scale data engineering.
• Proven expertise in using Databricks for data transformation and analytics.
• Solid working knowledge of SQL and cloud data platforms (e.g., Snowflake).
• Experience with major cloud providers (AWS ; Azure/GCP considered).
• Good understanding of data modelling, data warehouses, and data lakes.
• Exposure to orchestration tools like Airflow, AWS Glue, or Azure Data Factory.
• Familiarity with DevOps practices, CI/CD, and version control systems.
• Strong problem-solving and debugging skills in distributed data environments.