Lead Data Engineer (Databricks) at ShyftLabs

📋 Description

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
• Build data pipelines with Databricks Unity Catalog.
• Implement business logic, data transformations, and dimensional data models.
• Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
• Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, Gold).

🎯 Requirements

• Strong expertise in Python, PySpark, and Advanced SQL.
• Hands-on experience with the Databricks Lakehouse Platform.
• Good understanding of Unity Catalog, Delta Lake, Databricks Workflows/Jobs, Clusters, Notebooks
• Experience integrating with REST APIs for data ingestion and data export.
• Strong knowledge of ETL/ELT development, batch processing, incremental loading, and data
• Experience with data modeling (Star Schema, Snowflake Schema, Fact & Dimension tables, SCD

🎁 Benefits

• We are proud to offer a competitive salary alongside a strong insurance package.