Senior Data Engineer

This role is responsible for the design, development, and maintenance of data integration and reporting solutions. Working primarily in Databricks, you will build and maintain the pipelines and datasets that feed our reporting, analytics, and AI/ML workloads. The ideal candidate will possess expertise in Databricks and strong skills in SQL Server, SSIS and SSRS, and experience with other modern data engineering tools such as Azure Data Factory. This position requires a proactive and results-oriented individual with a passion for data and a strong understanding of data warehousing principles. A strong grasp of AI-assisted development tools as a regular part of the engineering workflow is expected.

Responsibilities

• Design, develop, and maintain robust and efficient ETL pipelines and processes on Databricks.

• Troubleshoot and resolve Databricks pipeline errors and performance issues.

• Troubleshoot and resolve SSIS package errors and performance issues.

• Optimize data flow performance and minimize data latency.

• Implement data quality checks and validations within ETL processes.

• Maintain legacy SSIS packages for ETL processes.

Databricks Development

• Develop and maintain Databricks pipelines and datasets using Python, Spark and SQL.

• Drive the design and implementation of batch and streaming data lake architectures supporting both analytics and AI/ML workloads.

• Optimize Databricks jobs for performance and cost-effectiveness.

• Integrate Databricks with other data sources and systems.

• Migrate legacy systems (e.g. SSIS packages, SQL Server procs) to Databricks pipelines.

Data Warehousing

• Participate in the design and implementation of data warehousing.

• Support data quality initiatives and implement data cleansing procedures.