Lead Data Engineer

Accountabilities:

You will lead the technical direction of data migration and modernization initiatives, ensuring the successful transition of legacy systems to scalable cloud-based infrastructure. Your responsibilities will include:

• Define and own the technical approach for migrating legacy data warehouse environments and pipelines to an Azure and Databricks lakehouse architecture.

• Establish standardized migration frameworks, technical patterns, and best practices to guide data engineering teams.

• Design and develop reusable data migration pipelines using Azure Data Factory and Azure Databricks.

• Define data reconciliation strategies and validation processes to ensure accuracy and consistency between source and target systems.

• Establish data quality controls, testing standards, and quality gates throughout the migration lifecycle.

• Provide technical leadership in resolving complex pipeline challenges, performance bottlenecks, and data engineering issues.

• Optimize data processing workflows to improve scalability, reliability, and overall system performance.

• Mentor data engineers, provide technical guidance, and review engineering deliverables to maintain high standards of quality.

• Collaborate with Product, Quality Assurance, and Platform teams to coordinate technical activities and ensure successful project delivery.

• Promote consistent engineering practices and scalable architectural solutions across migration workstreams.

• Communicate technical decisions, progress, risks, and recommendations clearly to internal stakeholders and customers.

• Contribute to additional technical and operational responsibilities as required.

Requirements:

The ideal candidate is an experienced data engineering leader with a strong background in cloud modernization, large-scale data migrations, and distributed data processing. You should bring:

• A Bachelor's degree in a relevant field.

• Proven experience leading data engineering teams or technical workstreams within large-scale migration or modernization programs.

• Deep expertise in Databricks and/or Apache Spark, with hands-on experience designing and implementing data processing solutions.

• Strong experience with Azure Data Factory and cloud-based data pipeline development.

• Advanced SQL and Python programming skills.

• Demonstrated experience migrating legacy ETL processes and data warehouse workloads to cloud environments.

• Strong understanding of lakehouse architecture, medallion architecture, and Delta Lake.

• Experience designing reusable data pipelines, data validation frameworks, and reconciliation processes.

• Ability to troubleshoot complex data engineering problems and optimize pipeline performance.

• Proven technical leadership skills, including mentoring engineers, reviewing code, and establishing engineering best practices.

• Excellent communication and collaboration skills, with the ability to explain technical concepts and align stakeholders around technical direction.

• Experience working effectively in remote, collaborative Agile environments.

• A proactive, adaptable, and solution-oriented mindset.

Additional qualifications that would be advantageous include:

• Experience working in federal, government, or other regulated data environments.

• Familiarity with government data modernization initiatives and associated compliance requirements.

Additional Employment Requirements:

• Must be legally authorized to work in the United States.

• Depending on federal contract requirements, certain positions may require U.S. citizenship.

• Ability to obtain and maintain a federal background investigation and/or security clearance, if required.

Benefits:

• Annual salary range of $110,000–$145,000.

• Fully remote work arrangement.

• Comprehensive employee benefits package.

• Technology and education stipend.

• 401(k) retirement plan with company matching.

• Flexible paid time off (PTO).

• Paid holidays.

• Opportunities for professional development and technical growth.

• Collaborative environment within a small, impact-driven organization.

• Opportunity to contribute to meaningful technology initiatives supporting the public sector.