Director of Technology, Enterprise Data Engineering

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Director of Technology, Enterprise Data Engineering based in the United States.

Lead the evolution of a modern, cloud-based enterprise data platform in a fully remote environment, driving innovation, reliability, and operational excellence.
You will oversee a talented data engineering team responsible for maintaining critical data pipelines while modernizing legacy systems and migrating to Databricks.
This strategic leadership role combines hands-on technical expertise with people management, cross-functional collaboration, and architectural decision-making.
You will establish scalable engineering standards, trusted data models, and self-service tools that empower distributed teams to work independently.
You will also champion responsible AI adoption, integrating intelligent automation into development, testing, documentation, and quality assurance.
Working closely with Enterprise Architecture, Analytics Operations, Information Security, and external partners, you will ensure that data solutions remain secure, compliant, and business-ready.
This is an opportunity to shape the future of enterprise data engineering while enabling faster innovation and measurable business impact in a regulated industry.

Accountabilities

Data Engineering Leadership: Lead, coach, and develop data engineering professionals, managing performance, staffing, workload distribution, and career development. Establish consistent standards for employees, contractors, vendors, and engineers working within cross-functional teams.
Platform Modernization: Lead the migration of legacy data lakes and curated data stores to Databricks, coordinating domain-by-domain delivery and resolving differences across development, testing, and production environments.
Data Architecture and Modeling: Partner with Enterprise Architecture to translate strategic direction into production-ready data pipelines, data models, platform designs, and consistent engineering practices.
Pipeline Development and Reliability: Own data ingestion and medallion architecture standards across bronze, silver, and gold layers. Maintain the availability, accuracy, performance, and scalability of existing pipelines while delivering new data integrations and supporting evolving business products.
Semantic Data and Analytics Enablement: Develop and maintain the semantic data layer, Unity Catalog objects, and curated datasets in collaboration with Enterprise Data and Analytics Operations. Ensure consistent business definitions and reliable data for analytics, governance, and AI-assisted querying.
Self-Service Engineering Enablement: Establish reusable pipeline templates, scaffolding, reference architectures, and automated tooling that allow autonomous teams to develop and operate compliant data pipelines with minimal centralized support.
AI-Driven Engineering: Integrate AI tools and agentic development practices into daily workflows, including code generation, testing, documentation, and automated reviews. Define standards for AI-assisted development and measure its impact on productivity, quality, cost, and delivery speed.
Governance, Security, and Risk Management: Ensure engineering deliverables meet data quality, lineage, documentation, privacy, and regulatory requirements. Collaborate with Information Security to support audits, remediate vulnerabilities, manage sensitive data appropriately, and implement automated compliance guardrails.
Vendor and Stakeholder Management: Manage relationships with system integration partners and platform vendors, taking ownership of architectural decisions and technical deliverables. Communicate progress, risks, dependencies, and recommendations clearly to senior leaders and business stakeholders.
Performance and Continuous Improvement: Define and monitor service-level agreements, operational KPIs, platform consumption costs, and data quality metrics. Lead complex troubleshooting efforts, maintain technical documentation and runbooks, and continuously evaluate emerging technologies such as Lakebase, Agent Bricks, and MLflow.
Requirements
Professional Experience: At least 10 years of experience in data engineering or a related specialized technology field, including at least three years of team leadership or management experience.
Education: A bachelor's degree in Computer Science, Information Systems, or a related discipline, or equivalent professional experience.
Platform Modernization: Demonstrated experience leading legacy-to-cloud data platform migrations, managing enterprise data integrations, and working with both vendor-provided SaaS solutions and internally developed systems.
Databricks Expertise: Strong hands-on knowledge of Databricks, including Unity Catalog, Delta Lake, medallion architecture, and Databricks Asset Bundles or YAML-based configurations. Experience with Lakebase is an advantage.
Data Engineering and Modeling: Advanced SQL and PySpark skills, experience working with large governed datasets, and a strong understanding of dimensional modeling, semantic layers, domain-driven design, and data lifecycle management.
Data Quality and Analytics: Experience building certified datasets and semantic models for self-service analytics or AI-assisted natural-language querying. Familiarity with data quality frameworks, automated testing, reconciliation, and historical data management is essential.
Engineering Standards and Automation: Working knowledge of CI/CD practices for data pipelines, version control, governed deployment environments, and automated testing. Experience developing internal platforms or self-service infrastructure for distributed engineering teams is highly desirable.
AI and Automation Skills: Practical experience using tools such as Claude, Copilot, ChatGPT, or Gemini to improve engineering productivity and decision-making. Understanding of AI output validation, responsible AI principles, and AI governance is required, along with the ability to coach teams on effective adoption.
Agentic Development Practices: Experience designing or governing AI-assisted software development workflows, including specification-driven development, autonomous coding agents, automated code review, and testing pipelines.
Security and Compliance: Experience collaborating with Information Security and applying data privacy, access control, and regulatory requirements. Familiarity with insurance regulations, including NAIC guidelines and state insurance requirements, is beneficial.
Project and Vendor Management: Experience planning and prioritizing technical initiatives, managing resources and budgets, coordinating external vendors, and taking ownership of technical decisions and delivery outcomes.
Communication and Leadership: Strong analytical, organizational, and problem-solving skills, with the ability to translate complex technical concepts into business-friendly language. Demonstrated ability to influence senior stakeholders, hold teams accountable, challenge existing approaches constructively, and raise risks early.
Preferred Technical Experience: Familiarity with AWS-based data platforms, IBM DataStage, PostgreSQL, Oracle source systems, Lakebase, Agent Bricks, or MLflow is an advantage.
Additional Qualifications: Experience in insurance or another regulated industry, formal mentoring and coaching, and knowledge of agile methodologies, project management tools, Atlassian products, Lucidchart, and Microsoft Office are beneficial.
Work Environment: Candidates must currently reside in the United States and be authorized to work without sponsorship. The position is fully remote and requires a suitable private, quiet workspace. Occasional travel may be necessary for meetings or other business activities.
Benefits
Competitive Salary: Annual salary range of $120,000–$186,000, depending on qualifications, experience, and other relevant factors.
Comprehensive Benefits Package: Access to a range of employee benefits. Specific coverage and plan details are available through the employer's benefits information.
Remote Flexibility: Work remotely from within the United States.
Career Development: Lead a major enterprise data modernization initiative while expanding your expertise in cloud data platforms, AI-assisted engineering, and enterprise architecture.
Leadership Opportunities: Build and develop a data engineering team, shape technical strategy, and influence engineering standards across distributed teams.
Innovation and Impact: Work with modern data technologies and emerging AI capabilities to improve platform performance, scalability, governance, and delivery efficiency.
Collaborative Environment: Partner with senior technology leaders, enterprise architects, analytics specialists, security teams, and external technology providers.
Meaningful Business Contribution: Help strengthen data capabilities that support reliable operations and better decision-making in a regulated industry.