Databricks Engineer

Qualifications

• 5+ years of experience in data engineering, analytics engineering, or a similar role

• Expertise in Databricks, including Delta Lake, Spark, and PySpark

• Strong SQL and ETL pipeline development skills

• Experience designing canonical data models and enterprise-wide data structures

• Hands-on experience with CI/CD processes, version control (Git), and deployment automation

• Solid understanding of data governance, lineage tracking, and data cataloging tools

• Proven ability to work autonomously and direct the work of a second engineer

• Excellent communication skills to interact with business and technical stakeholders

• Be a leader of an energetic team of highly dynamic and talented individuals

Responsibilities

• Architect & Design: Develop canonical data models that enable efficient and scalable enterprise data consumption

• Data Transformation: Build, maintain, and optimize data pipelines that transform raw data into structured datasets within Databricks

• Leadership & Execution: Work independently to execute tasks while providing technical guidance to a second engineer

• Data Governance: Implement and oversee data quality, lineage tracking, and data cataloging best practices

• CI/CD & Automation: Ensure all code is version-controlled and integrated into a CI/CD pipeline for deployment and maintenance

• Stakeholder Collaboration: Engage with business stakeholders to understand requirements and translate them into scalable technical solutions

• Performance Optimization: Monitor and improve data processing efficiency, ensuring high availability and reliability

Full Description

About Us:

Quantiphi is an award-winning Applied AI and Big Data software and services company, driven by a deep desire to solve transformational problems at the heart of businesses.

Our signature approach combines groundbreaking machine-learning research with disciplined cloud and data-engineering practices to create breakthrough impact at unprecedented speed.

Company Highlights:

• Delivered 2.5x growth YoY since its inception in 2013

• Headquartered in Boston, with 4000+ Quantiphi professionals across the globe

• Great Places to Work certified for 2 consecutive years- 2022, 2021

• Recognized by Everest Group as Specialist Leader and Star Performer in Analytics and AI Services, 2022

• Recognized as an AIFinTech100 Company, 2022 by InsurTech

• Winner of Best in Business Award in Established Business category by INC., 2022

• Winner of Competitive Strategy Leadership Award in Artificial Intelligence Services in Healthcare by Frost & Sullivan, 2022

• Recognized in Gartner Hype Cycle Reports for AI Strategy, 2022

• Winner of 2021 Google Cloud Breakthrough Partner of the Year- North America

• Winner of 2021 AWS Canada Rising Star of the Year

• Recognized as Leader in IDC MarketScape: WorldWide AI IT Services, 2021

• Recognized in the Fast Company 2021 World Changing Ideas- AI and Data category

• Winner of NVIDIA's Americas Service Delivery Partner of the Year, 2021

Role & Responsibilities:

• Architect & Design: Develop canonical data models that enable efficient and scalable enterprise data consumption.

• Data Transformation: Build, maintain, and optimize data pipelines that transform raw data into structured datasets within Databricks.

• Leadership & Execution: Work independently to execute tasks while providing technical guidance to a second engineer.

• Data Governance: Implement and oversee data quality, lineage tracking, and data cataloging best practices.

• CI/CD & Automation: Ensure all code is version-controlled and integrated into a CI/CD pipeline for deployment and maintenance.

• Stakeholder Collaboration: Engage with business stakeholders to understand requirements and translate them into scalable technical solutions.

• Performance Optimization: Monitor and improve data processing efficiency, ensuring high availability and reliability.

Required Skills:

• 5+ years of experience in data engineering, analytics engineering, or a similar role.

• Expertise in Databricks, including Delta Lake, Spark, and PySpark.

• Strong SQL and ETL pipeline development skills.

• Experience designing canonical data models and enterprise-wide data structures.

• Hands-on experience with CI/CD processes, version control (Git), and deployment automation.

• Solid understanding of data governance, lineage tracking, and data cataloging tools.

• Proven ability to work autonomously and direct the work of a second engineer.

• Excellent communication skills to interact with business and technical stakeholders.

What is in it for you:

• Be part of the fastest-growing AI-first digital transformation and engineering company in the world

• Be a leader of an energetic team of highly dynamic and talented individuals

• Exposure to working with fortune 500 companies and innovative market disruptors

• Exposure to the latest technologies related to artificial intelligence and machine learning, data and cloud