Job Description

Job Summary:

We are seeking an experienced Data Engineer Lead to design, optimize, and deliver scalable, enterprise-grade data solutions across analytics, BI, and AI/ML workloads. This role requires advanced expertise in Databricks, Azure, Python, SQL, and Apache Spark, along with strong CI/CD, DevOps, and data governance practices. The ideal candidate will lead development efforts, architect cloud-native data platforms, and mentor junior engineers.

Key Responsibilities:

• Design, build, and optimize scalable data pipelines for analytics, AI/ML models, and business intelligence workloads.
• Architect and implement ETL/ELT pipelines for multi-source structured, semi-structured, and streaming data.
• Develop enterprise-level data solutions using Azure & Databricks.
• Lead production-grade pipeline orchestration using Databricks Workflows.
• Utilize Apache Spark / PySpark for distributed data processing and real-time analytics.
• Implement solutions leveraging Azure Cloud services including Azure Data Factory, Azure Data Lake, Azure Synapse Analytics, Azure Key Vault, Azure Monitor.
• Write optimized and high-quality Python & SQL code for ingestion, transformation, analytics, and ML workflows.
• Implement CI/CD automation using Terraform, GitHub Actions, and Azure DevOps.
• Build and consume RESTful APIs, including OAuth-based authentication and JSON-based integrations.
• Implement and enforce data governance frameworks including data quality, cataloging, lineage, and compliance standards.
• Ensure security and compliance across cloud data platforms.
• Build and maintain batch & streaming pipelines using Structured Streaming, Airflow, and Kafka.
• Ensure high availability, scalability, and reliability of data pipelines.
• Lead technical design discussions and code reviews.
• Mentor junior data engineers and enforce industry best practices.

Required Skills & Experience:

• 12+ years of IT experience with at least 5+ years in modern cloud data engineering.
• Strong experience in: Databricks (Workflows, Delta Lake, Spark), Azure Cloud ecosystem, Python & SQL, Apache Spark / PySpark, CI/CD (Terraform, GitHub Actions, Azure DevOps), Data governance, quality, lineage & cataloging, ETL/ELT design and orchestration, RESTful API integration & OAuth authentication.
• Experience leading teams or mentoring engineers.

Preferred Qualifications:

• Experience with real-time analytics and large-scale distributed systems.
• Knowledge of ML pipelines and MLOps concepts.
• Strong analytical, communication, and problem-solving skills.

Job Tags

Similar Jobs