Lead Data Engineer
Key Responsibilities
- Design and solution end-to-end data architecture for data hubs/data products, all the way from source systems to consumption.
- Oversee the design and management of data solutions to ensure data is stored, processed, curated, and utilized effectively.
- Own and build a reusable data pipeline utilizing Big Data and Azure/Databricks for Eligibility Program data products.
- Ingesting huge volumes of data from various platforms for Analytics needs and writing high-performance, reliable, and maintainable ETL code.
- Leadership: Lead and mentor a team of data engineers, ensuring the efficient flow of data within the organization with the defined processes and tools.
- Collect, store, process, and analyze large datasets to build and implement extract, transfer, load (ETL) processes.
- Develop reusable frameworks to reduce the development effort involved, thereby ensuring cost savings for the projects.
- Utilizing CI/CD Pipelines: Utilize and enhance CI/CD practices to automate the delivery of data solutions, ensuring reliability and scalability based on the defined tools.
- Utilize Cloud technologies (Azure Databricks) to enable data product solutions.
- Develop quality code through performance optimizations in place right at the development stage.
- Appetite to learn new technologies and be ready to work on new cutting-edge cloud technologies.
- Partner with Tech, Business, BI, and Data Science teams to create reusable data products.
- Work with a team spread across the globe in driving the delivery of projects and recommend development and performance improvements.
- Track and report on KPIs for solution delivery and data quality.
- Communicate and present use cases, solutions, and impact to business stakeholders and mid/senior management.
- Optimize reusable frameworks, Spark jobs for performance and cost efficiency in large-scale environments.
- Ability to interact with business stakeholders in getting the requirements and implementing solutions.
- Analyze the data in depth, using SQLs and other exploratory tools against various platforms such as Bigdata, Oracle, SQL Server, Databricks and others.
- Work with IT, business, and architects to develop and design requirements to formulate technical design.
Required Skills
Essential Business Experience and Technical Skills:
- 8+ years of Data Solutions, development, and delivery experience with 4+ years of recent experience in Azure/Databricks environments.
- Proficiency and extensive experience with SQL, Spark &/or Scala/Python and performance tuning.
- Hands-on expertise in: Big Data (ex: Hive and HBase), Azure Databricks, Azure Functions, Cosmos DB and/or Data Factory experience is a MUST.
- Strong experience in building/designing Data warehouses, data stores for analytics consumption on Cloud (real time as well as batch use cases).
- Design, build, and deploy robust data ingestion and curation pipelines utilizing cloud-based data platforms such as Azure Data Factory, Apache Spark (Scala or Python), Azure Databricks, and Delta Lake.
- Good scripting experience, primarily on shell/bash/ PowerShell.
- Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
- Experience and familiarity implementing data governance and data quality using the enterprise toolset.
- Skilled in drafting functional, technical requirements, creating high-level design documents, and data-flow diagrams, etc.
- Expertise in writing validation scripts to validate the data, data integrations and ETL transformations.
- Very good problem solver and excellent communication skills – both written and verbal.
