The Senior Lead Data Engineer will architect and maintain ELT pipelines, manage data infrastructure, optimize SQL queries, and mentor teammates.
Senior Lead Data Engineer
Primary Skills
- Architect, construct, test and maintain ELT pipelines using Hadoop and Spark SQL
Specialization
- Hadoop and Spark SQL
Job requirements
- Key Responsibilities:
- Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL
- - Manage and optimize the infrastructure ensuring reliability and performance
- -Design and implement data models and schemas for enterprise data warehouse and data marts
- -Write and maintain scripts in Python for various applications
- -Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases
- - Work closely with analysts and product managers to understand data requirements and deliver high-quality data
- -Provide technical guidance and mentorship to other team members fostering best practices in data engineering
- -Monitor system performance, identify bottlenecks, implement solutions to ensure high data availability
- -High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing
- -Good knowledge in Python for data manipulation and scripting
- -Mastery of advanced SQL and deep experience working with relational databases and datawarehouses
- -Hands-on experience with workflow management tools like Apache Airflow
- -Familiarity with Unix shell scripting for automation and system management
Required Skills:
Top Skills
Apache Airflow
Hadoop
Python
Spark Sql
SQL
Unix
Similar Jobs
Fintech
The Senior Data Engineer will maintain and optimize data systems, develop ETL workflows, and ensure platform stability while collaborating across teams to support data operations.
Top Skills:
AWSInformaticaOracleSQL ServerSsisSsrsTableau
Cloud • Fintech • Information Technology • Machine Learning • Software • App development • Generative AI
The Staff Software Engineer will design and build connectors for various systems, lead technical discussions, mentor engineers, and ensure high standards in software quality while delivering scalable backend services.
Top Skills:
Api GatewayAzureC#C++Cloud (AwsData IntegrationEtl ToolsEvent BusGcp)Identity ManagementJavaKafkaMicroservicesNo-SqlRabbitMQRestful ApisSQL
Aerospace • Cloud • Digital Media • Information Technology • Mobile • News + Entertainment • Generative AI
The Program Manager will enhance engagement and competition by implementing sales strategies, managing project execution, and coordinating communication across departments for Gen Mobile's sales team.
What you need to know about the Manchester Tech Scene
Home to a £5 billion digital ecosystem, including MediaCity, which consists of major players like the BBC, ITV and Ericsson, Manchester is one of the U.K.'s top digital tech hubs, at the forefront of advancements in film, television and emerging sectors like as e-sports, while also fostering a community of professionals dedicated to pushing creative and technological boundaries.
