The Role:
We are looking for a savvy Data Engineer to join our growing data and analytics team.
Our ideal candidate is an experienced data pipeline builder and data wrangler who enjoys both optimising existing data systems and building new capabilities from the ground up. They will be working alongside our Data Lead and Data Analysts to formalise and expand our data collection and delivery architecture - ensuring information is accurate, timely, consistent, meaningful, and enabling our teams to build analytics simply on top of robust fundamentals.
They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimising or even re-designing our company's data architecture to support our next generation of products and data initiatives.
What you'll do:
- Create, maintain and monitor an efficient data pipeline architecture, including working with scheduled jobs, replication, monitoring and partitioning.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS - big data- technologies.
- Handle changes in application logic and develop supporting data transformations.
- Ensure data integrity and consistency throughout, and enable teams to easily verify this in end analysis.
- Own database and data flow documentation.
- Develop and maintain a large event stream to enable real-time customer analytics.
- Work with a wide variety of business stakeholders to assist with data-related technical issues and support their data infrastructure needs.
- Create data tools alongside Data Analysts and Data Scientists in a collaborative fashion to support business outcomes
- Work alongside the rest of the Data team to enable best use of our Looker instance, ideally through writing and optimising LookML.
Requirements:
- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL), as well as working familiarity with a variety of databases (PostgreSQL and MySQL preferably).
- Database administration and data performance management, including the details of indices, normalisation, query optimisation and resultant execution plans.
- Building and optimising - big data- data pipelines, architectures and data sets.
- Root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management
- A successful history of manipulating, processing and extracting value from large disconnected datasets.
- Working knowledge of message queuing, stream processing, and highly scalable - big data- data stores
- The successful candidate will have 3+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Informatics, Information Systems or another quantitative field.
Must-haves:
- Cloud Based ETL Tools: Stitch, FiveTran, Matillion.
- Relational SQL and NoSQL databases: MySQL, Postgres, Redshift and DynamoDB
- Data pipeline and workflow management tools: DBT, Dataform, AWS Glue, AWS Data pipeline, Singer.io, Luigi, Airflow, etc.
- Data visualisation tools: Looker (including LookML experience)
- AWS cloud services: EC2, EMR, RDS, Redshift
- Object-oriented/object function scripting languages: Go, PHP, Python, Scala, etc.
Nice to haves:
- Kafka, Hadoop and Spark
- R and Jupyter Notebooks
Benefits:
- Company laptop (Mac or PC - you choose!)
- Choice of wellbeing benefits which include Private Medical Insurance and/or a gym membership
- 20 days holiday plus bank holidays
- Friday drinks
- Team lunches
- Team socials every month
Friday, November 13, 2020
Data Engineer - Python/SQL (3-15 yrs) (Risk Resources India)
