Data Engineer

  • 39193
  • Non-Life - Data Science
  • |
  • Ontario, Canada
  • |
  • Feb 28, 2021
Insurance
RESPONSIBILITIES
  • Build and optimize robust data pipelines between different systems with the Hadoop platform to produce high quality data structures
  • Develop and apply professional experience with Spark, big data environment and best in class expertise/training
  • Increase automation and scaling of complex data sets based on the customer’s analytic use cases, such as structured data delivery for business analysis
QUALIFICATIONS
  • 2+ years of experience with data engineering
  • University degree in Software Engineering, Core Computer Science or Computer Engineering
  • Experience creating and/or maintaining data pipelines, from batch job to real time implementations
  • Must have excellent communication and organizational skills
  • 3+ years of experience with Scala or Java programming
  • 1+ years of experience working with big data technologies such as Databricks, Redshift, Athena, AWS or ETC
  • 1+ years of experience with MapReduce and Spark
  • 2+ years of experience with Airflow, Kafka
  • Knowledge of Hadoop, Git is a plus