Data Engineer

  • 39190
  • Non-Life - Data Engineer
  • |
  • Ontario, Canada
  • |
  • Feb 28, 2021
Insurance
RESPONSIBILITIES
  • Design, implement, and maintain large-scale batch and real-time scalable data pipelines with complex data transformations
  • Perform data wrangling to transform and map data from raw data forms into formats more appropriate and valuable for analytics
  • Write and optimize complex queries on large data sets
  • Assemble large, complex data sets that meet functional / non-functional business requirements
  • Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs
  • Experience working with business teams to translate functional requirements into technical requirements
  • Conduct business and functional requirements gathering and provide projects estimates
  • Experience supporting and working with cross-functional teams in a dynamic environment
  • Develops workflows and tools that automate data loading processes and help ensure data quality and integrity
QUALIFICATIONS
  • A bachelor’s degree in computer science, Software Engineering or equivalent
  • Hands-on experience with leading commercial Cloud platforms, including AWS and Azure
  • Excellent SQL, Python and PySpark skills
  • Strong experience with Apache Spark(Databricks or similar), Kafka and NiFi
  • Hands on experience with Hadoop distributions (Cloudera, Hortonworks) and Hive
  • Strong experience with relational SQL and NoSQL databases like PostgreSQL, Oracle, Cassandra, Mongo DB
  • Hands on experience with Snowflake or Redshift, or similar cloud storage technologies
  • Experience with ElasticSearch and Kibana is a big plus
  • Experience with Alation and/or other data catalog tools (Azure, Informatica) is a big plus
  • Experience with Kubernetes, Docker is a big plus
  • Experience with Agile development methodologies
  • Strong presentation, facilitation, verbal and written communication skills, including interpersonal skills