Data Engineer
Data Engineer
The Data Engineer will be responsible for building the data pipeline, storage patterns, and access methods for the Commerce Data used across the entire company. This will involve coding in several languages, using streaming technologies, working with multiple data storage platforms, and delivering information via multiple access patterns such as at rest for analytics and programmatically through APIs.
What you’ll do:
· Design and implement large scale real-time & batch data pipelines on the AWS platform. Resolve problems and roadblocks as they occur and unblock team members. Follow through on details and drive issues to closure
· Prototype creative solutions quickly by developing minimum viable products and work with seniors and peers in crafting and implementing the technical vision of the team
· Ensure robust testing and performance optimization
Who you are:
· Bachelor’s or Master’s degree in Computer Science or Engineering or related field; or equivalent related professional experience
· 5+ years developing data and software solutions including ETLs
· 3+ years of experience in data steaming including Kafka
· 3+ years of experience in building data pipelines in the cloud
· Experience in Cloud computing platforms including AWS, EMR, or Kubernetes/Docker.
· Experience in Apache Spark, Apache Kafka, Splunk, DataDog, Jenkins
· Experience in NoSQL and SQL Databases including MongoDB and MySQL
· Experience with aggregation strategy and performance optimization
· Experience building data pipelines incrementally and ensure test cases and logging are built for all scenarios
· Experience in Java
· Agile methodologies and tools including Jira, Trello, or Scrum practices.
· Experience in Spring Frameworks
· Strong communication skills, experience communicating across various groups, and levels of leadership within a global organization
· Enthusiastically seek out solutions for data engineering problems
· Focused developer with a strong sense of ownership
· Ability to drive individual tasks to completion and production deployment
· Capable of working closely with team members to ensure data solutions are well built and of high quality