Senior Big Data Engineer will execute true end-to-end development and testing of software products in an agile lifecycle. Partnering with key technical and business stakeholders, Senior Big Data Engineer will help refine the product vision, estimate and design a solution, create, purchase, or customize the requisite hardware and software to create the solution, ensure the quality of the final product through manual and automated tests, and support the new software as it is moved to production.
• Design and build large scale data processing system (real-time and batch) to address growing AI/ML and Data needs of a Fortune 500 company
• Build a product to process large amount data/events for AI/ML and Data consumption
• Automate test coverage (90+%) for data pipelines. Best practices and frameworks for unit, functional and integration tests.
• Automate CI and deployment processes and best practices for the production data pipelines.
• Build AI/ML model based alert mechanism and anomaly detection system for the product. The goal is have a self-annealing product
• Bachelor's degree or equivalent work experience.
• 5+ years of relevant experience.
• 10+ years of overall experience in software development with 5-6 years of relevant experience in designing, developing, deploying and operating large data processing data pipelines at scale.
• 3-4 years' experience with Apache Spark Streaming and Apache Kafka
• Strong background in programming (Scala Must, good to have knowledge on Java, Python)
• Experience on building reusable data frameworks/modules
• Experience on Airflow scheduler
• Experience with Containers, Kubernetes and scaling elastically
• Strong background in algorithms and data structures
• Strong analytical and problem solving skills
• Strong bent towards engineering solutions which increase productivity of data consumers
• Strong bent toward completely automated code deployment/testing (DevOps, CI/CD)
• Passion for data engineering and for enabling others by making their data easier to access.
• Some experience with working with and operating workflow or orchestration frameworks, including open source tools like Activiti, Spring Boot, Airflow and Luigi or commercial enterprise tools.
• Excellent communication (writing, conversation, presentation) skills, consensus builder
Required Skills Summary: Apache Spark, Apache Kafka, Scala/Java/Python, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker Containers
Preferred Skills / Experience
• Knowledge of ReactJS, GraphQL and API Development
• Proficiency in working with Kafka
• Knowledge front end, back end services, or full stack engineering
• Demonstrated ability to tackle tough coding challenges independently and work closely with others on a highly productive coding team
• Continuously develop and acquire new technical skills
• Apache Flink experience
Apply on company website