This job has expired, please see additional jobs below
Senior Infrastructure Engineer - Data & Analytics Platform Team
Bloomberg
New York, NY, United States
Job Details - this job has expired, please see similar jobs below
Senior Hadoop Infrastructure Engineer - Data & Analytics Platform Team
Many of the most prevalent applications in Bloomberg need real-time data analysis and business intelligence. Instead of using off-the-shelf applications, we use Hadoop and its ecosystem to provide a large-scale data platform with low-latency SLAs and large storage capabilities. This platform has revolutionized the way we manage and analyze data in a distributed environment.
We design software and hardware systems to support low-latency/high volume requests, security, fault tolerance/high availability and easy customization. Our Hadoop Infrastructure Platform is built to fully automate deployment and operations using Chef; developed and open sourced at https://github.com/bloomberg/chef-bach. With hundreds of applications depending on our platform, we are looking to grow our Hadoop Infrastructure team. That's where you come in.
We'll trust you to:
• Evaluate Hadoop projects across the ecosystem and extend and deploy them to exacting standards (high availability, big data clusters, elastic load tolerance)
• Develop automation, installation and monitoring of Hadoop ecosystem components in our open source infrastructure stack, specifically HBase, HDFS, Map/Reduce, Yarn, Oozie, Pig, Hive, Tez, Spark and Kafka
• Dig deep into performance, scalability, capacity and reliability problems to resolve them
• Create application patterns for integrating internal application teams and external vendors into our infrastructure
• Troubleshoot and debug Hadoop ecosystem run-time issues
• Provide developer and operations documentation to educate peer teams
You'll need to have most of the following:
• Experience building out and scaling a Hadoop-based or UNIX-hosted database infrastructure for an enterprise
• 2+ years of experience with Hadoop infrastructure or a strong and diverse background of distributed cluster management and operations experience
• Experience writing software in a continuous build and automated deployment environment
We'd love to see:
• 2+ years of DevOps or System Administration experience using Chef/Puppet/Ansible for system configuration, or quality shell scripting for systems management (error handling, idempotency, configuration management)
• In-depth knowledge of low-level Linux, UNIX networking and C system calls for high performance computing
• Experience with Java, Python or Ruby development (including testing with standard test frameworks and dependency management systems, knowledge of Java garbage collection fundamentals)
• Experience or exposure to the open source community (a well-curated blog, upstream accepted contribution or community presence)