Hadoop Engineer Job Description
Hadoop engineer
provides cloudera Distribution Hadoop (CDH) support to implementation, maintenance, and sustainment of CDH stack to include but not limited to: Solr, Nifi, Kafka, HDFS, Impala, Yarn, CDSW, Hive, Hue, Flume, Spark, Sqoop, Oozie, and ZooKeeper.
Hadoop Engineer Duties & Responsibilities
To write an effective hadoop engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included hadoop engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Competent with Hive table creation, loading, and querying newer technologies such as Spark and Jethro and be able to ingest data into Hadoop to multiple areas within the ecosystem such as HDFS
Ensure quality of the technology solution delivered, stability, scalability, availability, performance, cost, security
Automate operation, installation and monitoring of Hadoop ecosystem components in our open source infrastructure stack
Evaluate Hadoop infrastructure requirements and design/deploy solutions (high availability, big data clusters, elastic load tolerance)
Develop automation, installation and monitoring of Hadoop ecosystem components in our open source infrastructure stack
Performance Engineer will be responsible for analyzing existing Hadoop systems/Applications, making recommendations to improve performance, and possible implementing those recommendations
Responsibilities will also include analyzing planned usage patterns for a Hadoop cluster and developing a set of hardware specifications to support the usage
Assist client support team with Performance related issue resolution
Data Ingestion Engineer will be responsible for gathering requirements to load data into Hadoop, design a solution to meet these requirements, determine the effort to implement the solution, and lead a team to implement the solution
Assist client support team with Data Ingestion issue resolution
Hadoop Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Hadoop Engineer
List any licenses or certifications required by the position: SSL, MS, PMI, AWS, V3, ITIL, PMP, X.509, HDPCA, HDP
Education for Hadoop Engineer
Typically a job would require a certain level of education.
Employers hiring for the hadoop engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Technical, Engineering, Information Technology, Information Systems, Business, Computer Engineering, Education, Software Engineering, Statistics
Skills for Hadoop Engineer
Desired skills for hadoop engineer include:
Kafka
Hive
Spark
HBase
Oozie
Flume
Best practices related to security
Chef
Particularly Hadoop Security using Kerberos and Active Directory
Desired experience for hadoop engineer includes:
3+ years of experience with Cloudera Hadoop implementations a major plus
5+ years of experience with large complex data implementations
5+ years of experience on ETL and SQL
Working knowledge of Big Data/Analytical technologies, Cloudera Hadoop, MapReduce, MongoDB
Demonstrated experience in Data management tools
4+ years with UNIX operating system that includes customization, provisioning, and system management
Hadoop Engineer Examples
1
Hadoop Engineer Job Description
Job Description Example
Our growing company is looking for a hadoop engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for hadoop engineer
- Solid knowledge of ETL based technologies abinitio or talend
- As the Hadoop Engineer you will assist in the design of, and develops integrated solutions in accordance with, corporate standards and products
- Provides basic technical support to customers by handling systems-related issues
- Support the problem resolution of any technical issue they may raise
- Support the recommend changes to enhance the stability and performance of the relevant areas
- Contribute in medium internal and smaller external projects
- Support training/assistance to the Department Staff or other Departments
- Participate in the deployment automation and full industrialization of current and future catalog components
- Support application monitoring data system handling the reporting built in Platfora (existing) working on the new architecture for migration
- Help drive the optimization, testing and tooling to improve data quality
Qualifications for hadoop engineer
- Must demonstrate an understanding of core computer science concepts and demonstrate specific practical applications of those concepts
- 2+ years of Devops or System Administration experience (be conversant in Unix networking and C system calls)
- Experience writing software in a continuous build and automated deployment environment
- Team player with strong inter-personal and problem solving skills
- Java, Perl, PHP, and Python experience
- Experience with SQL (MySQL, MSSQL, Redshift, etc)
2
Hadoop Engineer Job Description
Job Description Example
Our innovative and growing company is searching for experienced candidates for the position of hadoop engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for hadoop engineer
- Develop measurement approaches to evaluate performance continuously
- Collaborate with software engineers, ML experts, and others, taking learning and leadership opportunities that will arise every single day
- Work in cross functional agile teams to regularly experiment, iterate, and deliver on new product objectives
- Define, develop, document and maintain a repository of test plans, cases and scripts
- Develop and review test requirements with input from product owners and developers
- Develop automated and manual testing procedures
- Report defects and interact with developers to ensure these are understood
- Contribute to ongoing improvement of quality assurance standards and procedures
- Responsible for that day to day operation and support of Hadoop and CICD environments
- Collaborate with Corporate IT function around integrating Hadoop ecosystem(s) with critical enterprise systems
Qualifications for hadoop engineer
- Coding experience with Scala
- Experience with Angular and D3.js will be a bonus
- BSc or MSc in Computer Science or related technical field
- This position requires deep data science and data engineering level knowledge and experience with building and deploying systems leveraging hadoop and database technologies
- RDBMS related product experience is strongly preferred (Teradata, Netezza, Vertica, Oracle, DB2, Postgres)
- Experience with cloud services like AWS and with container technologies such as Docker
3
Hadoop Engineer Job Description
Job Description Example
Our innovative and growing company is looking to fill the role of hadoop engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for hadoop engineer
- Administration of various Linux distros
- Git version control system knowledge
- Various Atlassian products (Jira, Bitbucket, Confluence)
- Analyzes, designs, creates and implements Big Data infrastructures, including access methods, device allocations, validation checks, organization and security
- Designs data models, logical and physical infrastructure designs
- Assists in system planning, scheduling, and implementation
- Initiates corrective actions to stay on schedule
- Installs, upgrades, and tests complex big data deployments
- Develops and implements recovery plans and procedures
- Coordinate with different technology teams and business stake holders for efficient problem analysis and resolution
Qualifications for hadoop engineer
- 2+ years building, scaling and supporting Hadoop clusters
- 4+ years building and maintaining distributed application platforms
- Automation with tools like Chef, Puppet or Ansible
- Experience/knowledge utilizing NoSQL technologies such as Hbase
- Good understanding of HDFS, Hive, map/reduce
- Knowledge of workflow/schedulers
4
Hadoop Engineer Job Description
Job Description Example
Our growing company is looking for a hadoop engineer. If you are looking for an exciting place to work, please take a look at the list of qualifications below.
Responsibilities for hadoop engineer
- Responsible for analyzing software issues pertaining to features and reporting content to determine appropriate resolution needed
- Plan, test, and execute upgrades involving Hadoop components
- Work collaboratively with business management and staff to deliver technical solutions that meet customer needs
- Work effectively with customers in all phases of application development projects, potentially including requirements gathering, design, testing and change management
- Provide input on strategies and tactics regarding platforms, tools and technical staffing to meet customer needs
- Follow and/or support prescribed project management guidelines
- May provide guidance and direction to other staff and may function as a team leader
- Configure and automatically deploy new services as a part of our large Big Data Analysis System
- Automate routine DevOps tasks for distributed cluster maintenance and troubleshooting
- Conduct systems performance analysis
Qualifications for hadoop engineer
- Proficiency in of Java, Python, SQL, Scala
- 2+ years writing Perl scripts for various task automation
- Located in San Diego, CA
- Lead or provide expertise in defining and managing detailed system design for data integration processes or data marts/ warehouse builds or extensions
- Interface with the architecture group to determine the overall solution and ensure conformance to established data management standards
- Work closely with various technology teams, business systems analysis, development and project teams in the creation of detailed system requirements, specification documents related to Data Management initiatives and resolve information flow and content issues
5
Hadoop Engineer Job Description
Job Description Example
Our growing company is looking to fill the role of hadoop engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for hadoop engineer
- Contribute to monitoring and reporting system
- Provide on-call operations support
- Act as a point of contact with our vendors
- Design, build, install, configure and support Hadoop-based applications
- Create scalable and high-performance rest-based APIs
- Optimize queries for high-speed, low-latency results
- Manage, deploy, and administer Hadoop-based components within the development environment
- Document and support handover to operational/production teams
- Maintain and propose best practices/standards
- Other duties as assigned by Scrum Master or Technical Lead (5%)
Qualifications for hadoop engineer
- Coordinate with various infrastructure teams for assessing impacts of projects on storage, network, database infrastructure
- Manage and coordinate efforts of data modelers, data warehouse/data integration developers, and Extract, Transform and Load (ETL) data management process in database usage and data warehousing
- Ensure metadata and data lineage is captured and compatible with enterprise metadata and data management tools and processes
- Provide data management expertise t to the development and testing teams to resolve data issues
- Oversee that requirements regarding privacy and security of data, data governance and access controls are appropriately captured
- Create development estimates for various phases of project requests and responsible for reviewing and approving business / system specification requirements, authoring system design specifications and system deployment plans