Big Data-Hadoop Job Description
Big data-hadoop
provides expertise in big data technologies in a Hadoop ecosystem including Hive, HDFS, MapReduce, Yarn, Kafka, Pig, Oozie, HBase, Python, R, Sqoop, Spark, and cloud components such as Google Cloud Platform, AWS or Azure.
Big Data-Hadoop Duties & Responsibilities
To write an effective big data-hadoop job description, begin by listing detailed duties, responsibilities and expectations. We have included big data-hadoop job description templates that you can modify and use.
Sample responsibilities for this position include:
Significant Hadoop ecosystem experience, with a proven ability to make use of key components preferably including Spark
Be developing Hadoop / Spark / MapReduce applications for our customers
Interacts with confidence and ease when interacting with middle- and senior-level managers (client and CSC)
Communicates in formal presentation settings including proposal orals, client workshops and regional internal events
Significant exposure to software development using Java
Demonstrable experience of large scale data warehouse date integration implementations, preferably using Apache Hadoop based technologies
The ability to take visible leadership responsibility in design, operational and team projects
Required skills to deal with ambiguity, manage both in plan and out of plan requirements, and to communicate and manage conflicts and escalations accordingly
The ability to collaborate and communicate with key stakeholders, to lead the resolution of end to end data issues, data design issues and end to end data test issues
The required level of knowledge to understand, navigate and develop with data tools and technologies
Big Data-Hadoop Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Big Data-Hadoop
List any licenses or certifications required by the position: DOD, AWS, SSL, STIBO, HDPCA, TOGAF
Education for Big Data-Hadoop
Typically a job would require a certain level of education.
Employers hiring for the big data-hadoop job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Education, Technology, Information Technology, Business, Information Systems, Mathematics, Computer Engineering
Skills for Big Data-Hadoop
Desired skills for big data-hadoop include:
Spark
HBase
Kafka
Python
Hive
HDFS
MapR FS
MapRDB
Amazon EMR
Application
Desired experience for big data-hadoop includes:
Experience with NoSQL technologies like HBase, MongoDB, Couchbase
Must have a track record of successfully building and maintaining large scale high performance technology products in the industry
Strong hands on experience is expected
Prior experience and knowledge of ad tech domain is a plus
Experience in leading open-source projects is a plus
4+ years experience using SQL and traditional data warehouses
Big Data-Hadoop Examples
1
Big Data-Hadoop Job Description
Job Description Example
Our company is growing rapidly and is hiring for a big data-hadoop. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for big data-hadoop
- Big Data Sr
- Attitude for simple, elegant design
- Build high-performance and reliable data transport mechanisms
- Implement and optimize API-level integrations with Hadoop and Hadoop technologies, or other relevant technologies
- For seamless/effortless/performant orchestration of data flow and data processing
- Leverage a variety of open source technologies to achieve an optimally flexible architecture, and provide guidance to ensure customer success on best-of-breed deployments
- Define and publish best practices
- Performs as subject matter advisor and may serve as point of escalation and become actively involved, as required, to meet schedules and resolve problems highly complex in nature
- Establishes credibility with partners while managing expectations
- May develop budget for functional area
Qualifications for big data-hadoop
- Bachelor's degree in a technical field such as Computer Science, or equivalent work experience, -5+
- Interpret Hadoop service and role logs (HDFS)
- Proficiency across the full range of database and business intelligence tools
- Linux command line proficiency, bash shell scripting, proficiency in one of Java or Python
- Working knowledge of common Hadoop tools including Flume, Sqoop, Oozie, PIG, MapReduce, HDFS, and YARN
- Hive and Hbase databases, Parquet and Avro file formats
2
Big Data-Hadoop Job Description
Job Description Example
Our company is growing rapidly and is searching for experienced candidates for the position of big data-hadoop. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for big data-hadoop
- Participating in full lifecycle implementation of the technical components of solutions, including tool selection, data architecture, database administration, ETL (extract, transform and load), reporting and integration
- Troubleshoot issues with Hadoop
- Based on functional specifications, develop test plans, and test cases for functional, regression, and system level test activities to ensure the accuracy and stability of the software applications
- Take ownership for the manual, automation, performance and security testing for our web applications
- Design and build automation frameworks to automate as much of the testing as possible for the web applications
- Prepare test documentation based on requirements
- Track and analyze test results, create report, draw conclusion, and make necessary recommendations for improvements
- Complete full system regression testing for releases
- Collaborate with development teams to understand how development changes affect ongoing Automation efforts
- Identify the components of the system to be automated based on business priority and benefits of automated testing
Qualifications for big data-hadoop
- Spark Streaming, Storm & Kafka
- At least 2 years exp in Hadoop, MapReduce, Pig, Hive, Oozie and HCatalog
- At least 2 years exp in HBase, Cassandra or MongoDB or any other NoSQL DB
- At least 5 years exp in programming in either core Java or C or C++ or Scala
- At least 2 years exp in scripting in either Python or Perl or Shell scripting
- At least 3 years exp in Representational State Transfer (REST) Application Program Interface (API) and Java Messaging Service (JMS) and WebServices
3
Big Data-Hadoop Job Description
Job Description Example
Our innovative and growing company is looking for a big data-hadoop. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for big data-hadoop
- Extensive usage and experience with Data Frames with in Spark
- HBase implementation for low latency requirements
- Experience in full lifecycle application development, including requirements gathering, design, coding, testing, QA and documentation and production support
- Write and review QA documentation
- Provide support for large scale Hadoop clusters across production and development environments
- Focus on automation to improve availability and scalability in our production environments
- Help architect existing and new services in our production environments
- Manage replication links between clusters to maintain high availability
- Work with SQL Server and Big Data platforms such as Splunk and Hadoop, for a variety of projects
- Handle development, optimization, testing, migration, security, tracing, logical/physical design and maintenance
Qualifications for big data-hadoop
- At least 4 years exp in JUnit and JMeter and MRUnit or other testing tools
- At least 3 years exp in distributed computing design patterns, algorithms, data structures, and security standards
- Basic knowledge in Kafka, Spark, and Storm
- At least 3 years exp in demonstrated effective verbal and written communication skills
- 5 + years of application development experience in Data warehouse environment in any RDBMS/ appliance data base environment
- 3 + years of application development experience in Java/ J2EE
4
Big Data-Hadoop Job Description
Job Description Example
Our innovative and growing company is looking for a big data-hadoop. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for big data-hadoop
- Participate in the 24×7 on-call rotation schedule, when ready, in order to support the data platforms that support the clinical staff in the enterprise and are available 24×7
- Implement ETL processes and document ETL processes
- Define data retention policies and assist in enforcement
- Design and manage the migration of data from legacy systems to new data solutions
- Monitor and analyze information and data systems and evaluate their performance to discover ways of enhancing them (new technologies, upgrades )
- Utilizes existing methods and procedures to create designs within the proposed solution to solve business problems
- Contributes to design of solution, executes development of design, and seeks guidance on complex technical challenges where necessary
- Understands methods and procedures on new assignments and executes deliverables with guidance as needed
- Develop new applications and data flows based on functional requirements
- Unit testing and debugging
Qualifications for big data-hadoop
- Understanding of Business Intelligence and Data Warehousing principles including staging source data, master data management and conformed dimensions, data stewardship, and consumption data models
- Experience working with large data sets, experience working with distributed computing a plus (Map/Reduce, Hadoop, Hive, Apache Spark)
- Programming experience in Java, Scala and Python
- Ideally familiar with the concepts of Hadoop and HANA Vora, Spark, HDFS, Java Script
- Must be proficient in Java development
- Must have exposure and knowledge of messaging frameworks
5
Big Data-Hadoop Job Description
Job Description Example
Our company is looking to fill the role of big data-hadoop. We appreciate you taking the time to review the list of qualifications and to apply for the position. If you don’t fill all of the qualifications, you may still be considered depending on your level of experience.
Responsibilities for big data-hadoop
- Hands-on development or testing experience in Java, Scala, Python and Big Data technologies required
- Lead small teams of developers and coordinate development activities
- Mentor junior members of the team in software development best practices
- Loading from disparate data sets in to Hadoop Data Lake
- Analyzing Big Data Analytic technologies and applications in both business intelligence analysis and new service
- Offerings, adopting and implementing these insights and standard methodologies
- 3+ years’ experience working with SQL and relational database structures
- Programming experience in VBA or other programming languages
- Write SQL codes
- Ability to work well with other team members to solve problems
Qualifications for big data-hadoop
- Exposure and or experience with Rules based frameworks
- Experience developing Map Reduce/YARN applications
- Experience and or exposure to Spark, Scala is a big plus
- Experience with Hbase, HIVE and/or Impala is a big plus
- SQL application development experience
- Very strong analytical and problem solving skills, great communication and team player skills