Data Platform Engineer Job Description
Data platform engineer
provides build and development support for tools like Git, Maven, Make, Docker, etc.
Data Platform Engineer Duties & Responsibilities
To write an effective data platform engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included data platform engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Tenaciously keep the Big Data infrastructure (Hadoop and peripheral tools) operational across various environments in datacenters & AWS Cloud
Install and manage Cloudera, Hortonworks, and Open source Hadoop, various tools & tech stack in datacenter & AWS Cloud environments
Experience working on Hortonworks Hadoop Distrbution
Candidates must possess a strong understanding of semantic knowledge base modeling techniques including knowledge store technologies
Design, code, and execute automated testing with some manual testing if automation is not possible
Work closely with multiple development teams on end-to-end testing
Build simulated full production environments for complex testing scenarios
Lift & shift of on premises Big Data applications / tools to AWS Cloud
Identify technical obstacles early and work closely with team to find creative solutions to build prototypes & develop deployment strategies, procedures
Lead delivery of new BI and Visual Analytics Platforms while supporting existing technologies
Data Platform Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Data Platform Engineer
List any licenses or certifications required by the position: AWS, ITIL, II, IAT, CFA, ML, TDWI, MS, SQL, MCITP
Education for Data Platform Engineer
Typically a job would require a certain level of education.
Employers hiring for the data platform engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Mathematics, Computer Engineering, Business, Education, Information Technology, MBA, Science
Skills for Data Platform Engineer
Desired skills for data platform engineer include:
Kafka
Spark
Hadoop
Kubernetes
Source code repository like SVN/GIT & build tools like Maven
Docker
Advocating change where needed for product development
Bank’s data structures and metrics
Modern code development practices
Networking
Desired experience for data platform engineer includes:
7+ years of relevant Database Platform Support experience
Deep understanding of the workings of a Massive Parallel Processing Database System with focus on Teradata and Redshift
Provide preventative maintenance, troubleshooting, and quick resolution of routine problems to ensure Database infrastructure and application stability
Experience with creating data pipelines between various systems using native connectors, sqoop and/or other tools
Experience in building tools to automate or monitor large scale distributed systems
Thrive working in a fast-paced, evolving, growing, and dynamic environment
Data Platform Engineer Examples
1
Data Platform Engineer Job Description
Job Description Example
Our company is hiring for a data platform engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for data platform engineer
- Write and analyze queries, object oriented and/or scripting languages
- Understand big data cloud based data technologies and applications
- Play active role in Communities of Practice and organizing Hackathon(s)
- Design and implement end-to-end engineering solutions for business opportunities
- Tenaciously keep the big data platform operational
- Stand up and configure AWS environment
- Build tools that enable the team to scale, such as self-service console and monitoring
- Install, deploy, and manage tools built by us and the open source community
- Identify technical obstacles early and work closely with the team to find creative solutions
- Ability to grasp new technologies rapidly as needed to progress varied initiatives - Break down data issues and resolve them
Qualifications for data platform engineer
- Experience working with REST-based API's and JSON data formats
- Problem solving large data sets in your sleep
- Passionate for building highly scalable consumer facing services
- Familiar with CD/CI
- Experience building real time data pipelines with Storm, Spark, or equivalent
- Expert in at least one JVM language
2
Data Platform Engineer Job Description
Job Description Example
Our company is looking for a data platform engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for data platform engineer
- Master in all things Kubernetes!
- Flexible in container technology – you know the difference between Docker and Rocket
- Scaled a distributed messaging / streaming system – we’d like to use Apache Kafka
- Work with Business and IT groups to design and deliver a data integration and workflow framework that provides for the loading of data, data integration, quality management, aggregation, joins and distribution to various systems using advanced Big data technologies
- Go-to person for technical guidance, help and expertise to consumer questions on technologies and services on the platform like Hive, Pig, Power BI, SAS, R
- Standard DBA responsibilities backup, performance, maintenance, patching
- Work with different teams of data scientists, everything from inventory forecasting to styling recommendations
- In particular, you’ll work closely with data scientists to make our serving infrastructure support on-demand API driven statistical forecasting
- Experience in working large volume data and Datawarehousing & BI
- Experience in Elastic and Kibana
Qualifications for data platform engineer
- Mastery of SQL
- Data structures and algorithms are essentials
- 1-2 years of experience in full-stack web development (preferably in Python-based server frameworks)
- Practical experience with modern javascript/HTML standards such as Ecmascript 6 and HTML5
- 1-2 of years experience in data science using R, Matlab, or Python-based analytics frameworks and working with large-scale datasets
- Experience in data-focused Python development with numpy/pandas is a huge plus
3
Data Platform Engineer Job Description
Job Description Example
Our company is growing rapidly and is hiring for a data platform engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for data platform engineer
- Investigate emerging technologies in the Hadoop ecosystem that relate to our needs and implement those technologies
- Optimize and tune the Hadoop cluster to meet the performance requirements of teams
- Partner with Hadoop developers in building best practices for warehouse and analytics environment
- Design and implement a highly availability, scalable, low-latency, and efficient hosted multi-tenant data platform (ingestion, storage, indexing, query, management services)
- Actively work with all partner teams inside Puppet to align around use-cases, API & data contracts
- Work with the product team, to translate functional requirements into pragmatic logical, service-based technical designs
- Foster continuous learning, by quickly iterating through design and implementation, validation and delivery
- Be a technical leader both within the team and organization through coaching, mentoring, code reviews focussing on continuously improving our engineering processes and practices, quality and deliverables
- Take responsibility for and participate in the support of hosted data platform and related services
- But foremost be a great colleague, who enjoys working closely with all your other talented colleagues around the world in a collaborative and respectful manner
Qualifications for data platform engineer
- Working knowledge of D3.js or another visualization language/framework is a plus
- Knowledge of streaming technologies such as Websockets, Spark Streaming, and ZeroMQ is a plus
- Many of the changes we need would also benefit others in the big data community
- You’ll help us investigate various file formats
- Migration and upgrading from previous version of SQL Server
- At least 5 years of experience administering and supporting a SAP Business Objects and/or Tableau platform
4
Data Platform Engineer Job Description
Job Description Example
Our company is looking for a data platform engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for data platform engineer
- Help build the data platform which will have a wide range in terms of reach, from supporting consumer facing smart data products, industrializing capabilities to power internal needs
- Be in an environment that supports tight collaboration between data scientists and data engineers, from designing, modeling, experimenting, all the way to deployment and operating production level code
- Find the best and most creative solution to many different types of problems and help our client to achieve their real goals
- Find the best and most creative solution to many different types of problems and help our
- Participate in architecture discussions, influence product roadmap, and take ownership and responsibility of new projects
- Maintain and support existing platforms and evolve to newer technology stacks and architecture
- Establishing, creating and executing the testing strategy at team level
- You are Zen when you write code
- Familiar with development workflows – Git, GitLab CI, Jenkins
- You have worked with configuration management and orchestration tools such as Chef, Puppet, Ansible, Terraform
Qualifications for data platform engineer
- Proficient with data processing systems such as Hadoop and/or Spark
- At least 2 years of experience working with diverse data sources, including both structured and unstructured data (#ilovedata)
- Masters Degree in Computer Science, Math, Statistics or equivalent
- 2+ years operating & managing environments in AWS Cloud
- 2+ years of experience in object oriented programming languages such as Python and Java
- 2+ years of experience with Hadoop & other Big Data Technologies #bigdata
5
Data Platform Engineer Job Description
Job Description Example
Our company is looking for a data platform engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for data platform engineer
- Design systems and solutions for all phases of product lifecycle, freeing up engineers and key resources to focus on features and business tasks, by leveraging extensive experience in automation, system administration, advanced troubleshooting and performance management
- Implement advanced monitoring and metrics to provide operational rigor and oversight for all levels of the business from engineering to sr
- Bring in new ideas, whether it's a new tool or technology, that will help us massively innovate
- Bring an automation mindset and ensure manual tasks are automated
- Effectively manage delivery of the next generation of data warehouse technology providing best-in-class scaling, availability, and operational support to deliver a WOW experience for our customers
- Design, develop and manage diagnostic and instrumentation tools for troubleshooting and in-depth analysis
- Design, development, continuously refine and improve of Data Analytics platforms and systems
- Provide an accessible interface for analytics purposes
- Be a trusted advisor in discussions with GTT internal stakeholders, Group IT and other suppliers of Data Analytic platforms and tools
- Be part of a team responsible for technical vision and development of platform components of the next generation bureau platform
Qualifications for data platform engineer
- 3+ year’s professional experience in software development using Java or C++
- Experience with Hadoop/MapReduce/Pig/Hive/EMR/Redshift for large data processing
- Experience with a DW technology (Oracle, Teradata, Netezza, Redshift, HDFS Hive ) and relevant data modeling practices
- At least 2 years of experience working with code repositories and build tools
- At least 2 years of experience in programming with one of the high level languages
- 2+ years of experience with database and other persistence storage administration