Principal Data Engineer Job Description
Principal data engineer
provides expertise on batch & stream analytics with HDFS, Kafka, Spark, Web-HDFS and Hortonworks stack.
Principal Data Engineer Duties & Responsibilities
To write an effective principal data engineer job description, begin by listing detailed duties, responsibilities and expectations. We have included principal data engineer job description templates that you can modify and use.
Sample responsibilities for this position include:
Strong knowledge of shell scripting or Python
Ability to analyze big data elements to determine what areas require coding using JAVA programming language
Responsible for the building, deployment, and maintenance of mission critical analytics solutions that process data quickly at big data scales
Advocates and supports Agile development methodologies
The individual is responsible for executing the various software development lifecycle (SDLC) activities – user stories, use cases, software requirements, design, development and design verification (incl
Provide technical direction to a world class engineering teamCollaborate with sales and Client Partners to develop new business and manage existing client relationships
Help establish thought leadership in the big data space by contributing to white-papers, technical commentary, and representing our company at industry conferences
Design and develop code, scripts, and data pipelines that leverage structured and unstructured data
Participate in requirements and design workshops with our clients
Must be adept at sizing and estimating new projects, supporting new business development
Principal Data Engineer Qualifications
Qualifications for a job description may include education, certification, and experience.
Licensing or Certifications for Principal Data Engineer
List any licenses or certifications required by the position: CCIE, AWS, ITIL, GCP, SCTE, IBM, ML, TDWI, CCDA, CCNA
Education for Principal Data Engineer
Typically a job would require a certain level of education.
Employers hiring for the principal data engineer job most commonly would prefer for their future employee to have a relevant degree such as Bachelor's and Master's Degree in Computer Science, Engineering, Technical, Information Systems, Mathematics, Business, Education, Statistics, Science, Software Engineering
Skills for Principal Data Engineer
Desired skills for principal data engineer include:
Current and emerging technologies and products
SQL
Message queuing
Stream processing
Advertising platform
Python
Data integrity protocols
Java
Kafka
Spark
Desired experience for principal data engineer includes:
Undergraduate degree in Computer Science or Engineering from a top CS program required
Experience defining system architectures and designs, exploring technical feasibility tradeoffs, creating POCs using new technologies, and productizing the best solutions in line with business requirements
A consistent track record of delivering exceptionally high quality software on large, complex, cross-functional projects
Experience with building backend systems to support mobile applications preferred
Experience as a gamer or strong interest in the gaming industry preferred
Experience as a gamer or strong interest in the gaming industry a plus
Principal Data Engineer Examples
1
Principal Data Engineer Job Description
Job Description Example
Our company is growing rapidly and is looking for a principal data engineer. To join our growing team, please review the list of responsibilities and qualifications.
Responsibilities for principal data engineer
- Devops for unit, functional, integration and regression test plans
- Design and develop code, scripts and data pipelines that leverage structured and unstructured data integrated from multiple sources
- Develop project deliverable documentation
- Build flawless data feeds from enterprise data warehouse and other data sources into the CI Data Mart which we are currently in the process of building
- Take complete ownership of the data quality for the data feeds built and have a passion for high quality data
- Always be on the lookout to automate and improve existing data processes for quicker turnaround and high productivity
- Be quick to learn new technologies deliver on them in short order
- Collaboratively work with IT and Product teams in an agile environment, to meet deadlines
- The availability, security, scalability of our e-commerce platform
- Create and manage Redshift clusters
Qualifications for principal data engineer
- Knowledge of SQL and MPP databases
- Direct experience with SQL and MPP databases
- Strong Java programming skills required
- Strong communications skills with different levels in the organization business acumen to understand various types of business problems
- Coursework and/or experience with communication theory, digital signal processing, probability theory, stochastic signal processing, RF and electromagnetic theory
- Master’s/Ph
2
Principal Data Engineer Job Description
Job Description Example
Our company is growing rapidly and is looking to fill the role of principal data engineer. If you are looking for an exciting place to work, please take a look at the list of qualifications below.
Responsibilities for principal data engineer
- Perform and managing migrations
- Documenting and practicing failovers and DR plans
- Automation of recurring DBA tasks
- Understand endpoints of where the data will be pushed by creating a coding template, structure, and attributes
- Establish and maintain a data catalog associated with a big data data lake
- Lead establishment of common data models for concerns like storage, data exchange with external institutions and use for visualization
- Lead message schema governance body
- Message schema standards management and communication
- Assist teams in message definition to maximize value and re-use
- Analyze and design appropriate scheduling and data movement choreography solutions using multi-tenant, highly available software components
Qualifications for principal data engineer
- Experience with Hadoop or/and Hbase
- Experience with Analytics suites like Tableau, MicroStrategy, Bussiness Objects, Cognos
- OExperience with polyglot databases, nosql, rdbms and other new sql
- OExperience with RDF, formal logic, or other advanced modeling tools
- OExperience with data mining techniques
- OExperience in life sciences is a strong plus
3
Principal Data Engineer Job Description
Job Description Example
Our growing company is looking to fill the role of principal data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for principal data engineer
- Research, design and implement big data tools and frameworks required to provide request capabilities
- Identify and integrate new technologies to enhance existing infrastructure (AWS, SaaS)
- Designs highly scalable ETL processes with complex data transformations, from disparate applications and data formats
- Designs secure cloud solutions using AWS services like EC2, S3, IAM roles and policies, Auto scaling, Elastic Load Balancing, and VPC
- Support of technology implementations from a technical perspective including experience with cloud based technologies, data cleansing and conversions, interface design/development, systems development life cycle
- Design, develop, execute and document software solutions to address complex data collection, processing, transformation and reporting issues
- Provide architectural blueprints and technical leadership
- Evaluate and recommend tools, technologies and processes to ensure the highest quality and performance is achieved
- Collaborate with peer organizations, dev ops, support organizations on technical issues and provide guidance
- Interpret and translate business needs to technical requirements
Qualifications for principal data engineer
- Experience working with NoSQL and large amounts of data highly desired
- 5+ years of experience with data pipelines and familiarity with analytics and concepts
- OOP expertise required, preferably C#/C++/Java - Bachelor degree or an advanced degree (Masters and/or PhD) in computer science, mathematics, data mining, statistics, machine learning, or related field
- Experience working on a large codebase
- 12+ years hands-on experience with shipping services and databases
- Ability to work in a fast-changing and dynamic environment
4
Principal Data Engineer Job Description
Job Description Example
Our company is looking for a principal data engineer. Thank you in advance for taking a look at the list of responsibilities and qualifications. We look forward to reviewing your resume.
Responsibilities for principal data engineer
- Work with team to troubleshoot code level problems quickly and efficient on as need basis
- Build technical road-map and technical debt elimination program
- Focus on scalability, security and availability of all applications and processes
- Participate in technical audit programs and provide guidance to all team members
- Motivate and educate team members on using coding standards and best practices
- Participate in code and document reviews
- The data platform must be highly available and scalable
- It should be able to look at both historic real time data to compute
- Provide hands-on subject matter expertise to building reliable and highly available systems
- End to end ownership of your service including design, deployment and management as a docker container
Qualifications for principal data engineer
- Dynamic and/or funtional languages (Python, Ruby, Scala, Clojure)
- Experience working with cloud-based marketing and sales data a plus (Marketo, Salesforce, etc)
- Familiarity with other programming languages (Python, C, C++)
- Background which includes mathematics, statistics, machine learning and data mining
- Data security & governance expertise
- Bachelors or Masters in Computer Science or related field with at least 5 years experience in a highly technical data organization
5
Principal Data Engineer Job Description
Job Description Example
Our innovative and growing company is looking for a principal data engineer. Please review the list of responsibilities and qualifications. While this is our ideal list, we will consider candidates that do not necessarily have all of the qualifications, but have sufficient experience and talent.
Responsibilities for principal data engineer
- Work cross-functionally to understand the linkage between business goals, business architectures, and technology architectures – and strive to further business goals via organic or inorganic evolution of cloud architectures
- Can oversee multiple projects, have in-depth knowledge to make technical decisions and lead technical discussions
- Act as application service manager and provide technical support for our global IT Portfolio tracking and analysis tool (Web application based on Ruby and an Oracle database)
- Provide technical expertise for organizing our Business Operations Core Data and their relationships held in various databases/tools
- Use our currently productively used reporting and analyses solutions (Spotfire, SSAS, SSRS) to maintain and evolve all productive reports and analyses based on them, the creation of new report and analysis solutions
- Identify new opportunities and innovative ways to enhance our entire Business Operations Core Data management, the reporting and analysis environment and methods used with the goal to constantly increase efficiency, reliability, stability, consistency, the flexibility to address new portfolio analysis related requirements in a timely fashion
- Define proper data management solution(s) to meet our data management, reporting and analyses needs and requirements
- Perform prototyping to refine the requirements with proper documentation
- Implement new, agreed solutions, maintain and evolve them technically over time
- Actively work with and for the P&O team and other stakeholders/customers in P&O and NIBR Informatics, NX operations teams providing services needed to deliver the intended solutions
Qualifications for principal data engineer
- At least 3 years solid hands-on experience in Hadoop Ecosystem of Tools (Spark, Hive, Pig, Oozie, Impala, MapReduce) is a must
- 5+ years experience in writing complicated database queries in SQL language (Oracle, Vertica, Hive, etc) designing robust relational database objects
- Good proficiency in Python scripting is a must
- 3+ years of experience in ETL design, implementation and maintenance (Informatica/Talend/Pentaho)
- Strong understanding of database performance concepts like indices, segmentation, projections, and partitions
- Experience working in a data warehouse environment with diverse data sources is necessary