What are the responsibilities and job description for the Research Data Engineer Lead position at Elicit?
Data Engineering Opportunities at Elicit
Elicit's mission is to make us the most complete and up-to-date database of scholarly sources. We are seeking a talented Data Engineer to join our team and contribute to this mission.
- Build and Optimize Academic Research Paper Pipeline
- The ideal candidate will architect and implement robust, scalable solutions to handle our growing data needs while maintaining high performance and data quality.
- They will work on efficiently processing, deduplicating, and indexing hundreds of millions of research papers.
- Enhance Elicit's Data Infrastructure
- The successful candidate will optimize our Spark jobs and data pipelines to handle large amounts of data efficiently.
- They will implement data partitioning strategies in our distributed systems to improve performance.
- Maintain and Improve Data Quality
- The Data Engineer will implement robust data quality management processes to ensure the accuracy and reliability of our academic database.
- They will work on developing defenses against unexpected changes from publishers to maintain data integrity.
