What are the responsibilities and job description for the Applied Scientist position at Shrive Technologies?
Job Title: Applied Scientist
Location: Redmond / Remote
Duties/Responsibilities
Job Summary:
We focus on evaluating BizChat quality across various scenarios, such as summarizing content from documents, generating images, and projecting data into graphs. For each scenario, we have a set of concrete user tasks (e.g., "Could you extract the KPIs from the document and project them into a table?"). We will obtain responses from different versions of AI Assistants. The Applied Scientist (AS) will help us[KM7] [LY8] [YW9] develop an LLM-based labeler that can evaluate the quality of responses given the task.
To achieve this, the AS is expected to combine different methods as appropriate.
Location: Redmond / Remote
Duties/Responsibilities
Job Summary:
We focus on evaluating BizChat quality across various scenarios, such as summarizing content from documents, generating images, and projecting data into graphs. For each scenario, we have a set of concrete user tasks (e.g., "Could you extract the KPIs from the document and project them into a table?"). We will obtain responses from different versions of AI Assistants. The Applied Scientist (AS) will help us[KM7] [LY8] [YW9] develop an LLM-based labeler that can evaluate the quality of responses given the task.
To achieve this, the AS is expected to combine different methods as appropriate.
- For example, one approach is to define the correct answer for a given user task and match the chatbot response to this answer, such as a mathematical task like "Can you help me sort the following numbers: 67, 111, ...". The AS can generate these answers by combining results from multiple LLM models.
- For creative open-answer questions such as "Summarize the key points from section 4 in this document," the AS can help develop an LLM-based judge to evaluate the quality of the summary. To do so, they are expected to refine the prompt to the LLM judge, guiding the LLM-based judge to score different aspects of the summary based on dimensions such as accuracy, comprehensiveness, and conciseness.
- Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, Applied Mathematics, or a related field AND 3 years of related experience (e.g., statistics, predictive analytics, research).
- Experience working on successful applied research projects in industry environments.
- Applicants with the following skills would be a plus:
- Good coding/engineering skill to build production quality solution.
- Applications of Foundation Models
- Application of Vision, Audio, and Multimodal Foundation Models
- Prompt Tuning
- Natural Language Processing
- Applied Machine Learning
- Data Mining with Language Models
- Data Annotation Specialty
- Strong English reading comprehension and ability to quickly analyze and interpret complex information or Applied Linguistics background is also a plus.
