What are the responsibilities and job description for the Local Activity Experience Evaluator – Weekend AI Response Reviewer (New Orleans, LA) position at Volga Partners?
Important Notice Before Applying:
CANDIDATES MUST PASS THE FOLLOWING IN ORDER TO WORK ON THIS PROJECT:
- Interview
- Pass Assessment (This is based on their expertise of their city)
- Once the assessment is submitted, you must allow up to 7 business days for review and results. This does not include weekends.
- Successfully Complete Compliance Onboarding
Job Description:
We are seeking detail-oriented individuals to participate in a short-term pilot project evaluating AI-generated weekend mini-plans tailored to local activities and attractions in your city. This role involves assessing prompts based on predefined criteria such as activity relevance, compelling-ness, and diversity. Strong familiarity with your city’s attractions, cultural nuances, and hidden gems is essential for providing valuable insights to enhance the quality of AI recommendations.
This project is expected to require up to 5 hours of effort, with the potential for extension based on feedback and performance. Evaluators will play a critical role in refining AI-generated responses to improve the user experience.
Key Responsibilities:
Prompt Evaluation:
- Evaluate 20 AI-generated prompts, each consisting of a preamble and mini-plan (a set of 2 local activity suggestions).
- Assess prompts using the following criteria:
- Activity Relevance: Determine how well activities align with the user’s query (e.g., location, time, audience).
- Description Interestingness: Evaluate how engaging and unique the activity descriptions are.
- Mini-plan Compelling-ness: Judge the likelihood that a local user would enjoy the combination of activities.
- Preamble Relevance: Review the introduction to ensure it aligns with the suggested activities and user context.
- Mini-plan Diversity: Ensure the activities offer variety and avoid repetition.
Feedback and Scoring:
- Provide detailed justifications for your ratings to ensure actionable feedback.
- Identify opportunities to improve activity descriptions or suggestions to make them more relevant and engaging.
- Use the defined rating scale and evaluation process for consistency.
Data Labeling and Reporting:
- Accurately label data per project guidelines and evaluation criteria.
- Submit thorough feedback and justifications within specified deadlines.
Qualifications:
- In-depth local knowledge: Strong familiarity with your city’s attractions, cultural highlights, and recreational options.
- Attention to detail: Ability to critically evaluate activity suggestions and provide clear justifications for ratings.
- Reliability: Demonstrated ability to meet deadlines and maintain high-quality deliverables.
- Familiarity with diverse cultural experiences and a commitment to inclusivity in evaluations.
Project Scope and Commitment:
- Estimated Hours: Each prompt takes 10–15 minutes; completing all 20 prompts requires approximately 3 hours.
- Start Date: Will be solidified once all resources are identified for each city.
- Compensation: $12.00 - $17.00 per hour based on experience and location.
- Type: 1099 Contract
- Backup Team: You will work with 5–7 other evaluators per city to ensure coverage and mitigate risks of non-engagement.
Target Cities:
- New York, NY
- Los Angeles, CA
- Chicago, IL
- Houston, TX
- Phoenix, AZ
- San Francisco, CA
- New Orleans, LA
- Cincinnati, OH
- Reno, NV
- Baton Rouge, LA
Job Types: Contract, Temporary
Pay: $12.00 - $17.00 per hour
Expected hours: 3 per week
Compensation Package:
- Hourly pay
People with a criminal record are encouraged to apply
Application Question(s):
- Can you describe your familiarity with your city’s local attractions, events, and hidden gems?
- This role requires evaluating AI-generated responses based on criteria like relevance, diversity, and compelling-ness. Can you give an example of how you would assess a suggested activity for a weekend itinerary?
- This project requires detailed justifications for your ratings. Can you give an example of a time you had to analyze and explain why something was good or needed improvement?
- This is a short-term project with strict deadlines. Are you available to commit to completing 20 evaluations within the required 3 hours? How do you manage your time when working on tasks with a deadline?
Work Location: Remote
Salary : $12 - $17