Data Evaluator

About this opportunity

When a child types a question into a search box, the gap between what they typed and what they meant is usually wide. "Why is the sky blue", "how do volcanoes work", "is my hamster sick" - each has an intent behind it that the words only partly carry.

Search results evaluators close that gap. You will see a query and a set of results, work out what the person was actually trying to find, and rate how well each result meets that need. You will also judge the result itself: is the page trustworthy, is it written at a level the person could follow, and is it something we would be comfortable putting in front of a ten-year-old.

This is judgement work against a written rubric, not data entry. The rating matters, and the short written reason you give for it matters just as much - it is what makes the rating reviewable and what teaches the system.

No search-industry background is required. If you can read carefully, hold a rubric in your head and explain a decision in three sentences, you can do this work well.

What you'll do

  • Interpret the intent behind a query, including ambiguous and misspelled ones
  • Rate how well each result meets that intent against a written scale
  • Assess page quality: who wrote it, whether it is accurate, whether it is trying to sell something
  • Judge reading level and age-appropriateness for a child audience
  • Flag results that are harmful, misleading, or dressed-up advertising
  • Write a short, specific reason for every rating that another evaluator could follow
  • Raise queries the rubric does not cleanly cover rather than forcing a rating

What we're looking for

  • Excellent written English and careful reading
  • Sound judgement on ambiguous questions with no single right answer
  • Able to apply a detailed written rubric consistently across long sessions
  • Comfortable assessing whether a web page is trustworthy
  • Reliable internet connection and a computer

How the work is scheduled

  • Fully asynchronous and pick up task batches when it suits you
  • No minimum weekly hours and no fixed shifts
  • Paid per hour of reviewed work, invoiced every two weeks
  • Calibration tasks at the start of each project are paid

How we assess applications

A short paid calibration set, graded against the same rubric you would use on the job. We look at how consistent your ratings are and how clearly you justify them - not at how fast you work.

Good to know

  • This is evaluation work on a children's product, not general content moderation - you will not be reviewing graphic or illegal material
  • Queries occasionally touch sensitive subjects (illness, family difficulty). These are flagged in advance and you may decline any batch without explanation
  • Locale matters: tell us which country's results you can judge confidently
  • We are unable to support H1-B or STEM OPT contractors at this time.

Pay: $20.00 - $25.00 per hour

Expected hours: 15.0 – 20.0 per week

Benefits:

  • Flextime
  • Work from home

Work Location: Remote

Share Share
Apply Now →