- •Releasing Tasks
- •Introduction to Page Quality Rating
- •Understanding Webpages and Websites
- •Important Definitions
- •What is the Purpose of a Webpage?
- •Your Money or Your Life (ymyl) Pages
- •Understanding Webpage Content
- •2.4.1 Identifying the Main Content (mc)
- •2.4.3 Identifying Advertisements/Monetization (Ads)
- •2.4.2 Identifying the Supplementary Content (sc)
- •2.4.4 Summary of the Parts of the Page
- •Understanding the Website
- •2.5.1 Finding the Homepage
- •2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page
- •2.5.3 Finding About Us, Contact Information, and Customer Service Information
- •Website Maintenance
- •Website Reputation
- •2.7.1 Reputation Research
- •2.7.2 Sources of Reputation Information
- •2.7.3 Customer Reviews of Stores/Businesses
- •2.7.4 How to Search for Reputation Information
- •2.7.5 What to Do When You Find No Reputation Information
- •Overall Page Quality Rating Scale
- •High Quality Pages
- •Characteristics of High Quality Pages
- •A Satisfying Amount of High Quality Main Content
- •A High Level of Expertise/Authoritativeness/Trustworthiness (e-a-t)
- •Positive Reputation
- •Helpful Supplementary Content
- •Functional Page Design
- •A Satisfying Amount of Website Information
- •A Well Cared For and Maintained Website
- •Examples of High Quality Pages
- •Highest Quality Pages
- •Very High Quality mc
- •Very High Level of e-a-t
- •Very Positive Reputation
- •Examples of Highest Quality Pages
- •6.0 Low Quality Pages
- •Low Quality Main Content
- •Unsatisfying Amount of Main Content
- •Lacking Expertise, Authoritativeness, or Trustworthiness (e-a-t)
- •Negative Reputation
- •Characteristics Which May Be Evidence of Low Quality
- •6.5.1 Unhelpful or Distracting Supplementary Content
- •6.5.2 Lacking Supplementary Content
- •6.5.3 Poor Page Design
- •6.5.4 Lacking Care and Maintenance
- •Unsatisfying Amount of Information about the Website
- •Examples of Low Quality Pages
- •7.0 Lowest Quality Pages
- •Harmful or Malicious Pages
- •Lack of Purpose Pages
- •Deceptive Pages
- •7.3.1 Deceptive Page Purpose
- •7.3.2 Deceptive Page Design
- •7.3.3 Sneaky Redirects
- •Lowest Quality Main Content
- •7.4.1 No Main Content
- •7.4.2 “Keyword Stuffed” Main Content
- •7.4.3 Gibberish or Meaningless Main Content
- •7.4.4 Automatically-Generated Main Content
- •7.4.5 Copied Main Content
- •7.4.6 More About Copied Content
- •7.4.7 How to Determine if Content is Copied
- •No Website Information
- •Highly Untrustworthy, Unreliable, Unauthoritative, Inaccurate, or Misleading
- •Abandoned Websites or Spammed Pages on a Website
- •Extremely Negative or Malicious Reputation
- •Examples of Lowest Quality Pages
- •8.0 Medium Quality Pages
- •Examples of Medium Quality Pages
- •9.0 Page Quality Rating: Important Considerations
- •Instructions for Rating Page Quality Tasks
- •The Top Three pq Considerations
- •Page Quality Considerations for Specific Types of Pages
- •Ratings for Encyclopedia Pages
- •Ratings for Pages with Error Messages or No mc
- •Ratings for Forums and q&a pages
- •Page Quality Rating faQs
- •Understanding Mobile Users, Mobile Queries, and Mobile Results
- •Important Rating Definitions and Ideas
- •Understanding the Query
- •Task Location (Locale) and User Location
- •Queries with an Explicit Location
- •Queries with Multiple Meanings
- •Query Meanings Can Change Over Time
- •Understanding User Intent
- •Know and Know Simple Queries
- •Do and Device Action Queries
- •Website Queries
- •Visit-in-Person Queries and User Location
- •Queries with Multiple User Intents
- •Understanding Result Blocks
- •Web Search Result Block Examples
- •Special Content Result Block Examples
- •Device Action Result Block Examples
- •How Device Action Results are Displayed in Rating Tasks
- •Rating Using the Needs Met Scale
- •Rating on Your Phone Issues
- •Rating Result Blocks: Block Content and Landing Pages
- •Fully Meets (FullyM)
- •Examples of Fully Meets (FullyM) Result Blocks
- •Examples of Result Blocks that Cannot be Fully Meets
- •Highly Meets (hm)
- •Examples of Highly Meets (hm) Result Blocks
- •Moderately Meets (mm)
- •Examples of Moderately Meets (mm) Result Blocks
- •Slightly Meets (sm)
- •Examples of Slightly Meets (sm) Result Blocks
- •Fails to Meet (FailsM)
- •Examples of Fails to Meet (FailsM) Result Blocks
- •14.0 Rating Porn, Foreign Language, Didn’t Load, and Hard to Use Results
- •Porn Flag
- •Needs Met Rating for Porn Results
- •Needs Met Rating for Clear Non-Porn Intent Queries
- •Needs Met Rating for Possible Porn Intent Queries
- •Needs Met Rating for Clear Porn Intent Queries
- •Reporting Illegal Images
- •Foreign Language Flag
- •14.4.1 Using the Foreign Language Flag
- •14.4.2 Needs Met Ratings for Foreign Language Results
- •14.4.3 English Language Results
- •Didn’t Load Flag
- •14.5.1 Using the Didn’t Load Flag
- •14.5.2 Needs Met Rating for Didn’t Load Results
- •Hard to Use Flag
- •14.6.1 Using the Hard to Use Flag
- •The Relationship between e-a-t and Needs Met
- •Rating Queries with Multiple Interpretations and Intents
- •Rating Queries with Both Website and Visit-in-Person Intent
- •Specificity of Queries and Landing Pages
- •Needs Met Rating and Freshness
- •Misspelled and Mistyped Queries and Results
- •Misspelled and Mistyped Queries
- •19.2 Name Queries
- •19.3 Spelling Suggestion Result Blocks
- •21.0 Product Queries: Action (Do) vs. Information (Know) Intent
- •22.0 Rating Visit-in-Person Intent Queries
- •26.0 Needs Met Task Page Screenshot
- •27.0 Notes about Using the Needs Met Rating Interface
- •28.0 Using the “Report a Problem / Release this Task” Button
- •29.0 Reporting Results with Duplicate Landing Pages
- •Rater-Identified Duplicates
Slightly Meets (sm)
A rating of Slightly Meets is assigned to results that are helpful and satisfying for some or few users. Slightly Meets results may serve a minor interpretation, be low quality, have stale or outdated information, be too specific, too broad, etc. to receive a higher rating.
Examples of Slightly Meets (sm) Result Blocks
|
Query, User Location, |
|
|
Result |
|
|
Rating |
|
|
Explanation |
|
|
|
User Intent |
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Query: [britney spears] |
|
|
|
|
|
|
|
|
|
|
|
|
User Location: |
|
|
|
|
|
|
|
The LP of this mobile-friendly |
|
||
|
|
|
|
|
|
|
|
web result has a 2006 article |
|
|||
|
Baltimore, Maryland |
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
about Britney Spears filing for |
|
|||
|
|
|
|
|
|
|
|
|
|
|
||
|
User Intent: Find |
|
|
|
|
|
|
|
divorce. This is very old, |
|
||
|
|
|
|
|
|
|
|
stale news, making the result |
|
|||
|
information about |
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
helpful for few users. |
|
|||
|
Britney Spears (e.g., |
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
|
||
|
current news, pictures). |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
Query: [honda odyssey] |
|
|
|
|
|
|
|
Although the LP has |
|
||
|
User Location: Miami, |
|
|
|
|
|
|
|
comprehensive information |
|
||
|
|
|
|
|
|
|
|
from a reputable source, and |
|
|||
|
Florida |
|
|
|
|
|
|
|
is easy to use on a mobile |
|
||
|
User Intent: We will |
|
|
|
|
|
|
|
device, the information is |
|
||
|
|
|
|
|
|
|
|
about the 2010 Honda |
|
|||
|
assume that users are |
|
|
|
|
|
|
|
Odyssey. This information |
|
||
|
interested in the current |
|
|
|
|
|
|
|
would be considered stale for |
|
||
|
Honda Odyssey model, |
|
|
|
|
|
|
|
the query today, making this |
|
||
|
unless specified |
|
|
|
|
|
|
|
result helpful for some or few |
|
||
|
otherwise. |
|
|
|
|
|
|
|
users. |
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 109
|
Query, User Location, |
|
|
Result |
|
|
Rating |
|
|
Explanation |
|
|
||
|
User Intent |
|
|
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The LP of this web result is |
|
|
|
Query: [hot dog] |
|
|
|
|
|
|
|
|
|
about the movie “Hot Dog,” |
|
||
|
|
|
|
|
|
|
|
|
|
which came out in 1984 and |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Location: |
|
|
|
|
|
|
|
|
|
is a minor interpretation of |
|
||
|
|
|
|
|
|
|
|
|
|
this query. Because this is an |
|
|||
|
Sunnyvale, California |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
unlikely interpretation, this |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Intent: Find |
|
|
|
|
|
|
|
|
|
web result would be helpful |
|
||
|
|
|
|
|
|
|
|
|
|
for few users, even though |
|
|||
|
information about hot |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
the landing page has good |
|
|||
|
dogs, such as recipes or |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
content from a reputable |
|
|||
|
nutrition information. |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
source, and is easy to use on |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
a mobile device. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This result provides contact |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
Query: [motorcycles] |
|
|
|
|
|
|
|
|
|
information for one of the two |
|
||
|
|
|
|
|
|
|
|
|
|
motorcycle shops in |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Location: |
|
|
|
|
|
|
|
|
|
Potsdam, New York. It is |
|
||
|
|
|
|
|
|
|
|
|
|
possible this would be helpful |
|
|||
|
Potsdam, New York |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
for some users, but many |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Intent: Find |
|
|
|
|
|
|
|
|
|
users would be looking for |
|
||
|
|
|
|
|
|
|
|
|
|
information. Additionally, |
|
|||
|
information about |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
there is a distracting |
|
|||
|
motorcycles. |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
information (a listing for an |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
insurance agent). |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This is a Wikipedia page that |
|
|
|
Query: [abe lincoln’s |
|
|
|
|
|
|
|
|
|
has birthdays for all U.S. |
|
||
|
|
|
|
|
|
|
|
|
|
presidents, including |
|
|||
|
birthday] |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
Abraham Lincoln. However, |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Location: Fort |
|
|
|
|
|
|
|
|
|
his birthday is not prominently |
|
||
|
|
|
|
|
|
|
|
|
|
displayed and it is not |
|
|||
|
Davis, Texas |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
obvious that the answer is |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
User Intent: Find this |
|
|
|
|
|
|
|
|
|
there—users have to do |
|
||
|
|
|
|
|
|
|
|
|
|
some work and search |
|
|||
|
specific piece of |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
around on the page to find |
|
|||
|
information. |
|
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
the answer, making SM to |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
SM+ an appropriate rating. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
Query: [lack of sex and |
|
|
|
|
|
|
|
|
|
The quality of writing in this |
|
||
|
problems with my |
|
|
|
|
|
|
|
|
|
article, which was created by |
|
||
|
marriage] |
|
|
|
|
|
|
|
|
|
a person without expertise in |
|
||
|
User Location: Waco, |
|
|
|
|
|
|
|
|
|
marriage or relationship |
|
||
|
|
|
|
|
|
|
|
|
|
counseling, is poor. Even |
|
|||
|
Texas |
|
|
|
|
|
|
|
|
|
though the article is about the |
|
||
|
User Intent: Find help |
|
|
|
|
|
|
|
|
|
query, the page is low quality |
|
||
|
|
|
|
|
|
|
|
|
|
and untrustworthy. It would |
|
|||
|
for marital issues. |
|
|
|
|
|
|
|
|
|
be helpful for few users. |
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 110
|
Query, User Location, |
|
|
|
Result |
|
Rating |
|
|
|
Explanation |
|
|
|
User Intent |
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Query: [nail spa] |
|
Burke Williams is a day spa |
|
|
|
|
|
User Location: San |
|
that offers many treatments |
|
|
like massage therapy, skin |
|
|
Jose, California |
|
|
|
|
care, etc. While it does have |
|
|
|
|
|
|
User Intent: Find a nail |
|
nail care options, it is not a |
|
|
dedicated nail spa. |
|
|
spa in or near San Jose. |
|
|
|
|
|
|
|
|
|
|
|
Query: [what type of |
|
|
|
|
|
sharks live in rivers] |
|
|
|
This result block does not |
|
User Location: Tucson, |
|
|
|
|
|
|
|
|
contain enough information to |
|
|
Arizona |
|
|
|
be fully satisfying and |
|
User Intent: Find |
|
|
|
mentions only one type of |
|
|
|
|
shark. Most users would |
|
|
information about what |
|
|
|
need to do further research. |
|
types of sharks live in |
|
|
|
|
|
rivers. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The answer in this result |
|
|
|
|
|
|
|
Query: [210 area code] |
|
|
|
block is really confusing. |
|
|
|
|
Even if users can figure it out, |
|
|
|
|
|
|
|
|
User Location: Austin, |
|
|
|
they would likely still have to |
|
|
|
|
click on the link, or go to |
|
|
Texas |
|
|
|
|
|
|
|
|
another website, to confirm |
|
|
|
|
|
|
|
|
User Intent: Find what |
|
|
|
the information. |
|
|
|
|
|
|
|
area uses the 210 area |
|
|
|
While there is a map, you |
|
code for phone |
|
|
|
|
|
|
|
|
would have to expand or click |
|
|
numbers. |
|
|
|
|
|
|
|
|
on it to see the details, so it is |
|
|
|
|
|
|
|
|
|
|
|
|
not that helpful. |
|
|
|
|
|
|
|
|
|
|
|
|
|
Query: [ibm] |
|
|
|
|
|
|
|
|
|
|
|
User Location: |
|
|
|
This block contains images of |
|
Rockville, Maryland |
|
|
|
|
|
|
|
|
the logo for IBM, but these |
|
|
|
|
|
|
|
|
User Intent: Go to the |
|
|
|
images are not particularly |
|
|
|
|
helpful for this query. |
|
|
IBM website or find |
|
|
|
|
|
|
|
|
|
|
|
information about the |
|
|
|
|
|
company. |
|
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 111
