- •Releasing Tasks
- •Introduction to Page Quality Rating
- •Understanding Webpages and Websites
- •Important Definitions
- •What is the Purpose of a Webpage?
- •Your Money or Your Life (ymyl) Pages
- •Understanding Webpage Content
- •2.4.1 Identifying the Main Content (mc)
- •2.4.3 Identifying Advertisements/Monetization (Ads)
- •2.4.2 Identifying the Supplementary Content (sc)
- •2.4.4 Summary of the Parts of the Page
- •Understanding the Website
- •2.5.1 Finding the Homepage
- •2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page
- •2.5.3 Finding About Us, Contact Information, and Customer Service Information
- •Website Maintenance
- •Website Reputation
- •2.7.1 Reputation Research
- •2.7.2 Sources of Reputation Information
- •2.7.3 Customer Reviews of Stores/Businesses
- •2.7.4 How to Search for Reputation Information
- •2.7.5 What to Do When You Find No Reputation Information
- •Overall Page Quality Rating Scale
- •High Quality Pages
- •Characteristics of High Quality Pages
- •A Satisfying Amount of High Quality Main Content
- •A High Level of Expertise/Authoritativeness/Trustworthiness (e-a-t)
- •Positive Reputation
- •Helpful Supplementary Content
- •Functional Page Design
- •A Satisfying Amount of Website Information
- •A Well Cared For and Maintained Website
- •Examples of High Quality Pages
- •Highest Quality Pages
- •Very High Quality mc
- •Very High Level of e-a-t
- •Very Positive Reputation
- •Examples of Highest Quality Pages
- •6.0 Low Quality Pages
- •Low Quality Main Content
- •Unsatisfying Amount of Main Content
- •Lacking Expertise, Authoritativeness, or Trustworthiness (e-a-t)
- •Negative Reputation
- •Characteristics Which May Be Evidence of Low Quality
- •6.5.1 Unhelpful or Distracting Supplementary Content
- •6.5.2 Lacking Supplementary Content
- •6.5.3 Poor Page Design
- •6.5.4 Lacking Care and Maintenance
- •Unsatisfying Amount of Information about the Website
- •Examples of Low Quality Pages
- •7.0 Lowest Quality Pages
- •Harmful or Malicious Pages
- •Lack of Purpose Pages
- •Deceptive Pages
- •7.3.1 Deceptive Page Purpose
- •7.3.2 Deceptive Page Design
- •7.3.3 Sneaky Redirects
- •Lowest Quality Main Content
- •7.4.1 No Main Content
- •7.4.2 “Keyword Stuffed” Main Content
- •7.4.3 Gibberish or Meaningless Main Content
- •7.4.4 Automatically-Generated Main Content
- •7.4.5 Copied Main Content
- •7.4.6 More About Copied Content
- •7.4.7 How to Determine if Content is Copied
- •No Website Information
- •Highly Untrustworthy, Unreliable, Unauthoritative, Inaccurate, or Misleading
- •Abandoned Websites or Spammed Pages on a Website
- •Extremely Negative or Malicious Reputation
- •Examples of Lowest Quality Pages
- •8.0 Medium Quality Pages
- •Examples of Medium Quality Pages
- •9.0 Page Quality Rating: Important Considerations
- •Instructions for Rating Page Quality Tasks
- •The Top Three pq Considerations
- •Page Quality Considerations for Specific Types of Pages
- •Ratings for Encyclopedia Pages
- •Ratings for Pages with Error Messages or No mc
- •Ratings for Forums and q&a pages
- •Page Quality Rating faQs
- •Understanding Mobile Users, Mobile Queries, and Mobile Results
- •Important Rating Definitions and Ideas
- •Understanding the Query
- •Task Location (Locale) and User Location
- •Queries with an Explicit Location
- •Queries with Multiple Meanings
- •Query Meanings Can Change Over Time
- •Understanding User Intent
- •Know and Know Simple Queries
- •Do and Device Action Queries
- •Website Queries
- •Visit-in-Person Queries and User Location
- •Queries with Multiple User Intents
- •Understanding Result Blocks
- •Web Search Result Block Examples
- •Special Content Result Block Examples
- •Device Action Result Block Examples
- •How Device Action Results are Displayed in Rating Tasks
- •Rating Using the Needs Met Scale
- •Rating on Your Phone Issues
- •Rating Result Blocks: Block Content and Landing Pages
- •Fully Meets (FullyM)
- •Examples of Fully Meets (FullyM) Result Blocks
- •Examples of Result Blocks that Cannot be Fully Meets
- •Highly Meets (hm)
- •Examples of Highly Meets (hm) Result Blocks
- •Moderately Meets (mm)
- •Examples of Moderately Meets (mm) Result Blocks
- •Slightly Meets (sm)
- •Examples of Slightly Meets (sm) Result Blocks
- •Fails to Meet (FailsM)
- •Examples of Fails to Meet (FailsM) Result Blocks
- •14.0 Rating Porn, Foreign Language, Didn’t Load, and Hard to Use Results
- •Porn Flag
- •Needs Met Rating for Porn Results
- •Needs Met Rating for Clear Non-Porn Intent Queries
- •Needs Met Rating for Possible Porn Intent Queries
- •Needs Met Rating for Clear Porn Intent Queries
- •Reporting Illegal Images
- •Foreign Language Flag
- •14.4.1 Using the Foreign Language Flag
- •14.4.2 Needs Met Ratings for Foreign Language Results
- •14.4.3 English Language Results
- •Didn’t Load Flag
- •14.5.1 Using the Didn’t Load Flag
- •14.5.2 Needs Met Rating for Didn’t Load Results
- •Hard to Use Flag
- •14.6.1 Using the Hard to Use Flag
- •The Relationship between e-a-t and Needs Met
- •Rating Queries with Multiple Interpretations and Intents
- •Rating Queries with Both Website and Visit-in-Person Intent
- •Specificity of Queries and Landing Pages
- •Needs Met Rating and Freshness
- •Misspelled and Mistyped Queries and Results
- •Misspelled and Mistyped Queries
- •19.2 Name Queries
- •19.3 Spelling Suggestion Result Blocks
- •21.0 Product Queries: Action (Do) vs. Information (Know) Intent
- •22.0 Rating Visit-in-Person Intent Queries
- •26.0 Needs Met Task Page Screenshot
- •27.0 Notes about Using the Needs Met Rating Interface
- •28.0 Using the “Report a Problem / Release this Task” Button
- •29.0 Reporting Results with Duplicate Landing Pages
- •Rater-Identified Duplicates
Ratings for Forums and q&a pages
Ratings for forum and Q&A pages can be challenging. Keep in mind the following:
The Main Content on forum and Q&A pages includes both the question as well as the results and resulting discussions.
PQ ratings for a forum or Q&A page should include assessments of the expertise involved in the discussion on the page, as well as the reputation and expertise of the website itself.
Rate forum and Q&A pages from the point of view of a user who visits the page, rather than a participant involved in the discussion.
Assessing expertise for discussion pages can be difficult. Forums and Q&A websites can be expert sources of information on a wide variety of topics. In fact, some types of information are found almost exclusively on forums and discussions, where a community of experts can provide valuable perspectives on very specific topics. When evaluating a forum or Q&A website, do reputation research. Also look for evidence of an active community with a high level of expertise that’s appropriate for the discussion topic. Participation, in-depth discussions, and new threads are signs of an active and engaged community.
On forum and Q&A pages, you should accept claims of everyday expertise and experience unless you have reason to be suspicious of the page. Please value everyday expertise and experience, especially when there are no formal or authoritative sources of information. Keep in mind that everyone is an expert in his or her own life experience. On the other hand, there are many forum or Q&A pages with low quality, untrustworthy MC written by people with little expertise or experience. If the advice or information is not obviously based on life experience or personal expertise, think about how trustworthy the information is.
For example, a patient describing his or her experience with a disease may be considered an expert in his or her own personal experience, but only doctors or other medical experts should offer medical advice to others. “I had the flu for two weeks and felt miserable” is likely an accurate report of a real life experience. However, “You should eat three pounds of chocolate every day that you have the flu, because it made me feel better” is untrustworthy advice.
Proprietary and Confidential – Copyright 2015 61
|
Webpage/Type of Content |
|
Characteristics of the Page |
|
Explanation |
|
||
|
Q&A page about chest |
|
• |
YMYL page with inaccurate potentially |
|
We must evaluate this page from the point of view |
|
|
|
pains and smoking (YMYL) |
|
|
|
||||
|
|
|
dangerous medical advice |
|
of a user visiting this page from a search engine, |
|
||
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
rather than a participant. The question is poorly |
|
|
|
|
|
|
|
|
worded and difficult to understand. The answers |
|
|
|
|
|
|
|
|
are poorly worded and have incorrect and |
|
|
|
|
|
|
|
|
potentially dangerous medical advice, making it |
|
|
|
|
|
|
|
|
lowest quality MC. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In addition to having no answer, this page has |
|
|
|
|
|
• |
Misleading page design |
|
Ads and links to other questions (misleadingly |
|
|
Q&A page with unanswered |
|
• |
Unsatisfying amount of MC for the purpose |
|
labeled as “Relevant answers”) displayed |
|
|
|
question |
|
|
of the page |
|
prominently, which users may mistake for |
|
|
|
|
|
|
|
|
|
answers to the question. It takes a moment to |
|
|
|
|
|
|
|
|
notice that this page actually has no |
|
|
|
|
|
|
|
|
answer. Deceptive design and lack of an answer |
|
|
|
|
|
|
|
|
make this page a frustratingly poor user |
|
|
|
|
|
|
|
|
experience and cause this page to completely fail |
|
|
|
|
|
|
|
|
to achieve its purpose. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In this example, the MC is boxed in red. Please |
|
|
|
|
|
|
|
|
read the MC, including the completely unhelpful |
|
|
|
|
|
|
|
|
"answer" to the question in the red box. This |
|
|
|
|
|
• |
Misleading page design |
|
answer is so unhelpful, we can consider this |
|
|
Q&A page about a 2002 |
|
|
question to be unanswered. This page has an |
|
|||
|
|
• |
Unsatisfying amount of MC for the purpose |
|
|
|||
|
Volvo part |
|
|
unsatisfying amount of MC. |
|
|||
|
|
|
of the page |
|
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In addition to a very unhelpful “answer,” the page |
|
|
|
|
|
|
|
|
design makes it difficult to distinguish the MC from |
|
|
|
|
|
|
|
|
Ads. For example, below the answer, we see a |
|
|
|
|
|
|
|
|
"sponsored answer," which has the same format |
|
|
|
|
|
|
|
|
as the real answer, but is actually an Ad and not |
|
|
|
|
|
|
|
|
an answer to the question. This page design is |
|
|
|
|
|
|
|
|
somewhat misleading or mildly deceptive. |
|
|
|
|
|
|
|
|
|
|
|
Q&A page about a game |
|
• |
Unsatisfying amount of MC for the purpose |
|
Some websites rely on users to create virtually all |
|
|
|
console issue |
|
|
of the page |
|
|
||
|
|
|
|
|
|
|
of their MC. In this case, the MC is the user’s |
|
|
|
|
|
|
|
|
question. If there are no answers, the amount of |
|
|
|
|
|
|
|
|
MC on the page is unsatisfying. |
|
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 62
|
Webpage/Type of Content |
|
Characteristics of the Page |
|
Explanation |
|
|
|
Q&A page about Native |
|
• |
Lacking expertise; not very trustworthy or |
|
|
|
|
American customs |
|
|
authoritative for the purpose of the page |
|
There are 94 answers to this question with a few |
|
|
|
|
|
|
|
results that seem helpful. Many of the posts are |
|
|
|
|
|
|
|
wrong or misleading, including the top answer, |
|
|
|
|
|
|
|
which is labled the “best answer.” |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This forum is about dance topics. Many |
|
|
|
|
|
|
|
participants have everyday experience washing |
|
|
Forum page on how to |
|
|
|
|
ballet shoes and make recommendations based |
|
|
|
|
|
|
on their experience. |
|
|
|
wash ballet shoes |
|
• |
Mixed, but with some redeeming qualities |
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
This page is “mixed” because there is distracting |
|
|
|
|
|
|
|
content which makes it hard to read the MC, |
|
|
|
|
|
|
|
potentially because the page uses an older and/or |
|
|
|
|
|
|
|
outdated design. The page also has distracting |
|
|
|
|
|
|
|
buttons and Ads. On the other hand, there is |
|
|
|
|
|
|
|
some valuable everyday expertise and helpful |
|
|
|
|
|
|
|
MC, making Medium a good rating for this page. |
|
|
|
|
|
|
|
|
|
|
Q&A page about whether a |
|
• |
Everyday expertise for the purpose of the |
|
Many participants share their personal |
|
|
Roomba will work |
|
|
page |
|
experiences with these products, giving details |
|
|
|
|
|
|
|
such as how well certain models work with pet |
|
|
|
|
|
|
|
hair. There are many descriptions of participants’ |
|
|
|
|
|
|
|
own experiences with this product and how well it |
|
|
|
|
|
|
|
works for them. |
|
|
|
|
|
|
|
|
|
|
Q&A page about |
|
• |
Everyday expertise for the purpose of the |
|
The answer on this page is written by an |
|
|
|
|
employee of Build-A-Bear, so it offers a somewhat |
|
|||
|
interviewing at Build-A-Bear |
|
|
page |
|
unique and presumably somewhat expert answer. |
|
|
|
|
|
|
|
The page design allows users to read the MC as |
|
|
|
|
|
|
|
the Ads are clearly labeled, as well as both the |
|
|
|
|
|
|
|
question and answer. |
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 63
|
Webpage/Type of Content |
|
Characteristics of the Page |
|
Explanation |
|
|
|
Forum post on the magic |
|
• |
High level of expertise; authoritative and |
|
The person who posted the first message on this |
|
|
|
|
forum page provides a helpful resource on how to |
|
|||
|
loop technique in knitting |
|
|
trustworthy for the purpose of the page |
|
|
|
|
|
|
|
master the magic loop technique in knitting. She |
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
also shares pictures of her own version using an |
|
|
|
|
|
|
|
old pair of blue jeans. With over 20 years of |
|
|
|
|
|
|
|
experience knitting socks, we would consider her |
|
|
|
|
|
|
|
to be an expert on the topic. |
|
|
|
|
|
|
|
|
|
|
Q&A page about how long |
|
• |
Everyday expertise |
|
The question on the page asks how long people |
|
|
|
|
live with cancer. There are many results |
|
|||
|
most cancer patients live |
|
• |
A satisfying amount of high quality MC |
|
|
|
|
|
|
describing how long a loved one lived after |
|
|||
|
|
|
|
|
|
diagnosis. There is very little medical advice and |
|
|
|
|
|
|
|
the focus of the page is sharing personal |
|
|
|
|
|
|
|
experience. Many results are heartfelt and well |
|
|
|
|
|
|
|
written. |
|
|
|
|
|
|
|
|
|
|
|
|
• |
High level of expertise for this topic; |
|
This forum is well known for discussions on luxury |
|
|
|
|
|
authoritative and trustworthy for the purpose |
|
|
|
|
Forum page on |
|
|
|
designer purses. On this particular forum page, |
|
|
|
|
|
of the page |
|
|
||
|
authenticating a purse |
|
|
|
members are consulting forum experts who have |
|
|
|
|
• |
A satisfying amount of high quality MC |
|
|
||
|
|
|
|
expertise authenticating bags from this brand. |
|
||
|
|
|
|
|
|
These experts can tell if a particular bag is |
|
|
|
|
|
|
|
authentic or fake. While there is an ad at the top |
|
|
|
|
|
|
|
and a few ads within the forum message, it does |
|
|
|
|
|
|
|
not distract from the MC, which is easy to find. |
|
|
|
|
|
|
|
|
|
|
|
|
• |
High level of expertise for this topic; |
|
This forum is well known for its community of |
|
|
Forum page on KPIs to |
|
|
authoritative and trustworthy for the purpose |
|
|
|
|
track |
|
|
of the page |
|
experts on quailty assurance. The responses are |
|
|
|
|
• |
A satisfying amount of high quality MC |
|
authoritative and trustworthy for the purpose of |
|
|
|
|
|
|
|
the page. While there are Ads on the page, it is |
|
|
|
|
|
|
|
easy to find the MC and they are clearly labeled |
|
|
|
|
|
|
|
as “Sponsored Link.” |
|
|
|
|
|
|
|
|
|
|
|
|
• |
High level of expertise for this topic; |
|
This discussion focuses on the landscaping for a |
|
|
Forum page on landscaping |
|
|
authoritative and trustworthy for the purpose |
|
|
|
|
|
|
|
particular paludarium (an aquarium with terrestrial |
|
||
|
an aquarium |
|
|
of the page |
|
|
|
|
|
|
|
and aquatic elements). There is a lot of |
|
||
|
|
|
• |
A satisfying amount of high quality MC |
|
|
|
|
|
|
discussion and interaction between forum |
|
|||
|
|
|
|
|
members about the types of materials and |
|
|
|
|
|
|
|
species used in the aquarium. The posts show |
|
|
|
|
|
|
|
expertise in a niche topic aquarium landscaping. |
|
|
|
|
|
|
|
|
|
|
Proprietary and Confidential – Copyright 2015 64
|
Webpage/Type of Content |
|
Characteristics of the Page |
|
Explanation |
|
|
|
|
|
• |
High level of expertise for this topic; |
|
|
|
|
|
|
|
authoritative and trustworthy for the purpose |
|
Many participants share advice and personal |
|
|
Forum page on cloth |
|
|
of the page |
|
|
|
|
|
|
|
experiences on how to clean cloth diapers, |
|
||
|
diapers |
|
• |
A satisfying amount of high quality MC |
|
|
|
|
|
|
including their own specific washing routine. This |
|
|||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
forum site is an excellent resource for this type of |
|
|
|
|
|
|
|
expertise. |
|
|
|
|
|
|
|
|
|
|
|
|
• |
A satisfying or comprehensive amount of |
|
|
|
|
Q&A page where Hillary |
|
|
high quality MC |
|
Since the question asks for ordinary people’s |
|
|
Clinton asks users for their |
|
• |
Very high level of expertise, highly |
|
|
|
|
|
|
opinions, everyday expertise is OK. Many of the |
|
|||
|
personal opinions |
|
|
authoritative/highly trustworthy for the |
|
|
|
|
|
|
|
answers are well written and thoughtful. It is clear |
|
||
|
|
|
|
purpose of the page |
|
|
|
|
|
|
|
|
that many people spent time answering this |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
question, doing their best to help Hillary Clinton |
|
|
|
|
|
|
|
understand their perspectives. |
|
|
|
|
|
|
|
|
|
|
|
|
• |
Very high level of expertise, highly |
|
|
|
|
|
|
|
authoritative/highly trustworthy for the |
|
The purpose of the page is to answer questions |
|
|
Q&A page on abbreviations |
|
|
purpose of the page |
|
|
|
|
|
|
|
about the usage of abbreviations. This website |
|
||
|
|
|
• |
Very positive reputation, and the website is |
|
|
|
|
|
|
|
has a very good reputation as a reference for |
|
||
|
|
|
|
responsible for all content on the website |
|
|
|
|
|
|
|
|
information on writing, publishing, etc. It is |
|
|
|
|
|
|
|
|
considered highly authoritative and trustworthy for |
|
|
|
|
|
|
|
the topic of the page. |
|
|
|
|
|
|
|
|
|
