9. Rating on Your Phone Issues

We understand that raters using different phones, operating systems, and browsers may have different experiences. In general, do what you would do naturally, and rate based on your experience. Here are some additional points to keep in mind when you are rating on your phone:

• If you see one of these messages when you open a landing page on your phone:

15. Asks whether you want to open the page in the browser or the website’s app: select and evaluate the webpage. However, if the result automatically opens an app by default based on your phone’s

settings (e.g. some users have set all YouTube pages to automatically open the result in the YouTube app), you should rate your natural app experience—you do not have to change your phone’s default settings.

15. Asks whether you want to visit the mobile page or desktop page: it is fine to select the mobile page as long as the landing page is the same. Sometimes, the mobile option will bring you to the mobile homepage instead of the specific URL in the task. You may need to check that the mobile page is in fact the same URL as the desktop page.

• Please open and look at PDF files. Your experience may be easier or harder than other users depending on your phone and browser, but you should still open the PDF file and look at it.

• Occasionally, you may be assigned some of these rating tasks on a desktop computer, but please rate from the perspective of a mobile user unless otherwise instructed.

• When rating Device Action queries and Device Action Result Blocks, including queries for installing or opening apps, please assume that queries were issued on an Android device unless explicitly stated otherwise in the instructions.

Proprietary and Confidential – Copyright 2015 86

Part 3: Needs Met Rating Guideline

0. Rating Using the Needs Met Scale

There are many different kinds of queries and results, but the process of rating is the same: Needs Met rating tasks ask you to focus on mobile user needs and think about how helpful and satisfying the result is for the mobile users.

This is what the Needs Met rating slider looks like:

	Rating		Description
	Fully Meets (FullyM)		A special rating category, which only applies to certain queries and results. All or almost all mobile
			users would be immediately and fully satisfied by the result and would not need to view other results
			to satisfy their need.
	Highly Meets (HM)		Very helpful for many or most mobile users. Some users may wish to see additional results.
	Moderately Meets (MM)		Helpful for many users OR very helpful for some mobile users. Some or many users may wish to see
			additional results.
	Slightly Meets (SM)		Helpful for fewer mobile users. There is a connection between the query and the result, but not a
			strong or satisfying connection. Many or most users would wish to see additional results.
	Fails to Meet (FailsM)		Completely fails to meet the needs of the mobile users. All users would wish to see additional results.

Please note that you may assign in-between ratings. Use in-between ratings if you think the rating of a result falls between two labels. You can either drag the slider or click on the point that you want the slider to land on.

Proprietary and Confidential – Copyright 2015 87