Online studies are an attractive alternative to the labor- intensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon’s Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one of which was identical to a previous study performed in our lab. We report on our experiences and present ways to avoid common problems by taking them into account in the study design, and taking advantage of Mechanical Turk’s features.

  1. BELIV
    Do Mechanical Turks Dream of Square Pie Charts?. Robert Kosara, and Caroline Ziemkiewicz In Proceedings BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV), pp. 373–382, 2010 Abstract PDF