Prototype tasks: Improving crowdsourcing results through rapid, iterative task design
Conference or workshop item
Gaikwad, S.S., Chhibber, N., Sehgal, V., Ballav, A., Mullings, C., Nasser, A., Richmond-Fuller, A., Gilbee, A., Gamage, D., Whiting, M., Zhou, S., Matin, S., Niranga, S., Goyal, S., Majeti, M., Srinivas, P., Ginzberg, A., Mananova, K., Ziulkoski, K., Regino, J., Sarma, S., Sinha, A., Paul, A., Diemer, C., Murag, M., Dai, W., Pandey, M., Vaish, R. and Bernstein, M. 2017. Prototype tasks: Improving crowdsourcing results through rapid, iterative task design.
Authors | Gaikwad, S.S., Chhibber, N., Sehgal, V., Ballav, A., Mullings, C., Nasser, A., Richmond-Fuller, A., Gilbee, A., Gamage, D., Whiting, M., Zhou, S., Matin, S., Niranga, S., Goyal, S., Majeti, M., Srinivas, P., Ginzberg, A., Mananova, K., Ziulkoski, K., Regino, J., Sarma, S., Sinha, A., Paul, A., Diemer, C., Murag, M., Dai, W., Pandey, M., Vaish, R. and Bernstein, M. |
---|---|
Description | Low-quality results have been a long-standing problem on |
Keywords | Crowdsourcing; Task deisgn; Microtasks |
Year | 23 Oct 2017 |
Conference | The fifth AAAI Conference on Human Computation and Crowdsourcing |
Official URL | https://arxiv.org/abs/1707.05645 |
Web address (URL) of conference proceedings | https://www.humancomputation.com/2017/papers.html |
Related Output | |
Cites | arXiv: 1707.05645 |
References | Baecker, R. M. 2014. Readings in Human-Computer Interaction: toward the year 2000. Morgan Kaufmann. Bernstein, M. S.; Teevan, J.; Dumais, S.; Liebling, D.; and Horvitz, E. 2012. Direct answers for search queries in the long tail. In Proceedings of the SIGCHI conference on human factors in computing systems, 237–246. ACM. Callison-Burch, C., and Dredze, M. 2010. Creating speech Cheng, J.; Teevan, J.; and Bernstein, M. S. 2015. Measuring crowdsourcing effort with error-time curves. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 205–221. ACM. Christoforaki, M., and Ipeirotis, P. 2014. Step: A scalable testing and evaluation platform. In Second AAAI Conference on Human Computation and Crowdsourcing. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and FeiFei, L. 2009. Imagenet: A large-scale hierarchical image database. In Proc. CVPR ’09. Dix, A. 2009. Human-computer interaction. Springer. Design method: I like, i wish, Gaikwad, S. N.; Morina, D.; Nistala, R.; Agarwal, M.; Cossette, A.; Bhanu, R.; Savage, S.; Narwal, V.; Rajpal, K.; Regino, J.; et al. 2015. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 101–102. ACM. Harper, E. R.; Rodden, T.; Rogers, Y.; Sellen, A.; Human, B.; et al. 2008. Human-computer interaction in the year Hinds, P. J. 1999. The curse of expertise: The effects of expertise and debiasing methods on prediction of novice performance. Journal of Experimental Psychology: Holtzblatt, K.; Wendell, J. B.; and Wood, S. 2004. Rapid Ipeirotis, P. G.; Provost, F.; and Wang, J. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, 64–67.ACM. Ipeirotis, P. G. 2010. Analyzing the amazon mechanical turk marketplace. In XRDS: Crossroads, The ACM Magazine for Students - Comp-YOU-Ter, 16–21. ACM. Irani, L. C., and Silberman, M. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 611–620. ACM. Khanna, S.; Ratan, A.; Davis, J.; and Thies, W. 2010. Evaluating and improving the usability of mechanical turk for low-income workers in india. In Proceedings of the first ACM symposium on computing for development, 12. ACM. Kim, J., and Monroy-Hernandez, A. 2016. Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 1018–1027. ACM. Kittur, A.; Nickerson, J. V.; Bernstein, M.; Gerber, E.; Shaw, A.; Zimmerman, J.; Lease, M.; and Horton, J. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, 1301–1318. ACM. Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing Lasecki, W. S.; Kushalnagar, R.; and Bigham, J. P. 2014. Le, J.; Edmonds, A.; Hester, V.; and Biewald, L. 2010. Ensuring quality in crowdsourced search relevance evaluation. In SIGIR workshop on crowdsourcing for search eval. Marcus, A., and Parameswaran, A. 2015. Crowdsourced data management industry and academic perspectives. Foundations and Trends in Databases 6(1-2):1–161. Martin, D.; Hanrahan, B. V.; O’Neill, J.; and Gupta, N. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, 224–235. ACM. McInnis, B.; Cosley, D.; Nam, C.; and Leshed, G. 2016. Metaxa-Kakavouli, D.; Rusak, G.; Teevan, J.; and Bernstein, M. S. 2016. The web is flat: The inflation of uncommon experiences online. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2893–2899. ACM. Mitra, T.; Hutto, C.; and Gilbert, E. 2015. Comparing Nielsen, J. 2000. Why you only need to test with 5 users. Preece, J.; Rogers, Y.; Sharp, H.; Benyon, D.; Holland, S.; and Carey, T. 1994. Human-computer interaction. AddisonWesley Longman Ltd. Ross, L. 1977. The intuitive psychologist and his shortcomings: Distortions in the attribution process. Advances in experimental social psychology 10:173–220. Rzeszotarski, J. M., and Kittur, A. 2011. Instrumenting Sharp, H.; Jenny, P.; and Rogers, Y. 2007. Interaction design:: beyond human-computer interaction. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 614–622. ACM. Stephane, K., and Morgan, J. 2015. Hire fast & build things: How to recruit and manage a top-notch team of distributed engineers. In Upwork. Von Ahn, L., and Dabbish, L. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference Wilson, J. Q., and Kelling, G. L. 1982. Broken windows. Wu, S.-Y.; Thawonmas, R.; and Chen, K.-T. 2011. Video |
Additional information | Stanford Crowd Research Collective |
Publication process dates | |
Deposited | 16 Nov 2020 |
Files | File Access Level Open |
https://repository.canterbury.ac.uk/item/8wqv8/prototype-tasks-improving-crowdsourcing-results-through-rapid-iterative-task-design
Download files
Files
Prototype Tasks_ Improving Crowdsourcing Results through Rapid Interative Task Design.pdf | ||
File access level: Open |
67
total views60
total downloads2
views this month0
downloads this month
Export as
Related outputs
Utilization of ChatGPT in CDIO projects to enhance the literacy of international students
Manna, S., Williams, S., Richmond-Fuller, A. and Nortcliffe, A. 2024. Utilization of ChatGPT in CDIO projects to enhance the literacy of international students.![](/~419/qpk/default-thumbnail.png)