Mixing Crowds, Computers, and Experts for Scalable Annotation of Biomedical Images
Biomedical researchers are running image-based studies to systematically study fundamental biological processes. The larger goal of this effort is to contribute to discoveries and innovations that, for example, address society’s health care problems or lead to new bio-inspired technology. However, the key bottlenecks for extracting the desired information from images lie in unreliable annotation from algorithms and costly annotation by experts, especially at scale. Given the rise of crowdsourcing, I will discuss how we can utilize online crowds to better annotate biomedical images. I will present research on demarcating objects in images (segmentation), a critical and time-consuming precursor to many downstream applications. I will begin the talk with a detailed analysis of the relative strengths and weaknesses of three different image segmentation approaches: by experts, by crowd workers, and by algorithms. Then, I will describe a hybrid system design for intelligently distributing segmentation efforts between algorithms and crowds. Results show how to efficiently leverage crowd and algorithm efforts in order to optimize cost/quality trade-offs as well as how to produce segmentations comparable to those created by experts.