Imagine internet browsing without the ability to make sense of images. It鈥檚 a problem that visually impaired computer users face every day. While screen reading technology gives users audible access to written content, it needs written descriptions to interpret images, and often there isn鈥檛 any.
Some website developers include descriptions of images in the code (called 鈥渁lt text鈥�) because it improves their websites鈥� search engine rankings. However, there鈥檚 no mechanism for determining whether these descriptions are accurate or informative. As a result, developers often enter one-word descriptions such as 鈥渋mage鈥� or 鈥減hoto,鈥� leaving the visually impaired with no useful information about the image.
To help address this problem, an ATLAS Institute researcher developed a system that collects captions and alt text associated with other instances of the same photo elsewhere online, associating human-authored descriptions with every website where it appears. Called Caption Crawler, the image captioning system compiles descriptions in a database: if a photo has never been queried, it will offer alt text in about 20 seconds; if the photo has previously been processed, alt text is available almost immediately.
The technology was developed by Darren Guinness, a PhD student in the ATLAS Interactive Robotics and Novel Technologies (IRON) Lab and the Superhuman Computing Lab, working in conjunction with Microsoft Research鈥檚 Edward Cutrell and Meredith Ringel Morris. The research, which merges the benefits of a fully automated system with the quality of human-authored content, will be presented at the Association for Computing Machinery鈥檚 (ACM) 2018 Conference on Human Factors in Computing Systems (CHI) in Montreal in April.
Users who want Caption Crawler to replace poor-quality alt text, press a keyboard shortcut to request a replacement. 听The screen reader automatically speaks the new caption, which is the longest caption found for a particular photo. Users can also use a different shortcut to access any additional found captions.
Caption Crawler only works with images used on multiple websites, but the approach is effective because about half of website administrators provide informative photo descriptions, Guinness says.
鈥淎lthough this approach cannot caption unique images that only appear in a single place online, it can increase the accessibility of many online images,鈥� he says. 鈥淐aption Crawler is a low latency, incredibly low-cost solution to a big problem. It produces human-quality captioning without incurring additional costs in human labeling time.鈥�
Caption Crawler combines a Google Chrome Browser Extension with a Node.js cloud server. The browser extension searches the Document Object Model (DOM) 听of the active webpage for image tags and background images, which are then sent to the server for caption retrieval. When Caption Crawler finds a caption for an image, the caption is streamed back to the browser extension, which then associates the caption to the image.
Research shows humans produce higher quality captions than automated computer and machine-learning based approaches, Guinness says. Caption Crawler uses a hybrid system that captures both, prioritizing human captioning over machine learning and computer vision-based approaches. If no human-authored captions can be found, computer-generated captions from Microsoft鈥檚 CaptionBot are used to describe the image. When the text from CaptionBot is read aloud, the screen reader first speaks the words 鈥淐aptionBot,鈥� so that the user is aware that the caption is not human-authored.
鈥淗ybrid systems that meld both human-quality text and machine learning approaches hold a lot of promise for improving access to online media,鈥� Guinness says.
听