Facebook To Audio-Describe Photos

Facebook has launched a tool that describes photographs to visually impaired users. However, it seems to be keeping expectations low.

The tool is the work of Facebook engineer Matt King, who lost his sight through a medical condition. He wanted to deal with the way screen readers currently “speak” the name of the person who uploaded the picture, along with any comments, but in between simply says “photo.”

The automatic alt text tool won’t attempt to describe the image in detail as such. Instead it will simply list a range of items that appear to be in the photograph.

While that’s clearly more efficient than manual descriptions (which simply wouldn’t be viable given more than two billion photos are uploaded to Facebook-owned services every day), it will make it limited to items in the database.

At launch that covers around 100 items, described as concepts including:

• people’s appearance (e.g., baby, eyeglasses, beard, smiling, jewelry);
• nature (outdoor, mountain, snow, sky);
• transportation (car, boat, airplane, bicycle);
• sports (tennis, swimming, stadium, baseball); and food (ice cream, pizza, dessert, coffee).

There’s also a database of settings including:

• people (e.g., people count, smiling, child, baby);
• objects (car, building, tree, cloud, food);
• settings (inside restaurant, outdoor, nature); and other image properties (text, selfie, close-up).

The system is set up to only include a concept or setting in the description when there’s at least an 80 percent confidence it is correctly identified. According to Facebook, at the moment more than half of pictures will have at least one item listed.

To make the description more useful, after an “Item may contain” disclaimer, the list will be presented with the number of people first, along with whether they are smiling. The list then details any detected objects and then finally any concepts. The idea is that, while hardly natural language, this most closely matches the way a human would describe the content of an image.

Automatic alt text will initially be added to English language descriptions on the iOS Facebook app before being rolled out more widely.