Using the Internet to Revolutionize Search and Rescue

By Kenneth Barbalace

[Tuesday, September 18, 2007]

For over a week now I have been participating in the Internet search for lost aviator Steve Fossett using Amazon.com's Mechanical Turk and Google Earth. The premise is simple. Via Mechanical Turk, anyone anywhere in the world who has an Internet connection can help in the search for Steve Fossett by joining Mechanical Turk and reviewing little squares of updated satellite photos which represent a swath of land which is around 7,225 sq/m (1.78 acres) in size.

If there is something of interest on the photo the searcher can then double check the Mechanical Turk photo against updated and old satellite photos in Google Earth using a special KML file. The KML file provides Google Earth with updated satellite photos of the search area which are overlaid on top of the normal older satellite photos. The searcher can then zoom in to the area of interest and turn on and off the overlay. If an object exists in the new photos, but does not exist in the old photos, it might be of interest. If, however, the object of interest exists in both the old and new photos, then it can not be Steve Fossett's plane. If a searcher does come across a "HIT" in Mechanical Turk that the searcher feels merits further review, they can flag the photo and add comments via a web form directly below the satellite photo on the Mechanical Turk webpage containing the photo of interest. These flagged HITs are then reviewed and filtered by trained individuals, with the most promising leads being forwarded on to the physical search and rescue efforts. To help reduce the odds of Steve Fossett's plane being overlooked, Mechanical Turk feeds each photo or "HIT" to several different searchers so that each photo is looked at several times.

Being able to allow thousands of volunteers to comb through thousands of square miles of satellite photos could in the long run revolutionize the way search and rescue is conducted. No longer would searches be restricted to daylight hours, subject to bad weather, limited by available aircraft, limited by the logistics of housing and feeding hundreds of on the ground searchers, etc. Anyone in the world could participate from the comfort of their home day and night. They would simply need to be provided updated satellite photos and a structured method of sifting through all of the photos.

The Steve Fossett search, however, is showing that there is a long way to go before "crowdsourcing" searches via the Internet will become a truly effective tool that is of a true value without being an unwelcome distraction to the physical search and rescue operation itself. In a way, the Steve Fossett search is a proving ground that will hopefully provide valuable insights as to how to build truly effective tools to conduct these searches and how to manage the flow of tips from untrained volunteers through teams of trained searchers who can sift out the most promising leads and pass them on to the physical search and rescue operation.

My searching experience

In my first week of participating in the Steve Fossett search (Monday 9/10/2007 – Friday 9/14/2007) I spent around 50 to 56 hours searching and completed over 15,000 Mechanical Turk "HITs". Of those HITs, I found no photos of interest that could not be ruled out by comparing the new photos against old photos using Google Earth. I frequently came across light or dark airplane like crosses in photos, but by simply taking the time to sleuth them out and compare the old and new photos against each other I quickly discovered the shape to be a combination of geologic formations, vegetation and/or shadows. Now 15,000 photos may seem like a lot of photos to look at, but think of it this way, that works out to only around 108 sq/km (~42 sq/mi) in a total search area that is around 42,000 sq/km (17,000 sq/mi) in size. I did come across one photo that had a twin engine commercial jet (e.g. maybe a Boeing 737) flying through when the picture was taken. From what I have read all confirmed sightings of planes via Mechanical Turk have turned out to be planes flying though when the satellite photo was taken.

False reports and reports bypassing proper channels frustrating for CAP

One of the great frustrations I have seen expressed by CAP (Civil Air Patrol) officials in regards to the Google Earth search for Fossett has been the steady stream of false leads being fed to them directly from the public outside of the proper Mechanical Turk channels. People are so eager to find Steve Fossett that their eyes begin to see what their mind desires to find. On top of this many searchers appear to be using Google Earth directly without using the Mechanical Turk part of the equation and often times they don't even have the updated satellite photos. Then when they do find things that they think are Steve Fossett's plane, they report it directly to the CAP and/or their local newspaper rather than allowing the lead to flow through proper channels that have been set up to filter all of these leads.

I have looked at many of the coordinate locations that were reported online in either news articles or blog/forum discussions and invariably, I could quickly rule out the sighting by simply looking at old satellite photos. In many cases, the reported sighting wasn't even in an area that was covered by up to date satellite photos via the special KML file overlays, thus whatever it was couldn't be Steve Fossett because the photo had been taken before he went missing.

Don't get me wrong, if people participating in this search think they see something that needs further attention, they should flag it and leave some comments with the photo in question via Mechanical Turk. I just don't think people should try to go outside of proper channels to report their possible findings.

What can be learned from the Internet search for Steve Fossett

To be successful, search and rescue can not be a freelance operation. It must be structured and organized. In order for the Internet to become a successful tool for harnessing the altruistic nature of people and their desire to help in a search, it must be structured in a way that allows Internet generated leads to be filtered properly. Volunteer searchers can't just go surfing satellite photos looking for "interesting" objects in an unstructured way even if the provided photos through the structured search are boring or tedious (trust me I've seen my share of flat featureless photos in the past week). Even the boring photos need to be reviewed and ruled out in a systematic way.

I stated that the Steve Fossett search is a proving ground to help develop the potential of using the Internet to revolutionize search and rescue. Having looking at almost 16,000 Mechanical Turk HITs, I have compiled a list of things that need to be refined to really make this an effective method to conduct searches:

1. First and foremost, the resolution of the satellite photos is way too low to be really effective. Currently the best resolution provided by commercial satellites is about one square meter. This means that on one of these photos the overstuffed arm chairs in my home would only appear to be a single blurry pixel. This is not enough detail to really make out what objects are. I read the other day that a new generation of commercial satellites are getting ready to be launched, which could improve the resolution of photos by an order of magnitude. This would mean my laptop computer might now show up as a single pixel and my chair could be a collection of pixels. This would be tremendously valuable in reducing the number of false hits as people could get a better idea of what they are looking for and if a plane was torn up into a debris field with lots of small pieces, it would be easier to spot.
2. The interface for reviewing photos desperately needs to be improved. Although great credit should be given to Amazon's Mechanical Turk, a web page based application may be the wrong tool for the job. Mechanical Turk is too clunky, requires too much scrolling of HIT pages and requires too many mouse movements/clicks. If the entire search area for Steve Fossett were to be searched using Mechanical Earth, there would be over six million HITs just to look at everything once. To look at the entire area say four time (each tile by four different people) would require 24 million HITs. This means people need to be able to review and sift through the HITs as quickly and efficiently as possible, which requires there to be as little scrolling, mouse movement and clicks as possible. It is also too cumbersome to switch back and forth between Mechanical Turk and Google Earth. I think the best solution would be for Google to bring the entire process into the Google Earth application itself. Google could create a sub-function of Google Earth or a derivative application called say (Google Earth SAR) that could feed people the photos to be looked at and allow the individual to easily compare and contrast new and old photos. Ideally individual photos could be reviewed and rejected with as few as one key stroke (say the "N" key). If the individual wanted to submit a possible finding they could press the "Y" key and then be prompted to provide further details. A two step process for submitting positive findings could help prevent accidental submission of possible leads. In my case, this would have help me advert some accidental lead reports via Mechanical Turk, because my mouse cursor had moved when I was looking at photos and not my mouse. As a result, I clicked on yes when I meant no.
3. Updated satellite photos should only be available to those who are actually participating in the search using the proper tools and reporting channels (this is another reason to bring everything together under one Google Earth application). It must also be made clear that NO satellite photo crash sightings would be accepted outside of the proper channels. This would significantly reduce the amount of noise and distraction experienced by the physical search and rescue teams because of improper Internet sighting reports.
4. It is important to know more about the aircraft being searched for. I did not learn until late last week that the airplane Fossett was flying was a fabric covered aircraft instead of having an aluminum skin. This could have been an important detail in helping one spot a mangled wreck.

Unexpected personal side effects

While participating in the Mechanical Turk search for Steve Fossett, I have been closely following news stories about the search (I don't want to keep searching if he has been found). Last week I started to get frustrated by news stories featuring possible Fossett sightings that, when the photo was looked at closely, clearly wasn't an airplane. I commented on one such report on the Wired.com blog "Danger Room" using my real name (Ken Barbalace) as I typically do when posting comments (see: Geeks spot Fossett). My name is fairly unique and this combined with my extensive web presence makes me easy to locate via Google. As a result of my comments, later that day I was called by a reporter from the Chicago Sun Times and then an hour later by the San Francisco Chronicle, which each then published stories featuring me. The Sun Times article by Frank Main was "Steve's eye in the sky" and the San Francisco Chronicle article by Henry K. Lee was "Thousands of Mechanical Turk users searching for Fossett's plane". My local paper the Portland Press Herald saw these articles and called me asking for an interview. They sent a reporter and photographer to my home to interview me for an article they published on the front page (below the fold) on Saturday 9/15/2007. The article by David Hench was titled "Searching from the Sofa: Portland techie joins hunt for missing adventurer". This article resulted in articles being published in the Boston Globe ("Search for Fossett spread to the Internet" and the Montpelier VT paper The Times Argus ("Search for Fossett spreads to the Internet"). The Boston Globe version of the article was then picked up on Sunday by the Lewiston Sun Journal, another local paper based in Lewiston Maine ("Via Internet, Mainer helps in Fossett search".

Talk about wild side effects, who would have thought that some simple comments in a blog post about Steve Fossett would lead to all of this attention? I certainly didn't. I was just leaving comments to blog posts like I always do while participating in something that I found really intriguing and demonstrated the great potential of the Internet.

Continuing the search

I did read last night that the authorities are scaling back the primary search and rescue efforts so it may be that the last best hope of finding Steve Fossett's plane is in the hands of those participating in the Mechanical Turk search. If you have time a keen eye and would like to participate in the Internet search for Fossett, you can go to http://www.mturk.com to sign up. Well I guess it is time to stop writing and go back to my staring at patches of sage brush, glaciers, and airplane shaped illusions/shadows.