Internet search for Steve Fossett eight weeks later

By Kenneth Barbalace

[Wednesday, October 31, 2007]

After eight weeks, the Internet search for Steve Fossett, which I have been participating in, via Amazon's Mechanical Turk has been called off (although other search efforts continue). Back on Sept. 18, 2007 I wrote with great enthusiasm how I thought this type of collaborative effort to review satellite photos via the Internet could revolutionize search and rescue (see: "Using the Internet to Revolutionize Search and Rescue").

After eight weeks of searching I still think that this type of distributed searching could help revolutionize search and rescue, but much of my original enthusiasm has been dampened. In order for collaborative Internet searches to be a useful option for search and rescue efforts, a lot of changes will need to be made to both the search tool and the overall process.

The biggest factor that doomed this effort to failure was the resolution of the imagery being used. The initial satellite photos being used had a resolution of one meter, which means you could identify an intact aircraft or vehicle, but a broken up aircraft would have been extremely difficult to discern from natural features. It was not until the last few weeks that we started working with imagery captured by specially fitted aircraft that we started to work with aerial photos that were of a sufficient resolution to clearly discern individual objects as small as individual sagebrush bushes. Being able to discern the smaller objects really did make a tremendous difference. Had this higher resolution of imagery been used early on there might have been a realistic chance of spotting Steve Fossett's plane had it crashed in an area being searched. What should be taken from this is that unless one is looking for an intact aircraft, one probably shouldn't rely on satellite photos of a one meter resolution. It would be much better to get aerial photography that has a resolution of at least 25 cm.

A factor that was a significant hindrance to the entire search effort was the need to use two different applications to review and report findings of individual photos. Via a webpage interface Amazon's Mechanical Turk would assign photos to be reviewed, unfortunately the area covered by the photos was small and it was not easy to compare new photos against old photos within the web browser. Early on in the search effort, master Google Earth KML overlay files were made available to facilitate the comparing of old and new imagery, but near the end of the search effort master Google Earth KML overlay files stopped being provided for areas being actively reviewed. Initially this meant that there was no way to compare old and new photos. Then a Google Maps applet was added to HIT pages, which was supposed to help provide the ability to compare old and new imagery, but the scale between the old images and the new images were way too different and the applet containing old imagery was completely useless for comparison purposes.

After some discussions with others who were participating in the search on a forum being used to discuss this effort, I was able to reverse engineer the logic behind the naming scheme of the aerial imagery being used and I figured out how to capture the coordinates needed for lining up the images within Google Earth. With a little trial and error and a lot of research, I was able create a script that generated Google Earth KML overlay files on the fly that would present the aerial image being reviewed in MTurk along with the adjoining photos in Google Earth. This gave individuals a means to effectively review old and new photos. The problem was that this required the user to take several steps to capture the required information from the MTurk HIT they were working on and provide it to my KML overlay generator. What was really needed was for the MTurk HITs to link directly to a KML overlay generator such that users would not need to do several steps just to be able to compare old and new images.

While the real reason is not known, there has been a lot of speculation as to why those running the MTurk search effort stopped providing Google Earth overlays to assist in the comparison of old and new photos. One possibility is that this was done to better protect the integrity of any crash site as one could easily get the exact coordinates of any object located within Google Earth. There did seem to be the potential for a few treasure hunters and/or glory seekers participating in the search effort. There might have been a real concern that one of these parties would take it upon themselves to physically investigate any reported sightings and disrupt the integrity of the crash site (either intentionally or unintentionally). Simply put, the comparison of old and new photos was absolutely critical, however, the way the MTurk search effort as it was set up, this comparison had to be done within Google Earth, which made the coordinates of any object found extremely easy to determine. It can not be over emphasized how important it is to be able to compare old and new photos. It is simply the best way to rule out potential objects of interest. In fact, if people had effectively compared new photos against old photos, probably upwards of 90% of objects that were flagged for further review could have been ruled out and not reported. Yet at the same time it is important to protect the crash site.

Beyond the technical challenges posed by the search tools being used, another very serious hindrance to the search effort was the complete inabilities of many participants to be able identify the difference between natural objects like dead trees, rocks, shadows, etc. and foreign objects that did not belong like airplane wreckage. While it is difficult to know what any crash site looks like before it is discovered, it is critical for those analyzing photos to be able to rule out natural objects. Yet throughout much of the search effort there were precious few photos provided that truly helped people understand what it was they were looking at. It was not until the lpast couple of weeks when there started to be good ground photos provided of various objects like exposed rock outcroppings, dead trees, sagebrush, etc. Some of the volunteers who had been participating in the MTurk search effort went out to the area of Nevada being searched and took pictures of various objects like exposed rock outcroppings, dead trees, sagebrush, etc. and posted them along with the coordinates for each photo on the forum being used to discuss the search. These photos gave the rest of us a much better idea of what we were looking at, which was most helpful.

In my own case over a period of around eight weeks of daily searching, I reviewed over 25,500 Mechanical Turk HITs, of these HITs, I ended up reporting only two as being worth further investigation. The first HIT I reported I later decided was probably just an odd shadow cast by a geologic formation. The second HIT I reported I suspected was most likely a large farm implement, but I could not definitively rule it out so I flagged the object out of caution.

Early on, a major problem was the failure of people to take time to read and understand what instructions were provided. Admittedly not enough instructions or tutorials were provided, but the ones that were provided were important. For instance, one of the most important instructions was about how to report findings online so that they could be reviewed and evaluated by a team dedicated to this purpose. Rather than reporting findings through proper channels, some people had a propensity of thinking that their findings were so important that they took their findings directly to CAP or Nevada law enforcement officials. This improper reporting of findings created an excessive level of distractions for search authorities, which again did not endear the Mechanical Turk and Google Earth search efforts to search officials. Later on this didn't appear to be as big of an issue as there was only a small corps of really dedicated searchers who were still reviewing imagery by the end.

Could collaborative photo analysis via the Internet be an effective aid in search and rescue efforts? Yes I believe it could, but the lessons learned with the Steve Fossett search need to be used to build better search tools and possesses:

1. The search tool being used to review photos needs to be a single integrated application that allows reviewers to very easily compare old and new photos and then report their findings in one place. Having to jump between two different applications is too cumbersome.
2. The tool must allow for the comparison of old and new photos while protecting the integrity of the crash site once it is located as such coordinates should only be accessible by search officials (currently it is not possible to really hide coordinates).
3. There must be robust tutorials that include actual photos from the search area with on ground examples of the flora and geologic formations of the area as well as the aerial views of these objects so that people can better understand what they are looking at.
4. Higher resolution photos must be used than was provided initially for the Steve Fossett search effort. One meter resolution is not of a high enough quality to be useful for this task.
5. There must be effective and regular communication from the top down such that volunteer searchers do not get frustrated and quit.
6. There must be a formalized process for reviewing images flagged for further review and feedback needs to be provided as to the determination of flagged images so that volunteers can learn to better identify objects.

It has been a long eight weeks of searching and it is disappointing that Steve Fossett has not been found. I hope, however, the collective knowledge gained from our collaborative Internet search effort for Steve Fossett doesn't go to waste. I really hope that one of the legacies of our efforts is that truly effective search tools are developed that that tap into the altruistic nature of people. Not every search and rescue effort can afford to spend a million dollars looking for a downed plane. A volunteer based collaborative Internet search could allow for more comprehensive search and rescue efforts when search budgets do not allow for highly advanced and expensive automated computer analysis of aerial photography.

I hope that in future I can participate in more collaborative Internet search efforts in a way that completely utilizes the skills I learned from the Steve Fossett search effort. It would be a shame to let these skills go to waste.