EnvironmentalChemistry.com
Environmental, Chemistry & Hazardous Materials News, Careers & Resources

Editor's Blog

Internet search for Steve Fossett eight weeks later

By Kenneth Barbalace
[Wednesday, October 31, 2007]

After eight weeks, the Internet search for Steve Fossett, which I have been participating in, via Amazon's Mechanical Turk has been called off (although other search efforts continue). Back on Sept. 18, 2007 I wrote with great enthusiasm how I thought this type of collaborative effort to review satellite photos via the Internet could revolutionize search and rescue (see: "Using the Internet to Revolutionize Search and Rescue").

After eight weeks of searching I still think that this type of distributed searching could help revolutionize search and rescue, but much of my original enthusiasm has been dampened. In order for collaborative Internet searches to be a useful option for search and rescue efforts, a lot of changes will need to be made to both the search tool and the overall process.

The biggest factor that doomed this effort to failure was the resolution of the imagery being used. The initial satellite photos being used had a resolution of one meter, which means you could identify an intact aircraft or vehicle, but a broken up aircraft would have been extremely difficult to discern from natural features. It was not until the last few weeks that we started working with imagery captured by specially fitted aircraft that we started to work with aerial photos that were of a sufficient resolution to clearly discern individual objects as small as individual sagebrush bushes. Being able to discern the smaller objects really did make a tremendous difference. Had this higher resolution of imagery been used early on there might have been a realistic chance of spotting Steve Fossett's plane had it crashed in an area being searched. What should be taken from this is that unless one is looking for an intact aircraft, one probably shouldn't rely on satellite photos of a one meter resolution. It would be much better to get aerial photography that has a resolution of at least 25 cm.

A factor that was a significant hindrance to the entire search effort was the need to use two different applications to review and report findings of individual photos. Via a webpage interface Amazon's Mechanical Turk would assign photos to be reviewed, unfortunately the area covered by the photos was small and it was not easy to compare new photos against old photos within the web browser. Early on in the search effort, master Google Earth KML overlay files were made available to facilitate the comparing of old and new imagery, but near the end of the search effort master Google Earth KML overlay files stopped being provided for areas being actively reviewed. Initially this meant that there was no way to compare old and new photos. Then a Google Maps applet was added to HIT pages, which was supposed to help provide the ability to compare old and new imagery, but the scale between the old images and the new images were way too different and the applet containing old imagery was completely useless for comparison purposes.

After some discussions with others who were participating in the search on a forum being used to discuss this effort, I was able to reverse engineer the logic behind the naming scheme of the aerial imagery being used and I figured out how to capture the coordinates needed for lining up the images within Google Earth. With a little trial and error and a lot of research, I was able create a script that generated Google Earth KML overlay files on the fly that would present the aerial image being reviewed in MTurk along with the adjoining photos in Google Earth. This gave individuals a means to effectively review old and new photos. The problem was that this required the user to take several steps to capture the required information from the MTurk HIT they were working on and provide it to my KML overlay generator. What was really needed was for the MTurk HITs to link directly to a KML overlay generator such that users would not need to do several steps just to be able to compare old and new images.

While the real reason is not known, there has been a lot of speculation as to why those running the MTurk search effort stopped providing Google Earth overlays to assist in the comparison of old and new photos. One possibility is that this was done to better protect the integrity of any crash site as one could easily get the exact coordinates of any object located within Google Earth. There did seem to be the potential for a few treasure hunters and/or glory seekers participating in the search effort. There might have been a real concern that one of these parties would take it upon themselves to physically investigate any reported sightings and disrupt the integrity of the crash site (either intentionally or unintentionally). Simply put, the comparison of old and new photos was absolutely critical, however, the way the MTurk search effort as it was set up, this comparison had to be done within Google Earth, which made the coordinates of any object found extremely easy to determine. It can not be over emphasized how important it is to be able to compare old and new photos. It is simply the best way to rule out potential objects of interest. In fact, if people had effectively compared new photos against old photos, probably upwards of 90% of objects that were flagged for further review could have been ruled out and not reported. Yet at the same time it is important to protect the crash site.

Beyond the technical challenges posed by the search tools being used, another very serious hindrance to the search effort was the complete inabilities of many participants to be able identify the difference between natural objects like dead trees, rocks, shadows, etc. and foreign objects that did not belong like airplane wreckage. While it is difficult to know what any crash site looks like before it is discovered, it is critical for those analyzing photos to be able to rule out natural objects. Yet throughout much of the search effort there were precious few photos provided that truly helped people understand what it was they were looking at. It was not until the lpast couple of weeks when there started to be good ground photos provided of various objects like exposed rock outcroppings, dead trees, sagebrush, etc. Some of the volunteers who had been participating in the MTurk search effort went out to the area of Nevada being searched and took pictures of various objects like exposed rock outcroppings, dead trees, sagebrush, etc. and posted them along with the coordinates for each photo on the forum being used to discuss the search. These photos gave the rest of us a much better idea of what we were looking at, which was most helpful.

In my own case over a period of around eight weeks of daily searching, I reviewed over 25,500 Mechanical Turk HITs, of these HITs, I ended up reporting only two as being worth further investigation. The first HIT I reported I later decided was probably just an odd shadow cast by a geologic formation. The second HIT I reported I suspected was most likely a large farm implement, but I could not definitively rule it out so I flagged the object out of caution.

Early on, a major problem was the failure of people to take time to read and understand what instructions were provided. Admittedly not enough instructions or tutorials were provided, but the ones that were provided were important. For instance, one of the most important instructions was about how to report findings online so that they could be reviewed and evaluated by a team dedicated to this purpose. Rather than reporting findings through proper channels, some people had a propensity of thinking that their findings were so important that they took their findings directly to CAP or Nevada law enforcement officials. This improper reporting of findings created an excessive level of distractions for search authorities, which again did not endear the Mechanical Turk and Google Earth search efforts to search officials. Later on this didn't appear to be as big of an issue as there was only a small corps of really dedicated searchers who were still reviewing imagery by the end.

Could collaborative photo analysis via the Internet be an effective aid in search and rescue efforts? Yes I believe it could, but the lessons learned with the Steve Fossett search need to be used to build better search tools and possesses:

  1. The search tool being used to review photos needs to be a single integrated application that allows reviewers to very easily compare old and new photos and then report their findings in one place. Having to jump between two different applications is too cumbersome.
  2. The tool must allow for the comparison of old and new photos while protecting the integrity of the crash site once it is located as such coordinates should only be accessible by search officials (currently it is not possible to really hide coordinates).
  3. There must be robust tutorials that include actual photos from the search area with on ground examples of the flora and geologic formations of the area as well as the aerial views of these objects so that people can better understand what they are looking at.
  4. Higher resolution photos must be used than was provided initially for the Steve Fossett search effort. One meter resolution is not of a high enough quality to be useful for this task.
  5. There must be effective and regular communication from the top down such that volunteer searchers do not get frustrated and quit.
  6. There must be a formalized process for reviewing images flagged for further review and feedback needs to be provided as to the determination of flagged images so that volunteers can learn to better identify objects.

It has been a long eight weeks of searching and it is disappointing that Steve Fossett has not been found. I hope, however, the collective knowledge gained from our collaborative Internet search effort for Steve Fossett doesn't go to waste. I really hope that one of the legacies of our efforts is that truly effective search tools are developed that that tap into the altruistic nature of people. Not every search and rescue effort can afford to spend a million dollars looking for a downed plane. A volunteer based collaborative Internet search could allow for more comprehensive search and rescue efforts when search budgets do not allow for highly advanced and expensive automated computer analysis of aerial photography.

I hope that in future I can participate in more collaborative Internet search efforts in a way that completely utilizes the skills I learned from the Steve Fossett search effort. It would be a shame to let these skills go to waste.

14 comments:

NOTICE: Comments are user generated feedback and do not represent the views and/or opinions of EnvironmentalChemistry.com.

Anonymous said...

Ken,
Thank you for this thoughtful review of the Fossett search effort via Mturk. After many hours spent on this effort, I have been disappointed by the poor communication with volunteers. I am sure we will find, as this technology moves forward, that good two-way communication with volunteers is a key to success. While there may have been technological breakthroughs in the Fossett search, if a well-designed survey of online participants is not undertaken, an entire body of learning will be missed.

Best regards,
LRM

The Browns said...

I am a retired geographer. When I worked was project manager on a very large geographic information system. Part of that system as an image analysis capability. You gave the most objective and detailed review of the MT process I have seen and included all of my complaints and recommendations. Thanks for doing such a terrific job on your review. I think the tool has great potential for SAR but as you point out further integration needs to be done. Change detection can only be done by overlaying older imagery with newer imagery. The process can be automated to flag potential sites for further investigation. Perhaps the search for Mr. Fossett will at least generate a useful process for SAR and that would be a great legacy for him to leave behind.

RTB

taagold said...

During my 2500 hits I submitted 80 possible sites in 8 weeks. 51 years as a crop duster,ferry pilot and VFR sectional surveyor,I know the lay of the land. The terrain is confused by the topography whether it be mountain flora or desert illusions. Given the coordinates a searcher can pin the location. In the air this proves to be extremely difficult. A refreshed Google map of Nevada would produce results if the viewers had a wide long grid line to examine. Not knowing the time line denied thousands of eyes from knowing if their search site,was past or present, before or after Fossett's departure. A submission with coordinates would take a map reader seconds to confirm. Please continue your dilligent effort, for future airmen will most certainly need your help. TAA

Anonymous said...

Your article was well thought out, and you made some very good points.
My biggest problem with the whole Turk search was lack of any response to what I thought was a very positive hit I found on Sept. 12.

500 ft. north of West Lake in Mono County Ca. there is an image of what appears to be an intact plane. This lake is about 50 mi. due south of Smith Nevada.
Coordinates off of Google Earth are:38.0948006957 119.317990728
I've sent this to multiple web sites, news sources and gov. agencies with no real response.
I'm fed up.

Ken (EnvironmentalChemistry.com) said...

Anonymous on 11/01/2007 at 11:03:00 AM,

I reviewed the area you reference using the GeoEye_IKONOS-2_1m overlay and I found nothing that I thought would be of interest to searchers. Airplane shapes seem to be very frequently replicated in nature. All it takes is for a couple boulders or trees to be positioned just right and shadows to be cast just so for there to be an illusion of an intact plane.

This problem of airplane illusions gets compounded with the one meter resolution photos that were being reviewed like in your sighting. It becomes very hard to discern natural formations from things that don't belong if one can not clearly see what the objects are.

Your frustration reinforces three of the points I made in my article:

1) one meter resolution photos is not of high enough quality for most people to fully analyze. Aerial photos of around the 25cm range is much more effective.

2) volunteers really need a basic level of instruction and tutorials. Without some guidance, one can start to see planes everywhere. Or as I like to say "the eyes see what the mind wants to find."

3) There needs to be a real feedback about reported sightings such that everyone can better learn to identify objects. If someone had told you weeks ago what the object was, it would have eliminated weeks of frustration over your sighting.

This is why there needs to be a formalized group of annalists systematically reviewing and reporting on submitted sightings. Volunteers may have a high false positive rate, but with the right imagery, they will be able to weed out the vast majority of uninteresting photos allowing experts to focus their attention on only the most promising photos.

Anonymous said...

I could send you a copy of what I found.
jpeg. or word doc. file

Ken (EnvironmentalChemistry.com) said...

Anonymous,

You could either post your JPEG to a image hosting service like photobucket and provide me with a link to the image in the comments here, or contact me via my contact form (link at the bottom of this page) and I would reply with an email address where you could email your photo. I don't post my email address on the Internet, as this is only an invitation to spammers to harvest my email address and spam me like crazy.

Anonymous said...

what aamazes me is the family of steve fossett refuses to offer a reward for the return of his body and Hiltons aircraft? it shows me that just dont care.

Ken (EnvironmentalChemistry.com) said...

Fossettfinder,

I totally disagree about offering a reward. A reward would not improve the odds of finding Steve Fossett, but it would bring all of the treasure hunters out of the woodwork. The last thing that is needed is countless treasure hunters combing through the back country of Nevada and then spoiling the crash site.

Besides, look at how many people have volunteered to help in the search without any monetary incentives.

Ken (EnvironmentalChemistry.com) said...

For those who would like to continue the Internet search efforts for Steve Fossett and help refine the use of the Internet for reviewing aerial imagery as part of a search effort, I was able to retool my KML overlay generator for Sanborn 2 imagery such that it can systematically assign small groups of Sanborn 2 images to be reviewed. You can download Google Earth KML overlay assignments to review at http://environmentalchemistry.com/fossettsearch /sanborn2.kml?dataset=assign (please note this requires that you have Google Earth installed on your computer).

Once the assigned imagery has loaded in Google Earth, you can click on the "thumb tack" in the center of the image set to get reporting instructions and guidance.

Anonymous said...

The reason the KML file didnt work is because it didnt cover the area Steve fossett went down in,

they just got totally convinced he was in that area of the kml file and closed there minds to oher possibilities, he is further north probably up around pyramid lake

this is the main reason he wasnt found

allenon said...

Hello...I read your artical on the steve fossett search.
I have participated with the "violent skies" manual search.
on the night of nov.2 I found a truely odd and bright object at
38 36'28.76" -118 52'40.89". on geoeye delivery4.
I reported this and got 6 replies, 3 of which agreed
it was suspicious.and not on old imagery. the sanborn search does not include this area.
anyway..with the search over.
I'm concerned that this might
not be checked by someone with ability to pass it on.
so I'd like you to take a look. visable at 1500 ft and below.
one reviwer commented that it was not on the DG image overlay. so I've re checked the dg and did not
specifically find it,but I believe that the sun angle may be the reason.
If this object is plexiglass then it would provide
a strong reflection when the sun's rays hit it just right. and be "invisable" other wise.
another reviewer commented "close to a road" not likely an aircraft.
I turned on "terrain" and found that this was on a mountain slope and the road was below the level this was at and would not be seen.
there are no other "pieces" nearby
that I could see. as you said
the resolution is not good enough.
thanks:owen allen
my email= pink@cconnect.net

Ken (EnvironmentalChemistry.com) said...

allenon,

Just to the north of your coordinates there is a plane shaped object, which is caused by a pair of trees, which are almost the same color as ground vegetation and are in old imagery. At the coordinates you provided there is a white square, which is too small to show up as an individual pixel in old imagery, so it is hard to say whether it is in old imagery or not.

I do not believe it is a crash site or of interest because there is no other debris like objects anywhere near it. What it probably is is a square concrete slab that I have seen in other imagery and is commonly used to cover abandoned mine shafts in this area.

Ken (EnvironmentalChemistry.com) said...

The Mechanical Turk effort to find Steve Fossett may have been suspended, but the MTurk Sanborn 2 imagery is still available online. I have developed a KML overlay generator is designed to provide access to those images via Google Earth. There are three ways to access those images by automatic assignment, coordinates, or image name.

You can learn more about accessing the Sanborn 2 imagery via my KML overlay generator and give it a try at http://EnvironmentalChemistry.com/fossettsearch/

EnviroChem Logo