Environmental, Chemistry & Hazardous Materials News, Careers & Resources

Editor's Blog

Using the Internet to Revolutionize Search and Rescue

By Kenneth Barbalace
[Tuesday, September 18, 2007]

For over a week now I have been participating in the Internet search for lost aviator Steve Fossett using Amazon.com's Mechanical Turk and Google Earth. The premise is simple. Via Mechanical Turk, anyone anywhere in the world who has an Internet connection can help in the search for Steve Fossett by joining Mechanical Turk and reviewing little squares of updated satellite photos which represent a swath of land which is around 7,225 sq/m (1.78 acres) in size.

If there is something of interest on the photo the searcher can then double check the Mechanical Turk photo against updated and old satellite photos in Google Earth using a special KML file. The KML file provides Google Earth with updated satellite photos of the search area which are overlaid on top of the normal older satellite photos. The searcher can then zoom in to the area of interest and turn on and off the overlay. If an object exists in the new photos, but does not exist in the old photos, it might be of interest. If, however, the object of interest exists in both the old and new photos, then it can not be Steve Fossett's plane. If a searcher does come across a "HIT" in Mechanical Turk that the searcher feels merits further review, they can flag the photo and add comments via a web form directly below the satellite photo on the Mechanical Turk webpage containing the photo of interest. These flagged HITs are then reviewed and filtered by trained individuals, with the most promising leads being forwarded on to the physical search and rescue efforts. To help reduce the odds of Steve Fossett's plane being overlooked, Mechanical Turk feeds each photo or "HIT" to several different searchers so that each photo is looked at several times.

Being able to allow thousands of volunteers to comb through thousands of square miles of satellite photos could in the long run revolutionize the way search and rescue is conducted. No longer would searches be restricted to daylight hours, subject to bad weather, limited by available aircraft, limited by the logistics of housing and feeding hundreds of on the ground searchers, etc. Anyone in the world could participate from the comfort of their home day and night. They would simply need to be provided updated satellite photos and a structured method of sifting through all of the photos.

The Steve Fossett search, however, is showing that there is a long way to go before "crowdsourcing" searches via the Internet will become a truly effective tool that is of a true value without being an unwelcome distraction to the physical search and rescue operation itself. In a way, the Steve Fossett search is a proving ground that will hopefully provide valuable insights as to how to build truly effective tools to conduct these searches and how to manage the flow of tips from untrained volunteers through teams of trained searchers who can sift out the most promising leads and pass them on to the physical search and rescue operation.

My searching experience

In my first week of participating in the Steve Fossett search (Monday 9/10/2007 – Friday 9/14/2007) I spent around 50 to 56 hours searching and completed over 15,000 Mechanical Turk "HITs". Of those HITs, I found no photos of interest that could not be ruled out by comparing the new photos against old photos using Google Earth. I frequently came across light or dark airplane like crosses in photos, but by simply taking the time to sleuth them out and compare the old and new photos against each other I quickly discovered the shape to be a combination of geologic formations, vegetation and/or shadows. Now 15,000 photos may seem like a lot of photos to look at, but think of it this way, that works out to only around 108 sq/km (~42 sq/mi) in a total search area that is around 42,000 sq/km (17,000 sq/mi) in size. I did come across one photo that had a twin engine commercial jet (e.g. maybe a Boeing 737) flying through when the picture was taken. From what I have read all confirmed sightings of planes via Mechanical Turk have turned out to be planes flying though when the satellite photo was taken.

False reports and reports bypassing proper channels frustrating for CAP

One of the great frustrations I have seen expressed by CAP (Civil Air Patrol) officials in regards to the Google Earth search for Fossett has been the steady stream of false leads being fed to them directly from the public outside of the proper Mechanical Turk channels. People are so eager to find Steve Fossett that their eyes begin to see what their mind desires to find. On top of this many searchers appear to be using Google Earth directly without using the Mechanical Turk part of the equation and often times they don't even have the updated satellite photos. Then when they do find things that they think are Steve Fossett's plane, they report it directly to the CAP and/or their local newspaper rather than allowing the lead to flow through proper channels that have been set up to filter all of these leads.

I have looked at many of the coordinate locations that were reported online in either news articles or blog/forum discussions and invariably, I could quickly rule out the sighting by simply looking at old satellite photos. In many cases, the reported sighting wasn't even in an area that was covered by up to date satellite photos via the special KML file overlays, thus whatever it was couldn't be Steve Fossett because the photo had been taken before he went missing.

Don't get me wrong, if people participating in this search think they see something that needs further attention, they should flag it and leave some comments with the photo in question via Mechanical Turk. I just don't think people should try to go outside of proper channels to report their possible findings.

What can be learned from the Internet search for Steve Fossett

To be successful, search and rescue can not be a freelance operation. It must be structured and organized. In order for the Internet to become a successful tool for harnessing the altruistic nature of people and their desire to help in a search, it must be structured in a way that allows Internet generated leads to be filtered properly. Volunteer searchers can't just go surfing satellite photos looking for "interesting" objects in an unstructured way even if the provided photos through the structured search are boring or tedious (trust me I've seen my share of flat featureless photos in the past week). Even the boring photos need to be reviewed and ruled out in a systematic way.

I stated that the Steve Fossett search is a proving ground to help develop the potential of using the Internet to revolutionize search and rescue. Having looking at almost 16,000 Mechanical Turk HITs, I have compiled a list of things that need to be refined to really make this an effective method to conduct searches:

  1. First and foremost, the resolution of the satellite photos is way too low to be really effective. Currently the best resolution provided by commercial satellites is about one square meter. This means that on one of these photos the overstuffed arm chairs in my home would only appear to be a single blurry pixel. This is not enough detail to really make out what objects are. I read the other day that a new generation of commercial satellites are getting ready to be launched, which could improve the resolution of photos by an order of magnitude. This would mean my laptop computer might now show up as a single pixel and my chair could be a collection of pixels. This would be tremendously valuable in reducing the number of false hits as people could get a better idea of what they are looking for and if a plane was torn up into a debris field with lots of small pieces, it would be easier to spot.
  2. The interface for reviewing photos desperately needs to be improved. Although great credit should be given to Amazon's Mechanical Turk, a web page based application may be the wrong tool for the job. Mechanical Turk is too clunky, requires too much scrolling of HIT pages and requires too many mouse movements/clicks. If the entire search area for Steve Fossett were to be searched using Mechanical Earth, there would be over six million HITs just to look at everything once. To look at the entire area say four time (each tile by four different people) would require 24 million HITs. This means people need to be able to review and sift through the HITs as quickly and efficiently as possible, which requires there to be as little scrolling, mouse movement and clicks as possible. It is also too cumbersome to switch back and forth between Mechanical Turk and Google Earth. I think the best solution would be for Google to bring the entire process into the Google Earth application itself. Google could create a sub-function of Google Earth or a derivative application called say (Google Earth SAR) that could feed people the photos to be looked at and allow the individual to easily compare and contrast new and old photos. Ideally individual photos could be reviewed and rejected with as few as one key stroke (say the "N" key). If the individual wanted to submit a possible finding they could press the "Y" key and then be prompted to provide further details. A two step process for submitting positive findings could help prevent accidental submission of possible leads. In my case, this would have help me advert some accidental lead reports via Mechanical Turk, because my mouse cursor had moved when I was looking at photos and not my mouse. As a result, I clicked on yes when I meant no.
  3. Updated satellite photos should only be available to those who are actually participating in the search using the proper tools and reporting channels (this is another reason to bring everything together under one Google Earth application). It must also be made clear that NO satellite photo crash sightings would be accepted outside of the proper channels. This would significantly reduce the amount of noise and distraction experienced by the physical search and rescue teams because of improper Internet sighting reports.
  4. It is important to know more about the aircraft being searched for. I did not learn until late last week that the airplane Fossett was flying was a fabric covered aircraft instead of having an aluminum skin. This could have been an important detail in helping one spot a mangled wreck.

Unexpected personal side effects

While participating in the Mechanical Turk search for Steve Fossett, I have been closely following news stories about the search (I don't want to keep searching if he has been found). Last week I started to get frustrated by news stories featuring possible Fossett sightings that, when the photo was looked at closely, clearly wasn't an airplane. I commented on one such report on the Wired.com blog "Danger Room" using my real name (Ken Barbalace) as I typically do when posting comments (see: Geeks spot Fossett). My name is fairly unique and this combined with my extensive web presence makes me easy to locate via Google. As a result of my comments, later that day I was called by a reporter from the Chicago Sun Times and then an hour later by the San Francisco Chronicle, which each then published stories featuring me. The Sun Times article by Frank Main was "Steve's eye in the sky" and the San Francisco Chronicle article by Henry K. Lee was "Thousands of Mechanical Turk users searching for Fossett's plane". My local paper the Portland Press Herald saw these articles and called me asking for an interview. They sent a reporter and photographer to my home to interview me for an article they published on the front page (below the fold) on Saturday 9/15/2007. The article by David Hench was titled "Searching from the Sofa: Portland techie joins hunt for missing adventurer". This article resulted in articles being published in the Boston Globe ("Search for Fossett spread to the Internet" and the Montpelier VT paper The Times Argus ("Search for Fossett spreads to the Internet"). The Boston Globe version of the article was then picked up on Sunday by the Lewiston Sun Journal, another local paper based in Lewiston Maine ("Via Internet, Mainer helps in Fossett search".

Talk about wild side effects, who would have thought that some simple comments in a blog post about Steve Fossett would lead to all of this attention? I certainly didn't. I was just leaving comments to blog posts like I always do while participating in something that I found really intriguing and demonstrated the great potential of the Internet.

Continuing the search

I did read last night that the authorities are scaling back the primary search and rescue efforts so it may be that the last best hope of finding Steve Fossett's plane is in the hands of those participating in the Mechanical Turk search. If you have time a keen eye and would like to participate in the Internet search for Fossett, you can go to http://www.mturk.com to sign up. Well I guess it is time to stop writing and go back to my staring at patches of sage brush, glaciers, and airplane shaped illusions/shadows.


NOTICE: Comments are user generated feedback and do not represent the views and/or opinions of EnvironmentalChemistry.com.

Anonymous said...

Dear Ken,

Thank you for the post. I hope some gets this to the right people at Google and they change the way its being done to Google Earth. I am not a Techie, I have been looking at images as well on Turk and I am learning to use Google Earth and doing exactly what they say to do. Just think! if 12 million people in the Untied States would take oh say 1/2 hr even and did 2 images (current images) we could cover the part that need to cover better yet not knowing what is flight plan was add a few more images to each person and cover up north too. I live a short distance away from Minden and boy did I want to go there and look for him. I figured I would be in the way of pro at this so I sit at my computer and look until I cant look anymore. I just want to see him alive!! I been over in the mountain there is still homes out in there too what know one heard anything or seen anything boy I find that so hard to beleive unless it didnt make noise! Off to start searching again we will find him!!! Thanks again for your Post like I said hopefully it gets to google and they work on that it can be done pretty fast.
Sheri W

Anonymous said...

Great article. The side effects I've experienced from searching online for Steve Fossett are almost the opposite of yours; after awhile, a Zenlike sense of peace.

I blogged about it here...

Ken (EnvironmentalChemistry.com) said...

Thank you Sheri W, and Flora68 for your comments.

Sheri, I'm not sure that 18 million people each taking 1/2 hour to search would be productive. I suspect it would be better for 10,000 people to take 8 hours to search. The reason is that it takes time to get used to looking at satellite maps and literally every few photos there are objects that can trick the mind into thinking one is looking at a downed airplane. If you gave me thirty random Mechanical Turk photos, at least ten would have objects that could easily be misinterpreted as a downed aircraft by someone who does not really know what they are looking for.

I am not at all surprised that no one heard or saw Steve Fossett go down, heck given how remote parts of the search area is a big jet could go down without anyone witnessing it. It is a very vast area and many parts of it are extremely rugged. This is in part why there are something like 200 unsolved missing plane cases in that area.

Flora68, yes it really does require a Zen like state to stare at the screen for hours working through the photos. Sometimes it gets to be almost tedious. The side effects I reported are kind of weird in a fun way. I never expected anything like that would happen, but it was fun talking to the reporters. I particularly enjoyed having the Portland Press Herald reporter visit my home. We talked about all kinds of stuff beyond just the search for Steve Fossett. It can be really fun talking shop with someone in a similar industry (him being a traditional newspaper man and me making a full time career out of publishing EnvironmentalChemistry.com).

Ken (EnvironmentalChemistry.com) said...

Following up on my suggestions above another suggestion would be an online "training manual" that contained satellite photos of some known (especially recent) crash sites for the size of plane being looked for. This would help searchers get an idea of what they might expect to see. Right now we are kind of searching blind with the direction being to report anything that looks like it doesn't belong. I'd love to have an idea of what a crash site might look like on a satellite photo.

okatepe said...

Hey this onur from istanbul,

First downed aircraft careers name "Ben-My-Chree" year 27 december 1916
mechanical officer turk his name is "Mustafa Ertugrul"

he was downed the first aircraft career in the 36 minutes.

Mustafa Ertugrul and his soldiers are when they are saw the first time aircraft ship, they are felt like the when american indians are introduct with the rifles. :)
however british armies are stopped the ship for the food.The are carryin witf sandals.then turkish commender mustafa ertugrul has putted the bomb in the last sandals.when sanadal was came to aircraft ship it was explosed.and downed in the 36 seconds.

in the history first turkish armies downed the aircrafr career.(AGAIN)

again because pls remember turkish army was created the war rules, and professionell war types :)

but no war make sex :) bye

ahh sorry downed place name is in the Mediterranean sea ...city name is turkey / antalya / kas

Anonymous said...

Some nice observations, and mostly dead on.

However, i think the "cumbersome" Mturk interface has some advantage.

When people repeat an action over and over, they are likely to make a mistake, as the process speeds up. Like clicking yes instead of no, or something like that.

The bit of scrolling involved on Mturk, slows down the process and gives users a little extra time with the frame. The rythm simply changes.

It might not have been intentionally, but designfeatures like this are used for this reason.

Ken (EnvironmentalChemistry.com) said...

There are much better ways to put "speed bumps" into the process. Excessive scrolling and mouse movements will exasperate the exact problems you think they will help reduce. Instead of spending more time looking at images, people become distracted by the process. A better way to do "speed bumps" would be to simply disable the submission of the form for two or three seconds (this is easy enough to do with HTML and JavaScript). A streamlined form with a timed submission delay would allow people to keep focused on looking at tiles WITHOUT being distracted by the process required to submit and retrieve HITs.

I will beg to disagree that the reason for the form design is to slow people down, instead I am fully convinced as a website developer for twelve years that what we see with MTurk and the Fossett search process is a total lack of any usability forethought being applied to the user interface design.

EnviroChem Logo