How to perform a Combined Correctness evaluation

General considerations

These instructions describe the process in general terms. They are not tied to a particular piece of software and so the researcher will need to figure out how to work with the software in question to perform the steps of the evaluation. It should be possible to use any analysis software for this purpose, with varying degrees of manual work.

It should be noted that the exactness of the measurement of Combined Correctness will naturally be greater the more spots and matching that are evaluated. It is suggested to evaluate at least 100 spots and 100 matchings, but you may choose to evaluate more or less than that number depending on your needs.

To get random numbers in a certain range you may use any tool of your choice such as spreadsheet software, or you can use this site:

http://www.random.org/integers/

It should further be noted that the aim of this initiative is to develop a freeware software that will allow researchers to easily do this quality control on their own 2D gel image analysis data.

Spot evaluation

  1. Set your software to not display spot borders.
  2. Divide the gels into a grid. Choose the number of rows and columns so that each rectangle formed by the grid contains approximately 10 spots on average.
  3. Randomly select a gel.
  4. Randomly select a grid rectangle in the gel (select the column and row at random).
  5. Count the total number of true spots spots in the grid rectangle, and write down the result. 
  6. Set your software to disaply spot borders.
  7. Calculate the number of correct spots in the grid rectangle and write down the result. Ignore empty spot borders. 
  8. Calculate the total number of true spots evaluated so far. If that number is less than the minimum required number, go to step 3 again.
  9. Calculate spot correctness using the total of the number from all grid rectangles as (correct spots)/(total number of spots evaluated).

spot correctness = correct spots / spots evaluated

 Pair matching evaluation

  1. Set your software to display spot borders.
  2. Select one detected spot at random out of all the spots in the project (spot refers to a single detected spot in a single gel).
  3. Select a gel at random out of all the gels except the one containing the spot selected in step 1.
  4. Zoom in on the area in the second gel that is the equivalent of the area in the first gel that contains the randomly selected spot. (This area may or may not contain a spot that has been matched to the spot in the first gel.)
  5. Observe the matching of the first spot to the second gel and place into either correct pair matching or incorrect pair matching, or skip it (see below).
  6. Start over at step 2 until the required number of pair matchings have been evaluated (excluding the skipped ones).
  7. Calculate pair matching correctness as: (the total number of correct pair matchings) / (the total number of pair matchings evaluated).

pair matching correctness = correct matchings / matchings evaluated

“Skip” if the spot is incorrectly detected

One goal of this evaluation is to discriminate between matching performance and spot detection performance as much as possible. To achieve this goal, only cases where the detection of the spots in both gels is correct should be considered, where the program had at least a chance to match correctly. If the spot detection in one or both gels is incorrect, skip the evaluation of this matching and move on to the next. In this way, non-decidable or borderline cases will be avoided.

Combined Correctness

The Combined Correctness is then finally calculated as:


Combined Correctness = spot correctness * pair matching correctness

 

This yields a number between 0 and 1, where the higher the number, the more likely that the measurements from the image analysis are correct.