INFO 370 - Lecture 15 - November 22, 2004 Notes By: Fortier, Podhola, Yaptinchay, Egaas Validity > construct validity - a measure that doesn't really assess the issue in question (something went wrong in the testing procedure) > content validity - does the nature of the test pertain to a large problem set? - ex: online searching behavior, timing them in seconds, but the set of tasks are what they normally do. Tests are not practical. > predictive validity - - > unable to control = threats to validity // start slides Level of Control > Experiment = a study in which at least one variable can be controlled and manipulated This s the opposite of > Naturalistic study = A study in which no variable is controlled or manipulated An Example: Relevance Feedback > Research Question: Does relevance feedback increase the quality of searching? > Hypothesis: The use of relevance feedback increases the quality of searching - Null Hypothesis: The use of relevance feedback does not increase the quality of searching. Conceptualization > Relevance feedback = marking the relevant items in a retrieved set so that the system can modify search results > Quality of searching: - The ability to retrieve only relevant items - The time it takes to get satisfactory results - The level of satisfaction the user expresses Pre-Test / Post-test > Task 1 (no relevance feedback): - "Find newspaper articles that report on workplace safety in terms of accidents but not with workplace violence." > Task 2 (relevance feedback for experimental group) - "Find newspaper articles that report on airplane safety in terms of mechanical issues but not terrorist threats." Operationalization Quality of Searching > Precision = the ability to retrieve only relevant items > The time it takes to get satisfactory results - Operationalization: The # of minutes lapsed from receiving the question and the decision to end the search. Comparison Groups > Experimental group = Receives the experimental manipulation > Comparison group = Receives a different manipulation (has a different value on the independent variable) > Control group = Receives no manipulation Randomizing Expiriments . Forces Acting Against Validity > Internal Validity: - Selection bias * Example: > People choose to be in a group because their friends are in it. > People in the same class or same work unit are in the same group. Selection Bias: Characteristics of the participants in the experimental and comparison group differ. - Endogenous change * Example: > During a protest with one group, participants learned how to use the system effectively. > The group using relevance feedback becomes more motivated. Endogenous Change: Participants develop or change during the experiment, independent of the experimental manipulation. - Contamination * Example: > The experimental and the comparison group are in the same room. Those without relevance feedback feel they got the short end of the stick. Contamination: One of the groups is aware of the other group and is influence in the posttest as a result. Solution: Separate the groups > External Validity - Treatment Misidentification * Example: 1. The researcher conveys to the participants his or her enthusiasm about relevance feedback (self-fulfilling prophecy) 2. Participants without relevance feedback may try to search better than usual just because they were selected to participate in the study (placebo effect) Solutions: Scripted instructions / Blind assignment Sample Generalizability > To what degree can we generalize form and experiment to the whole population > Can we generalize across populations? > General rule: The more controlled the study is, the less it is possible to generalize from it. Solutions: - Repeating an experiment with different samples - Accumulating results from different experiments Interaction of Testing and Manipulating Example: - Participants realize that the researcher is measuring time, precision, and satisfaction. Solutions: - Hide instrumentation - Do not react to subject's actions - Limited misdirection -- ASCII art section -- @@@@@@@@BBBBB$BBBBBB0000BBB000BBBBBBBBBBBBBBBBBBBBBBBB$BBBBBBBBB$$$$$$$$$$ @$8g8BB8808808888B88BB00000B00BBBBBBBBBBBBBBBBBB0BBBB00BBB$$BBB$$$$$$$$$$B @BGBGgBG888G0G8g8B88BBBBB00000g0BB00BBBBBBBBBBB0BB000BBBBBBBBBB$$$$$$$BBB$ @BGBGg0GgG8G0G88GBggBBBBB0g%VC3C/<<<