INFO 340 - May 24, 2004 - L17
Notes By: Prins, Fortier, Egaas

//Hendry draws a diagram about animals eating fresh flowers.
    many to many relationship...
    
    +--------+ *  Eat ->  * +--------+
    | Animal |______________| Flower |
    +--------+      |       +--------+
                    |
               +---------+
               | Yummy   |
               | Date    |
               | Calories|
               +---------+
+----------+
|    Eat   |
+----------+
| Eater    |
| Eatee    |
| Date     |
| Yummy    |
+----------+

Admin
    > Read Ch 3

Topics
    > IR System Evaluation
        - Is an IR system effective?
    > Usability
        - How easy is the system to learn, use, and remember?
    > Quality of results
        - How good are the documents that are retrieved?
NOTE: evaluate your results
        you don't know if the results are good, bad or in between
        there are methods to evaluate results like this.

Exercise #2

Wkd = Fkd * log(NDoc / Dk) + 1

Wkd = weight of the keyword in the document
Fkd = freqency of the keyword in the document
log(NDoc / Dk) = balance factor for the keyword in relation to the entire corpus.
1 = 1 = 1 = 1
as Fkd increases, Wkd insreases = more weight

There is no real reason for the +1 except to give it a minimum weight of 1.

Document collectoin: 1,000,000 documents
'Convivial' occurs three times in document 101 and 1 time in document 104
'Circus' occurs in 100 documents and three times in document 104
What is the weight Wkd for the word 'convivial circus' in document 101 and 104?

'Circus'
 log (NDoc / Dk)
 = log(1,000,000/100)
 = log(10,000)
 = 4

'Convivial'
 log (NDoc / Dk)
 = log (1,000,000/2)
 = log (500,000)
 ~ 5
 
101 (convivial 3, circus 0)
    (3 * 5 + 1) = 16
   
104 (convivial 1, circus 1)
    (1 * 5 + 1) + (3 * 4 + 1) = 19

Doc 101: 3 * log(1,000,000 / 2) + 1
        = 18.09691001300806 ~ 18

Doc 104: (1 * log(1,000,000 / 2) + 1)  +  (3 * log(1,000,000 / 100) + 1)
        = (6.698970004336019 ~ 6)      +  (13)                             = 19

//weird picture we've seen before
Last Time - Zipf's Law

PageRank of a webpage is an estimate of its quality
    Based on link structure

Why Evaluate?
    > Why should IR systems be evaluated?
        - Relevance of results
        - If the information meets your need

Evaluation of Effectiveness
    > Usability
        - Effort to learn, use, and remeber how to use a system
    > Quality
        - Coverage of document base
        - Completeness of output (recall)
        - Relevance of output (precision)
        - Novelty of output (timeliness)

Usability
    > Ease of learning
        - How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?
    > Efficiency of use
        - Once an experienced user has learned to use the system, how fast can te or she accomplish tasks?
    > Memorability
        - If a user has used the system before, can he or she remember tnough to use it effectively the next time?
    > Error frequency and severity
        - Ho often do users make errors while using the system, ho serious are these errors, and how do users recover from these errors?
    > Subjective satisfaction
        - Ho much does the user like using the system?

Quality
    > coverage
        - How extensive is the collection of documents?
    > Completeness of output (recall)
        - To what extent are all relevant documents returned
    > Relevance of output
        - To what extent are all returned documents relevant
    > Novelty of output
        - to what extent are the decouments 'fresh' (ie. timely)

Recall Performance Evaluation
    > Goal
        - Estimate the goodness of the ranked list of results that are returned by the system.
    > How? By calculating
        - Recall: Of all relevant documents how many are retuned?
        - Precision: Of all documents returned, how many are relevant?

Basic Approach
    > Components
        -test collections of documents
        - set of test queries
        - set of relavant judgements for test queries and document (Judged by experts)
    > Process
        - Run test queries against collection and compute precision and recall scores
        - No users are required!

Relevance Judgments
    > Important Point!
        - A panel of experts makes these relevance judgments by hand
        - For each query, this panel look at all documents in the collection
        - This is a lot of work!
    > Relevance can be tricky...
        - Is a known document relevant?
        - Is relevance in the "eyes of the beholder"?

Definition
    recall =  |Ra|
            -------
              |R|
    > The fraction of relevant documents that are retrieved

    precision =  |Ra|
                -------
                  |A|
    > The fraction of retrieved documents that are relevant


<<<*** THE RETURN OF GHETTO IRC CHAT ***>>>

HELL YES!!! OMG FINALLY!!!!!!

THO WILL DIE

/me slaps Tho around a bit with a large trout.
lol
haah gg aaron
the hat confused me

Questions to Hendry regarding 380.

1. Was yours taught by a prof? Ours is not.