Information Extraction (IE) is a technology which extracts pieces of information that are salient to the user’s needs. The kinds of information that systems extract vary in detail and reliability: named entities, attributes, facts and events.

Due to the complexity of the IE task, the limited performance of tools, there are few comparative evaluations in IE.

One can consider the Message Understanding Conference (MUC) as the starting point where most of IE evaluation methodology was defined.
The performance of a system is measured by scoring filled templates with the classical information retrieval (IR) evaluation metrics: precision, recall and the F-measure. Another evaluation metric, based on the classification error rate is also used for IE evaluation. The annotated data are required for training and testing.

Given a system response and a human-generated answer key, the system’s precision is defined as the number of slots it filled correctly, divided by the number of slots it attempted. Recall is defined as the number of slots it filled correctly, divided by the number of possible correct fills taken from the human-generated key. One general issue is how to define filler boundaries which is related to the question of how to assess an extracted fragment? Three criteria for matching reference occurrences and extracted ones are proposed (Freitag 1998):

 The system output matches exactly a reference
 The system output strictly contains a reference and at most k neighbouring tokens
 The system output overlaps a reference

In Automatic Content Extraction (ACE) and MUC evaluation conferences, the criteria used for assessing each system output item are: correct, partial, incorrect, spurious, missing and non committal.

 EVALITA 2009: two task related to Named Entity Recognition, and Local Entity Detection and Recognition
 BOEMIE (2006-2008)
 ACE (1999-2008)
 MUC (1987-1998)

 EvalECD’2010, Évaluation des méthodes d’Extraction de Connaissances dans les Données (FR)

 RANLP 2009, Workshop on Biomedical Information Extraction
 COLING 2008, Workshop MMIES-2: Multi-source, Multilingual Information Extraction and Summarization
 ECML PKDD 2008, Workshop on High-level Information Extraction]
 1st International Workshop on Automated Information Extraction in Media Production AIEMPro’08
 SAMT 2008, BOEMIE Workshop 2008
 Joint BOEMIE-LIVE 2007
 RANLP 2007, Workshop on Multi-source, Multilingual Information Extraction and Summarization
 BOEMIE Workshop 2006
 COLING/ACL 2006, Workshop on Information Extraction Beyond The Document
 ESWC06, Workshop on Mastering the Gap: From Information Extraction to Semantic Representation
 Automatic Content Extraction (ACE) evaluation workshop;
 Message Understanding Conference (MUC);
 PASCAL Challenge (PASCAL Network)

 BALIE
 JULIE Labs NLP Toolsuite

 Domain-independent annotated corpora:

• MUC corpora (newswire articles, also available in Spanish, Chinese and Japanese for multilingual entity task)
• ACE corpora (broadcast news, newswire, translated documents from Chinese and Arabic Treebank)
   Domain-specific annotated data:
• Job postings from WWW (Califf 1988)
• seminar announcements (Freitag 1998)

 RISE

Bibliography

• Maynard D., Peters W., and Li Y. (2006). Metrics for evaluation of ontology-based information extraction. In WWW 2006 Workshop on "Evaluation of Ontologies for the Web" (EON), Edinburgh, Scotland.
• Freitag D. (1998). Information Extraction From HTML: Application of a General Learning Approach. In the Proceedings of the 15th National Conference on Artificial Intelligence (AAAI-98).
• Califf M. E. and Mooney R. J. (1998). Relational Learning of Pattern-Match Rules for Information Extraction. In Proceedings of AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, Stanford, CA, March.