Abstract
The system for semantic evaluation VENSES (Venice Semantic Evaluation System) is organized as a pipeline of two subsystems: the first is a reduced version of GETARUN, our system for Text Understanding. The output of the system is a flat list of augmented head-dependent structures with Grammatical Relations and Semantic Roles labels. The evaluation system is made up of two main modules: the first is a sequence of linguistic rules; the second is a quantitatively based measurement of input structures and predicates. VENSES measures semantic similarity which may range from identical linguistic items, to synonymous, lexically similar, or just morphologically derivable. Both modules go through General Consistency checks which are targeted to high level semantic attributes like presence of modality, negation, and opacity operators, temporal and spatial location checks. Results in cws, recall and precision are homogeneous for both training and test corpus and fare higher than 60%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bos, J., Clark, S., Steedman, M., Curran, J., Hockenmaier, J.: Wide-coverage semantic representations from a ccg parser. In: Proc. of the 20th International Conference on Computational Linguistics, Geneva, Switzerland (2004)
Dagan, I., Glickman, O., Magnini, B.: The pascal recognising textual entailment challenge. In: Proceedings of the Recognising Textual Entailment Challenge, Southampton, UK, pp. 1–8 (2005)
Delmonte, R.: Evaluating GETARUNS Parser with GREVAL Test Suite. In: Proc. ROMAND - 20th International Conference on Computational Linguistics - COLING, University of Geneva, pp. 32–41 (2004)
Delmonte, R.: Text Understanding with GETARUNS for Q/A and Summarization. In: Proc. ACL 2004 - 2nd Workshop on Text Meaning & Interpretation, Barcelona, Columbia University, pp. 97–104 (2004)
Delmonte, R.: GETARUN PARSER - A parser equipped with Quantifier Raising and Anaphoric Binding based on LFG. In: Proc. LFG 2002 Conference, Athens, pp. 130–153 (2002), http://cslipublications.stanford.edu/hand/miscpubsonline.html
Delmonte, R.: Semantic Parsing with an LFG-based Lexicon and Conceptual Representations. Computers & the Humanities (5-6), 461–488 (1990)
Delmonte, R., Bianchi, D.: Binding Pronominals with an LFG Parser. In: Proceeding of the Second International Workshop on Parsing Technologies, Cancun, Mexico, pp. 59–72. ACL (1991)
Dolan, W.B., Quirk, C., Brockett, C.: Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In: Proceedings of the 20th International Conference on Computational Linguistics, Geneva, Switzerland (2004)
Harabagiu, S., Pasca, M., Maiorano, S.: Open-Domain Question Answering Techniques. Natural Language Engineering 1(1), 1–38 (2003)
Lin, D., Pantel, P.: DIRTdiscovery of inference rules from text. Knowledge Discovery and Data Mining, 323–328 (2001)
Punyakanok, V., Roth, D., Yih, W.: Natural language inference via dependency tree mapping: An application to question answering. Computational Linguistics (2004)
Raina, R., Haghighi, A., Cox, C., Finkel, J., Michels, J., Toutanova, K., MacCartney, B., Marneffe, M.C., Manning, C.D., Ng, A.Y.: Robust Textual Inference using Diverse Knowledge Sources. In: Proc. of the 1st. PASCAL Recognision Textual Entailment Challenge Workshop, Southampton, U.K., pp. 57–60 (2005)
Zaenen, A., Karttunen, L., Crouch, R.: Local Textual Inference: Can it be Defined or Circumscribed? In: Proc. Workshop Empirical Modeling of Semantic Equivalence and Entailment, ACL 2005, University of Michigan, Ann Arbor Michigan, pp. 31–36 (2005)
Glickman, O., Dagan, I., Koppel, M.: A Probabilistic Classification Approach for Lexical Textual Entailment. In: Twentieth National Conference on Artificial Intelligence, AAAI 2005 (2005)
Corley, C., Mihalcea, R.: Measuring the Semantic Similarity of Texts. In: Proceedings of the ACL 2005 Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, MI, pp. 13–18 (2005)
Braz, R., Girju, R., Punyakanok, V., Roth, D., Sammons, M.: An Inference Model for Semantic Entailment in Natural Language. In: Proc. of the 1st. PASCAL Recognision Textual Entailment Challenge Workshop, Southampton, U.K., pp. 29–32 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Delmonte, R., Tonelli, S., Boniforti, M.A.P., Bristot, A. (2006). VENSES – A Linguistically-Based System for Semantic Evaluation. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds) Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment. MLCW 2005. Lecture Notes in Computer Science(), vol 3944. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11736790_20
Download citation
DOI: https://doi.org/10.1007/11736790_20
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33427-9
Online ISBN: 978-3-540-33428-6
eBook Packages: Computer ScienceComputer Science (R0)