TSNLP Publications, Reports, and Bibliography

TSNLP Bibliography

Language Research Engineering Convention, London (July 1994).

This paper describes the LRE project TSNLP (Test Suites for Natural Language Processing), which is concerned with some central issues in the design and use of test suites. The project combines theoretical research with practical implementations, aiming to provide generally usable tools and test data together with reports discussing the theoretical background. The paper begins by setting out the motivation, aims, and present state of the project, then examines the methodological issues behind it.

Translation and the Computer, 16th ASLIB Conference (November 1994).

Cranfield International Conference on Machine Translation (November 1994)

Conference on Linguistic Databases, Groningen (March 1994)

We present recent results from the LRE project TSNLP (Test Suites for Natural Language Processing) which is concerned with central issues in the design and use of test suites. The paper focusses on (i) the motivation for test suites in comparison to (annotated) corpora, (ii) the theory-neutral annotation schema developed in TSNLP; and (iii) the construction of a linguistic database as a (virtual) meta test suite to ease applications in diagnosis and evaluation.

Fourth International Conference on Cognitive Science of Natural Language Processing, Dublin (1995)

Test suites have long been accepted as a useful evaluation tool in Natural Language Processing, since they provide a more or less systematic collection of specially constructed linguistic examples (e.g. sentences) with annotations and other information. However, existing test suites tend to be relatively unsystematic and lack generality (having been constructed with particular systems in mind). Moreover, there is no established methodology which a system developer or other evaluator can follow in constructing a test suite of their own.

The paper describes the goals and achievements to date of the TSNLP project (Test Suites for Natural Language Processing), an LRE project funded by the CEC, which seeks to address these problems. In particular, the project aims to produce realistic and general guidelines for test suite construction, and to construct substantial test data in three languages (English, French, and German). The bulk of the data (several thousand test items) will cover ``core'' syntactic phenomena and will be suitable for testing any syntactic-based system, but some application- specific data (for parsers, grammar checkers and controlled language checkers) will also be written.

To enhance their use and reusability, the data is being mounted onto a database, for ease of access and manipulation. The construction methodology and the test suites will be validated by testing a number of NLP products.

The paper concentrates on design issues (eg. the need for systematicity in both well-formed and ill-formed data, the ``exhaustive'' coverage of closed classes, and consistency of annotation across languages) and discusses the tools that have been designed to aid and semi-automate the construction process (namely a generation tool and lexical replacement tool).

All the results of the project, including actual test suites are, or will be, in the public domain.

Language Engineering Convention, London (October 1995).

Workshop on Controlled Language Applications (CLAW), Leuven (March 1996).

TSNLP produced guidelines and methodology for test suite writing, as well as a substantial amount of annotated test data. To test the guidelines, the created data, and the methodology for controlled language test suites, they were used to test the controlled language checker deveoped during the SECC project (Simplified English Controlled Language Checker). We report here on what test suites for controlled language should contain, how we realised the actual test suite, and on the conclusions we could draw from all this.

Traitement Automatique du Langage Naturel (TALN) Conference, Marseille (May 1996).

Le nombre d'applications dans le domaine du TALN n'a cessé d'augmenter lors de ces dernières années. Ce développement va de pair avec une demande croissante d'outils pour évaluer ces applications. Le projet TSNLP répond à cette demande en proposant une méthodologie et des outils pour l'évaluation à l'aide de jeux de phrases-test.

Mis à part une méthodologie élaborée pour la construction de matériel de test, TSNLP a créé la plus grand base de données de jeux de phrases-test actuellement disponible pour le français, l'anglais et l'allemand. En outre, ce projet a développé des outils qui facilitent la construction, le stockage et l'accès aux données.

Les résultats de TSNLP seront publiques. Le projet propose ainsi des ressources linguistiques qui pourraient devenir une proposition de standard pour un modèle d'évaluation pour tout utilisateur d'applications dans le domaine du TALN.

Le traitement automatique du langage et les applications industrielles
Natural Language Processing and Industrial Applications,
Moncton, New-Brunswick, Canada (June 1996).

Test suites are a useful evaluation tool for developers and users of NLP products. The TSNLP project sets new standards for test suite design. The paper gives an overview of the TSNLP design and methodology and describes how the TSNLP data and methodology can be used in practice to provide a reliable assessment method of the linguistic capabilities of NLP products.

COLING, Kopenhagen (August 1996).

The growing language technology industry needs measurement tools to allow researchers, engineers, managers, and customers to track development, evaluate and assure quality, and assess suitability for a variety of applications.

The TSNLP (Test Suites for Natural Language Processing) project has investigated various aspects of the construction, maintenance and application of systematic test suites as diagnostic and evaluation tools for NLP applications. The paper summarizes the motivation and main results of TSNLP: besides the solid methodological foundation of the project, TSNLP has produced substantial (i.e. larger than any existing general test suites) multi-purpose and multi-user test suites for three European languages together with a set of specialized tools that facilitate the construction, extension, maintenance, retrieval, and customization of the test data.

The publicly available results of TSNLP represent a valuable linguistic resource that has the potential of providing a wide-spread pre-standard diagnostic and evaluation tool for both developers and users of NLP applications.

John Nerbonne (editor): Linguistic Databases. CSLI Lecture Notes (forthcoming).

The objective of the TSNLP project is to construct test suites in three different languages building on a common basis and methodology. Specifically, TSNLP addresses a range of issues related to the construction and use of test suites. The main goals of the project are to:

• provide guidelines for the construction of test suites to facilitate a coherent and systematic data collection;
• develop a rich annotation schema that is maximally neutral with respect to particular linguistic theories or specific evaluation and application types (section 1.2.1);
• construct substantial test fragments in English, German, and French that are applicable to different NLP application types (viz. parsers, grammar and controlled language checkers) (section 1.2.2);
• design and implement an extensible linguistic database to store, access and manipulate the test data (section 1.2.3); and to
• investigate methods and tools for (semi-) automated test suite construction.

In the present paper the authors take the opportunity to present some of the recent outcome of TSNLP to the community of language technology developers as well as to potential users of NLP systems. Accordingly, the presentation puts emphasis on practical aspects of applicability and plausibility rather than on theoretically demanding research topics; the TSNLP results presented are of both methodological and technological interest.

3rd ALEP User Group Workshop, Saarbrücken (February 1997).

A recent addition to the ALEP grammar engineering platform is described: the test suite apparatus and test data produced in the TSNLP project have been seamlessly integrated with the ALEP task executor.

The resulting test suite extension to ALEP is well-suited to substitute for the existing naive testing environment, greatly increases testing and report generation flexibility and fixes several (previously unknown) errors in the timing and coverage measures computed by the test suite processor. For downward compatibility the previous testing functionality is preserved in the current ALEP version (3.2 as of jan-97).

University of Essex, March 1994

University of Essex, September 1994

University of Essex, September 1994

University of Essex, September 1994

University of Essex, December 1994

University of Essex, January 1995

University of Essex, April 1995

DFKI Saarbrücken, June 1995

Design parameters and key desiderata of the linguistic database used to store, maintain, and retrieve TSNLP test data are laid out, viz.

• usability: to allow for the application of the TSNLP methodology, technology, and test data to wide variety of diagnosis and evaluation purposes of very different applications by developers or users with varied backgrounds (e.g. academic vs. industrial);
• suitability: to meet the specific necessities of storing and maintaining natural language test data (e.g. in string processing) and to provide maximally flexible interfaces;
• extensibility: to enable and encourage users of the TSNLP database to add test data and annotations according to their needs without changes to the underlying database;
• portability and simplicity: to make the results of TSNLP available on several different hard- and software platforms free of royalties and easy to use.

ISSCO, Université de Genève, December 1995

University of Essex, November 1995