SimpleText@CLEF-2021 Pilot tasks
| Home | Call for papers | Important dates | Pilot tasks | |
| Program | Publications | Organisers | Contact |  |
SimpleText Pilot Task Guidelines
We invite you to submit both automatic and manual runs! Manual intervention should be reported.
Pilot Task 1: Selecting passages to include in a simplified summary – Content Simplification
Given an article from a major international newspaper general audience, this pilot task aims at retrieving from a large scientific bibliographic database with abstracts, all passages that would be relevant to illustrate this article. Extracted passages should be adequate to be inserted as plain citations in the original paper.
Output format:
A maximum of 1000 passages to be included in a simplified summary in a TSV (Tab-Separated Values) file with the following fields:
- run_id: Run ID starting with team_id_
- manual: Whether the run is manual {0,1}
- topic_id: Topic ID
- doc_id: Source document ID
- passage: Text of the selected passage
- rank: Passage rank
run_id manual topic_id doc_id passage rank
Evaluation
Sentence pooling and automatic metrics will be used to evaluate these results. The relevance of the source document will be evaluated as well as potential unresolved anaphora issues.
OUTPUT example:
| run_id | manual | topic_id | doc_id | passage | rank |
|---|---|---|---|---|---|
| ST1_1 | 1 | 1 | 3000234933 | People are becoming increasingly comfortable using Digital Assistants (DAs) to interact with services or connected objects. | 1 |
| ST1_1 | 1 | 1 | 3003409254 | big data and machine learning (ML) algorithms can result in discriminatory decisions against certain protected groups defined upon personal data like gender, race, sexual orientation etc. | 2 |
| ST1_1 | 1 | 1 | 3003409254 | Such algorithms designed to discover patterns in big data might not only pick up any encoded societal biases in the training data, but even worse, they might reinforce such biases resulting in more severe discrimination. | 3 |