GermEval2024 Shared Task: GerMS-Detect – Sexism Detection in German Online News Fora
UPDATE 2024-05-01:
- the development phase has started today and the training and test files have been added to the downloads.
- the targets file for the trial task has been added to the downloads.
- the source code for the scoring program is now available. See the README.
UPDATE 2024-04-30: please not that we have updated the rules for open track and closed track and corresponding paper submissions as well as the terms and conditions
This shared task is about the detection of sexism/misogyny in comments posted in (mostly) German language to the comment section of an Austrian online newspaper.
The data was originally collected for the development of a classifier that supports the human moderators in detecting potentially sexist comments or identify comment fora with a high rate of sexist comments.
The texts identified as being sexist are often especially challenging for automatic classification because they often refer to some implied context which is not available or are formulated in a subtle way, avoiding strong or outright offensive language.
Texts have been annotated by several human annotators, with a large portion of the corpus being annotated by at least two annotators out of 8 (7 of whom are forum moderators).
The main aim of annotating the presence and strength of sexism/misogyny in the corpus was to identify comments which make it less welcoming to women to participate in the conversation. The full annotator guidelines with examples are available.
Since the sexism/misogyny present in this corpus is often present in a subtle form that avoids outright offensiveness or curse words, there are many texts where annotators have different opinions on whether the text should be regarded as sexist, or which degree of sexism should be assigned to it.
This shared task therefore also provides an opportunity to learn about how to deal with diverging opinions among annotators and how to train models on such a corpus which potentially can also inform about how diverging the opinions on a new text might be.
Subtasks
The shared task is divided into two subtasks:
- Subtask 1: predict a binary label indicating the presence or absence of sexism in different ways, based on the original grading of the texts by several annotators; also predict the majority grading assigned by annotators.
- Subtask 2: predict binary soft labels, based on the different opinions of annotators about the text; predict the distribution of the original gradings by annotators.
Closed and open tracks
Each of the Subtask 1 and Subtask 2 competitions are organized into two different tracks:
- Closed Track: in this track, models can only be trained with the provided training set. Models are limited as to what kind of data for pretraining is allowed. Only the closed track counts towards the competition of the shared task and a closed track submission is required for the submission of a paper. See the linked document for details.
- Open Track: in this track, anything goes: you can use language models, use your own training data (but you have to share it with the community) or use other interesting approaches. The open track does NOT count towards the competition ranking but has its own leader board and has been added to allow for the exploration of interesting strategies which may be hard to reproduce.
Codabench Competitions
Before applying for participation in one or more of the Codabench competitions, make sure you have read the terms and conditions and you have also filled out the registration form for you team.
- Subtask 1
- Subtask 2
Files
The files can be downloaded from the downloads page as they become available.
Timeline
- Trial phase: April 20 - April 29, 2024
- A small labeled dataset for training and a small unlabeled dataset to use for the submission are provided. This phase is for getting to know the problem, the dataset format, how to submit predictions, how submissions are evaluated and how the evaluation shows up on the leaderboard etc.
- Development phase: May 1 - June 5, 2024
- During this phase, a labeled training set and an unlabeled test set are made available. The training set will contain the updated labeled versions of the training and test set of the previous phase plus additional labeled examples. Submissions have to contain the predictions for the unlabeled test set and the evaluation of the submission will show up on the leaderbord.
- Competition phase: June 7 - June 25, 2024
- During this phase, the training data will consist of the training data of the previous phase plus the labeled test data of the previous phase and a new unlabeled test set is made available. You can upload the labels for the test set to the competition site and your latest submission will be the one that will be used for ranking. During that phase, the leaderboard is not shown. Please note that only submissions which adhere to all terms and rules are considered for the final ranking.
- A preliminary leaderboard/ranking is shown after the end of the competition phase.
- The final leaderboard/ranking will be available once paper review and final submission is completed, on July 21th 2024.
- Paper submission due: July 1, 2024
- Camera ready due: July 20, 2024
- Shared Task @KONVENS: 10 September, 2024
Feedback
If you have questions, if you encounter problems or if you want to give any other feedback for a subtask/trak, please use the corresponding forum for the Codabench competition (or just any forum if your feedback applies to all of them):
- Subtask 1 Closed Track Forum
- Subtask 1 Closed Track Forum
- Subtask 1 Closed Track Forum
- Subtask 1 Closed Track Forum
Only if your feedback involves confidential information you do not want to share on one of these forums, contact the Organizers directly by email: germeval2024 (AT) ofai (DOT) at
Organizers
The task is organized by the Austrian Research Institute for Artificial Intelligence (OFAI). The organizing team are:
- Brigitte Krenn (brigitte.krenn (AT) ofai.at)
- Johann Petrak (johann.petrak (AT) ofai.at)
- Stephanie Gross (stephanie.gross (AT) ofai.at)