GermEval2024 Shared Task: GerMS-Detect – Sexism Detection in German Online News Fora

UPDATE 2024-09-10:

UPDATE 2024-07-26:

UPDATE 2024-07-31:

UPDATE 2024-07-12:

UPDATE 2024-06-25

UPDATE 2024-05-01:

UPDATE 2024-04-30: please not that we have updated the rules for open track and closed track and corresponding paper submissions as well as the terms and conditions


This shared task is about the detection of sexism/misogyny in comments posted in (mostly) German language to the comment section of an Austrian online newspaper.

The data was originally collected for the development of a classifier that supports the human moderators in detecting potentially sexist comments or identify comment fora with a high rate of sexist comments.

The texts identified as being sexist are often especially challenging for automatic classification because they often refer to some implied context which is not available or are formulated in a subtle way, avoiding strong or outright offensive language.

Texts have been annotated by several human annotators, with a large portion of the corpus being annotated by at least four annotators out of 10 (7 of whom are forum moderators).

The main aim of annotating the presence and strength of sexism/misogyny in the corpus was to identify comments which make it less welcoming to women to participate in the conversation. The full annotator guidelines with examples are available.

Since the sexism/misogyny present in this corpus is often present in a subtle form that avoids outright offensiveness or curse words, there are many texts where annotators have different opinions on whether the text should be regarded as sexist, or which degree of sexism should be assigned to it.

This shared task therefore also provides an opportunity to learn about how to deal with diverging opinions among annotators and how to train models on such a corpus which potentially can also inform about how diverging the opinions on a new text might be.

Subtasks

The shared task is divided into two subtasks:

Closed and open tracks

Each of the Subtask 1 and Subtask 2 competitions are organized into two different tracks:

Codabench Competitions

Before applying for participation in one or more of the Codabench competitions, make sure you have read the terms and conditions and you have also filled out the registration form for you team.

Files

The files can be downloaded from the downloads page as they become available.

Timeline

Feedback

If you have questions, if you encounter problems or if you want to give any other feedback for a subtask/trak, please use the corresponding forum for the Codabench competition (or just any forum if your feedback applies to all of them):

Only if your feedback involves confidential information you do not want to share on one of these forums, contact the Organizers directly by email: germeval2024 (AT) ofai (DOT) at

Organizers

The task is organized by the Austrian Research Institute for Artificial Intelligence (OFAI). The organizing team are: