Cargando...
Fecha
2025-03-13
Derechos de acceso
info:eu-repo/semantics/openAccess
Título de la revista
ISSN de la revista
Título del volumen
Editorial
Citas
Resumen
La experiencia de las mujeres en redes sociales se ve afectada negativamente por la publicación y difusión masiva de contenido sexista. Esta tendencia repercute en la salud mental de las mujeres, así como la aceptación de actitudes y discursos sexistas. La propagación de las ideas sexistas no solo se realiza a través de mensajes de texto, sino también por medio de imágenes y memes. En este contexto, la detección de sexismo en redes sociales se plantea como un problema de detección multimedia. Para ello, nuestro trabajo presenta la detección en dos tipos de formato: tuits, mensajes puramente textuales, y memes, mensajes que combinan texto e imagen. Este trabajo se enmarca en la competición EXIST 2024, una de las tareas competitivas organizadas en el congreso CLEF 2024, que tiene como meta la detección y caracterización del sexismo en espectro amplio. Además, la competición promueve la implementación de técnicas de Learning With Disagreement, un nuevo paradigma de anotación de corpora que defiende que todos los procesos de anotación muestran desacuerdos y, por tanto, hay que trabajar con un gold standard que no se obtenga mediante la agregación de etiquetas. En este trabajo se describe el proceso de creación del dataset de memes de EXIST 2024. Primero se explican las tareas de la competición: identificación de sexismo, categorización de intención del emisor y clasificación en tipos de sexismo. Definimos qué consideramos como meme, y explicamos la concepción de la lista de términos semilla, la descarga de los memes y posterior filtrado manual, así como la extracción de cada texto. Este dataset está formado por 5000 memes en dos idiomas: inglés y en español. Cada meme ha sido etiquetado por seis anotadores distintos, cada uno perteneciente a un grupo demográfico, que denominamos cohorte, determinado por su género (hombres y mujeres) y edad (de 18 a 22 años, de 23 a 45 años y mayores de 46 años). Por último, se analizan los resultados de la anotación de las instancias por tarea y cohorte. Para la detección automática de sexismo se han diseñado sistemas textuales para la identificación y categorización de la intención del emisor en tuits y memes. Nuestro objetivo es desarrollar sistemas entrenados con todas las anotaciones y que generen como salida soft labels, etiquetas formadas por la probabilidad de cada clase, y hard labels, etiquetas únicas agregadas. Los sistemas presentados aprovechan el desacuerdo para realizar predicciones en cascada por medio de dos sistemas: uno más sencillo que determina los casos más claros, y un sistema más complejo, entrenado con más información o que hace uso de un ensemble de modelos que caracterizan cada cohorte por separado. El sistema más sencillo determina las soft labels de las instancias, del que se obtiene el nivel de acuerdo predicho. Si en una instancia el nivel de acuerdo supera un nivel de acuerdo umbral, las predicciones de dicha instancia se dan por válidas, si no, el segundo sistema predice las etiquetas de dicha instancia. Estos sistemas se evalúan para cada tarea en un dataset de validación y se discute su efectividad. Los sistemas en cascada han resultado superiores en la identificación de memes, donde los ensembles de caracterización de las cohortes han obtenido resultados superiores en la determinación de desacuerdos. Algunos de los sistemas fueron seleccionados para participar en la competición de EXIST 2024. En la competición, nuestros sistemas destacan en la detección de sexismo en memes, tanto en la tarea de identificación del sexismo, donde dos de los tres sistemas presentados han logrado la 1ª y 2ª posición en el ránking de la evaluación soft-soft, como en la categorización de la intención del emisor en memes, donde en las evaluaciones hard-hard y soft-soft uno de nuestros dos sistemas ha logrado la 1ª posición.
The experience of women on social media is negatively affected by the widespread publication and dissemination of sexist content. This trend impacts women’s mental health as well as the acceptance of sexist attitudes and discourse. The spread of sexist ideas occurs not only through text messages but also via images and memes. In this context, the detection of sexism on social media is posed as a multimedia detection problem. To address this, our work presents detection in two formats: tweets, which are purely textual messages, and memes, which combine text and images. This work is part of the EXIST 2024 competition, one of the competitive tasks organized at the CLEF 2024 conference, which aims to detect and characterize sexism across a broad spectrum. Additionally, the competition promotes the implementation of Learning With Disagreement techniques, a new corpus annotation paradigm that acknowledges that all annotation processes involve disagreements. Therefore, a gold standard should not be obtained simply through label aggregation. This paper describes the process of creating the EXIST 2024 meme dataset. First, we explain the competition tasks: sexism identification, source intention categorization, and classification into types of sexism. We define what we consider a meme and explain the conception of the seed term list, the downloading of memes, subsequent manual filtering, and the extraction of each text. This dataset consists of 5,000 memes in two languages: English and Spanish. Each meme has been annotated by six different annotators, each belonging to a demographic group, which we refer to as a cohort, defined by gender (men and women) and age (18–22 years, 23–45 years, and over 46 years). Finally, we analyze the annotation results per task and cohort. For automatic sexism detection, we designed textual systems for identifying sexism and source intention categorization in tweets and memes. Our goal is to develop systems trained on all annotations that generate both soft labels (labels formed by the probability of each class) and hard labels (aggregated unique labels). The presented systems leverage disagreement to make cascade predictions through two systems: a simpler one that determines the clearest cases and a more complex system, trained with additional information or using an ensemble of models that characterize each cohort separately. The simpler system determines the soft labels of the instances, from which the predicted agreement level is obtained. If an instance’s agreement level exceeds a set threshold, its predictions are considered valid; otherwise, the second system predicts the instance’s labels. These systems are evaluated for each task using a validation dataset, and their effectiveness is discussed. The cascade systems have proven superior in meme identification, where cohort-characterizing ensembles have achieved better results in determining disagreements. Some of the systems were selected to participate in the EXIST 2024 competition. In the competition, our systems excelled in sexism detection in memes, both in the sexism identification task—where two of the three presented systems ranked 1st and 2nd in the soft-soft evaluation ranking—and in the categorization of the sender’s intent in memes, where in the hard-hard and soft-soft evaluations, one of our two systems achieved 1st place.
The experience of women on social media is negatively affected by the widespread publication and dissemination of sexist content. This trend impacts women’s mental health as well as the acceptance of sexist attitudes and discourse. The spread of sexist ideas occurs not only through text messages but also via images and memes. In this context, the detection of sexism on social media is posed as a multimedia detection problem. To address this, our work presents detection in two formats: tweets, which are purely textual messages, and memes, which combine text and images. This work is part of the EXIST 2024 competition, one of the competitive tasks organized at the CLEF 2024 conference, which aims to detect and characterize sexism across a broad spectrum. Additionally, the competition promotes the implementation of Learning With Disagreement techniques, a new corpus annotation paradigm that acknowledges that all annotation processes involve disagreements. Therefore, a gold standard should not be obtained simply through label aggregation. This paper describes the process of creating the EXIST 2024 meme dataset. First, we explain the competition tasks: sexism identification, source intention categorization, and classification into types of sexism. We define what we consider a meme and explain the conception of the seed term list, the downloading of memes, subsequent manual filtering, and the extraction of each text. This dataset consists of 5,000 memes in two languages: English and Spanish. Each meme has been annotated by six different annotators, each belonging to a demographic group, which we refer to as a cohort, defined by gender (men and women) and age (18–22 years, 23–45 years, and over 46 years). Finally, we analyze the annotation results per task and cohort. For automatic sexism detection, we designed textual systems for identifying sexism and source intention categorization in tweets and memes. Our goal is to develop systems trained on all annotations that generate both soft labels (labels formed by the probability of each class) and hard labels (aggregated unique labels). The presented systems leverage disagreement to make cascade predictions through two systems: a simpler one that determines the clearest cases and a more complex system, trained with additional information or using an ensemble of models that characterize each cohort separately. The simpler system determines the soft labels of the instances, from which the predicted agreement level is obtained. If an instance’s agreement level exceeds a set threshold, its predictions are considered valid; otherwise, the second system predicts the instance’s labels. These systems are evaluated for each task using a validation dataset, and their effectiveness is discussed. The cascade systems have proven superior in meme identification, where cohort-characterizing ensembles have achieved better results in determining disagreements. Some of the systems were selected to participate in the EXIST 2024 competition. In the competition, our systems excelled in sexism detection in memes, both in the sexism identification task—where two of the three presented systems ranked 1st and 2nd in the soft-soft evaluation ranking—and in the categorization of the sender’s intent in memes, where in the hard-hard and soft-soft evaluations, one of our two systems achieved 1st place.
Descripción
Categorías UNESCO
Palabras clave
Detección de sexismo, Categorización de Sexismo, Aprendizaje mediante desacuerdos, EXIST, Memes, Sexism Detection, Sexism Categorization, Learning with Disagreement, EXIST, Memes
Citación
Ruiz García, Víctor. Trabajo fin de Máster: "Dataset y Sistemas Basados en Niveles de Acuerdo para la Detección de Sexismo en Tuits y Memes ". Universidad Nacional de Educación a Distancia (UNED), 2025
Centro
E.T.S. de Ingeniería Informática
Departamento
Lenguajes y Sistemas Informáticos