Persona: Gonzalo Arroyo, Julio Antonio
Cargando...
Dirección de correo electrónico
julio@lsi.uned.es
ORCID
0000-0002-5341-9337
Fecha de nacimiento
Proyectos de investigación
Unidades organizativas
Puesto de trabajo
Apellidos
Gonzalo Arroyo
Nombre de pila
Julio Antonio
Nombre
15 resultados
Resultados de la búsqueda
Mostrando 1 - 10 de 15
Publicación Detectando Influencers en Medios Sociales utilizando la información de sus seguidores(Sociedad Española para el Procesamiento del Lenguaje Natural, 2020-03) Rodríguez Vidal, Javier; Gonzalo Arroyo, Julio Antonio; Plaza Morales, LauraDada la tarea de encontrar influencers en un dominio dado (i.e. banking) en una red social, en este artículo investigamos (i) la importancia de caracterizar a los seguidores para la detección automática de influencers; (ii) la manera más efectiva de combinar señales obtenidas de los seguidores y de los perfiles principales para la detección automática de influencers. En este trabajo, hemos modelado el discurso usado por los usuarios en dos dominios, automotive y banking, así como el lenguaje utilizado por los influencers en dichos dominios y por sus seguidores, y utilizamos estos Modelos de Lenguaje para estimar la probabilidad de ser un influencer. Nuestro mayor descubrimiento es que los influencers no sólo dependen de su conocimiento sobre el dominio sino del de sus seguidores; por lo tanto, cuanto mayor conocimiento y número de expertos haya entre sus seguidores, mayor será la probabilidad que el perfil sea de un influencer.Publicación Combining evaluation metrics via the unanimous improvement ratio and its application in weps clustering task(Association for the Advancement of Artificial Intelligence, 2011-12-01) Artiles Picón, Javier ; Verdejo, M. Felisa; Amigo Cabrera, Enrique; Gonzalo Arroyo, Julio AntonioMany Artificial Intelligence tasks cannot be evaluated with a single quality criterion and some sort of weighted combination is needed to provide system rankings. A problem of weighted combination measures is that slight changes in the relative weights may produce substantial changes in the system rankings. This paper introduces the Unanimous Improvement Ratio (UIR), a measure that complements standard metric combination criteria (such as van Rijsbergen's F-measure) and indicates how robust the measured differences are to changes in the relative weights of the individual metrics. UIR is meant to elucidate whether a perceived difference between two systems is an artifact of how individual metrics are weighted. Besides discussing the theoretical foundations of UIR, this paper presents empirical results that confirm the validity and usefulness of the metric for the Text Clustering problem, where there is a tradeoff between precision and recall based metrics and results are particularly sensitive to the weighting scheme used to combine them. Remarkably, our experiments show that UIR can be used as a predictor of how well differences between systems measured on a given test bed will also hold in a different test bed.Publicación An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results(Association for Computational Linguistics Note:, 2020-07-01) Amigo Cabrera, Enrique; Gonzalo Arroyo, Julio Antonio; Mizzarro, Stefano; Carrillo de Albornoz Cuadrado, Jorge AmandoIn Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as positive, neutral, negative in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.Publicación The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing(Association for Computational Linguistics, 2025-01-01) Marco Remón, Guillermo; Gonzalo Arroyo, Julio Antonio; Fresno Fernández, Víctor DiegoRecent studies comparing AI-generated and human-authored literary texts have produced conflicting results: some suggest AI already surpasses human quality, while others argue it still falls short. We start from the hypothesis that such divergences can be largely explained by genuine differences in how readers interpret and value literature, rather than by an intrinsic quality of the texts evaluated. Using five public datasets (1,471 stories, 101 annotators including critics, students, and lay readers), we (i) extract 17 reference-less textual features (e.g., coherence, emotional variance, average sentence length...); (ii) model individual reader preferences, deriving feature importance vectors that reflect their textual priorities; and (iii) analyze these vectors in a shared “preference space”. Reader vectors cluster into two profiles: _surface-focused readers_ (mainly non-experts), who prioritize readability and textual richness; and _holistic readers_ (mainly experts), who value thematic development, rhetorical variety, and sentiment dynamics. Our results quantitatively explain how measurements of literary quality are a function of how text features align with each reader’s preferences. These findings advocate for reader-sensitive evaluation frameworks in the field of creative text generation.Publicación Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?(Association for Computational Linguistics, 2024-11-01) Marco Remón, Guillermo; Gonzalo Arroyo, Julio Antonio; Mateo-Girona, M. Teresa; Castillo Santos, Ramón José del; European CommissionAre LLMs ready to compete in creative writing skills with a top (rather than average) novelist? To provide an initial answer for this question, we have carried out a contest between Patricio Pron (an awarded novelist, considered one of the best of his generation) and GPT-4 (one of the top performing LLMs), in the spirit of AIhuman duels such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol. We asked Pron and GPT-4 to provide thirty titles each, and then to write short stories for both their titles and their opponent’s. Then, we prepared an evaluation rubric inspired by Boden’s definition of creativity, and we collected several detailed expert assessments of the texts, provided by literature critics and scholars. The results of our experimentation indicate that LLMs are still far from challenging a top human creative writer. We also observed that GPT-4 writes more creatively using Pron’s titles than its own titles (which is an indication of the potential for human-machine co-creation). Additionally, we found that GPT-4 has a more creative writing style in English than in Spanish.Publicación Automated metric analysis of Spanish Poetry: two complementary approaches(IEEE, 2021-03-30) Marco Remón, Guillermo; De la Rosa, Javier; Gonzalo Arroyo, Julio Antonio; Ros Muñoz, Salvador; González Blanco, ElenaThe automatic metric analysis (commonly referred to as scansion) of Spanish poetry is not a trivial problem since it combines the nuances of the language, the different poetic traditions related to melodic patterns, and the personal stylistic preferences and intentions of the author. In this paper, we explore two alternative algorithmic approaches tailored to different applications scenarios. The first approach, Rantanplan, is a rule-based method that consists of four Natural Language Processing modules that work together to perform scansion and other related analysis: Part of Speech tagging, syllabification, stress assignment, and metrical adjustment. The second approach, Jumper, explores the possibility of performing scansion without syllabification, with a twofold purpose: to minimize the errors propagated in different parts of the linguistic processing pipeline (including the syllabification step), and to improve the efficiency of the process. Both systems outperform the state of the art and provide either a more informative solution (suitable, for instance, for teaching purposes) or a more efficient processing (when a correct scansion is all the linguistic knowledge required, as in scholar philological studies). The combined use of both systems turns out to provide a practical tool to clean-up manual annotation errors in corpora.Publicación Automatic Generation of Entity-Oriented Summaries for Reputation Management(Springer, 2020-04-01) Rodríguez Vidal, Javier; Verdejo, Julia; Carrillo de Albornoz Cuadrado, Jorge Amando; Amigo Cabrera, Enrique; Plaza Morales, Laura; Gonzalo Arroyo, Julio AntonioProducing online reputation summaries for an entity (company, brand, etc.) is a focused summarization task with a distinctive feature: issues that may affect the reputation of the entity take priority in the summary. In this paper we (i) present a new test collection of manually created (abstractive and extractive) reputation reports which summarize tweet streams for 31 companies in the banking and automobile domains; (ii) propose a novel methodology to evaluate summaries in the context of online reputation monitoring, which profits from an analogy between reputation reports and the problem of diversity in search; and (iii) provide empirical evidence that producing reputation reports is different from a standard summarization problem, and incorporating priority signals is essential to address the task effectively.Publicación The contribution of linguistic features to automatic machine translation evaluation(2009-08-02) Giménez, Jesús; Verdejo, M. Felisa; Amigo Cabrera, Enrique; Gonzalo Arroyo, Julio AntonioA number of approaches to Automatic MT Evaluation based on deep linguistic knowledge have been suggested. However, n-gram based metrics are still today the dominant approach. The main reason is that the advantages of employing deeper linguistic information have not been clarified yet. In this work, we propose a novel approach for meta-evaluation of MT evaluation metrics, since correlation cofficient against human judges do not reveal details about the advantages and disadvantages of particular metrics. We then use this approach to investigate the benefits of introducing linguistic features into evaluation metrics. Overall, our experiments show that (i) both lexical and linguistic metrics present complementary advantages and (ii) combining both kinds of metrics yields the most robust metaevaluation performance.Publicación Automatic Detection of Influencers in Social Networks: Authority versus Domain signals(Wiley, 2019-01-07) Rodríguez Vidal, Javier; Anaya Sánchez, Henry; Gonzalo Arroyo, Julio Antonio; Plaza Morales, LauraGiven the task of finding influencers (opinion makers) for a given domain in a social network, we investigate (a) what is the relative importance of domain and authority signals, (b) what is the most effective way of combining signals (voting, classification, learning to rank, etc.) and how best to model the vocabulary signal, and (c) how large is the gap between supervised and unsupervised methods and what are the practical consequences. Our best results on the RepLab dataset (which improves the state of the art) uses language models to learn the domain-specific vocabulary used by influencers and combines domain and authority models using a Learning to Rank algorithm. Our experiments show that (a) both authority and domain evidence can be trained from the vocabulary of influencers; (b) once the language of influencers is modeled as a likelihood signal, further supervised learning and additional network-based signals only provide marginal improvements; and (c) the availability of training data sets is crucial to obtain competitive results in the task. Our most remarkable finding is that influencers do use a distinctive vocabulary, which is a more reliable signal than nontextual network indicators such as the number of followers, retweets, and so on.Publicación MT Evaluation : human-like vs. human acceptable(2006-07-17) Giménez, Jesús; Màrquez, Lluís; Amigo Cabrera, Enrique; Gonzalo Arroyo, Julio AntonioWe present a comparative study on Machine Translation Evaluation according to two different criteria: Human Likeness and Human Acceptability. We provide empirical evidence that there is a relationship between these two kinds of evaluation: Human Likeness implies Human Acceptability but the reverse is not true. From the point of view of automatic evaluation this implies that metrics based on Human Likeness are more reliable for system tuning. Our results also show that current evaluation metrics are not always able to distinguish between automatic and human translations. In order to improve the descriptive power of current metrics we propose the use of additional syntax-based metrics, and metric combinations inside the QARLA Framework.