I am a part-time PhD Student working on evaluation of NLP datasets using tools such as taxonomies, inter-rater reliability statistics and item response models. In [2] we investigate how to automatically evaluate taxonomies using their intrinsic properties. In [1], we systematically compare distances that can be used to calculate annotator agreement in open-ended biomedical annotations. More recently, I have been working on leveraging item-response models to inform model training on the basis of annotations.
Recent Publications
All publications →Consensus Measures for Unstructured Biomedical Text Annotations · Abstract
Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.
(2026).
@misc{wullschleger2026consensusmeasuresunstructuredbiomedical, title = {Consensus Measures for Unstructured Biomedical Text Annotations}, author = {Pascal Wullschleger and Christian Kreis and Martin A. Walter and Jennifer Foster and Marc Pouly}, abstract = {Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.}, year = 2026, url = {https://arxiv.org/abs/2608.03529}, eprint = {2608.03529}, archiveprefix= {arXiv}, primaryclass= {cs.CL} }
Reference-Free Evaluation of Taxonomies
(2026). In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 25489–25507).
@inproceedings{wullschleger-etal-2026-reference, title = {Reference-Free Evaluation of Taxonomies}, author = {Wullschleger, Pascal and Zarharan, Majid and Daly, Donnacha and Pouly, Marc and Foster, Jennifer}, year = 2026, month = {jul}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, publisher = {Association for Computational Linguistics}, address = {San Diego, California, United States}, pages = {25489--25507}, doi = {10.18653/v1/2026.findings-acl.1273}, isbn = {979-8-89176-395-1}, url = {https://aclanthology.org/2026.findings-acl.1273/}, editor = {Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David}, abstract = {We introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, addressing error types not considered by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against ground truth taxonomies. We further demonstrate that our metrics can predict downstream performance in hierarchical classification when used with label hierarchies.} }
FoodTaxo: Generating Food Taxonomies with Large Language Models
(2025). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track) (pp. 784–803).
@inproceedings{wullschleger-etal-2025-foodtaxo, title = {FoodTaxo: Generating Food Taxonomies with Large Language Models}, author = {Wullschleger, Pascal and Zarharan, Majid and Daly, Donnacha and Pouly, Marc and Foster, Jennifer}, year = 2025, month = {jul}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)}, publisher = {Association for Computational Linguistics}, address = {Vienna, Austria}, pages = {784--803}, doi = {10.18653/v1/2025.acl-industry.55}, isbn = {979-8-89176-288-6}, url = {https://aclanthology.org/2025.acl-industry.55/}, editor = {Rehm, Georg and Li, Yunyao}, abstract = {We investigate the utility of Large Language Models for automated taxonomy generation and completion specifically applied to taxonomies from the food technology industry. We explore the extent to which taxonomies can be completed from a seed taxonomy or generated without a seed from a set of known concepts, in an iterative fashion using recent prompting techniques.Experiments on five taxonomies using an open-source LLM (Llama-3), while promising, point to the difficulty of correctly placing inner nodes.} }
Research Interests
NLP · Ontologies · Language models · RL · Annotation · Biomedical text mining · Deep learning