BA Colloquium- Cem Doruk Uenal

Abstract: Ubiquitous technologies train machine learning models on datasets collected from people, and the content quality of these datasets determine how well the model will perform once it is deployed. What a dataset contains, and therefore how a model trained on it may behave, can only be understood if the authors report it. For this thesis, all 85 articles in Volume 9, Issue 4 of the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies were reviewed and coded against a scheme of 24 parameters that cover aspects such as dataset characteristics, reporting on the participants the data was collected from, labeling procedures, and machine learning training details. The same corpus was then coded by an LLM-automated pipeline, and its coding was compared with the manual reference coding using accuracy, precision, recall and Cohen's Kappa to see whether an LLM can reproduce the same manual coding. In the manual coding, 63 (74.1%) of the 85 papers reported using a human-related dataset, and 42 of them trained a model on the dataset they reported. Ethnicity was the least reported participant characteristic, appearing in 11.9% of 42 papers using ML algorithms. The LLM agreed with the manual coding in 82.7% of 1833 coded cells; most disagreements (66.6%) were false positives. Basic information such as the data source and the number of participants was usually reported, but ethnicity, other participant characteristics and details of the labeling process were often missing. The LLM-automated annotation pipeline cannot replace human coding, but it can assist it, provided humans review its output again.