expert consensus

Large language models (LLMs) are increasingly used in dental education, but research has focused mainly on answer accuracy. Whether LLMs can identify the cognitive demand of examination items remains unclear. This study evaluated LLM classification fidelity for endodontic multiple-choice questions (MCQs) across three educational taxonomies and compared four publicly accessible models. Two hundred endodontic MCQs from the Turkish Dentistry Specialty Examination (DUS, 2012–2025) were classified by ChatGPT, Gemini, Kimi, and DeepSeek according to Revised Bloom’s Taxonomy, the SOLO Taxonomy, and Miller’s Pyramid. A three-layer expert consensus procedure comprising calibration training, procedural audit, and structured joint deliberation established the expert consensus reference standard. Agreement was assessed using weighted Cohen’s kappa (κw) with bootstrap 95% confidence intervals; classification performance was evaluated using accuracy and macro-averaged F1. Inter-model differences in item-level accuracy were tested using Cochran’s Q with Kendall’s W, followed by Holm–Bonferroni-corrected pairwise McNemar tests. For Revised Bloom’s Taxonomy, Gemini (κw=0.676) and DeepSeek (κw=0.667) achieved substantial agreement, while Kimi (κw=0.428) and ChatGPT (κw=0.365) performed lower. For SOLO, Gemini achieved substantial agreement (κw=0.622), while the remaining models showed moderate agreement. For Miller’s Pyramid, agreement did not exceed fair, with ChatGPT highest (κw=0.372). Inter-model differences were significant for all three taxonomies (all p<0.001). ChatGPT ranked lowest for Revised Bloom and SOLO but highest for Miller. Revised Bloom levels of expert consensus reference standard increased significantly across examination years (Spearman ρ=0.301, p<0.001). LLMs showed meaningful classification fidelity for Revised Bloom and SOLO taxonomies, particularly Gemini and DeepSeek, but performance for Miller’s Pyramid remained weak. Bloom, SOLO, and Miller should not be treated as interchangeable frameworks, and LLM selection should be tailored to the taxonomy and assessment purpose. Answer accuracy and taxonomy classification fidelity represent distinct capabilities that warrant independent evaluation.

Traditional statistical models face the challenges of insufficient modeling capabilities and difficulty in interpreting complex, nonlinear, and high-dimensional data. Therefore, this paper combines fuzzy logic and deep learning methods to construct a framework model for identifying key factors for high-quality employment for college students. The model in this paper uses multi-channel encoding technology. We use the attention mechanism to fuse the features obtained from multi-channel encoding and input the self-organizing fuzzy inference structure for regression and classification analysis. The system can transform numerical prediction results into a semantic rule-driven decision-making process, which can significantly improve the model’s modeling capabilities and interpretability for uncertainty. Our experiments were conducted on 5,623 graduates. Experimental results on the data show that our model achieved an average mean square error of 0.831, a coefficient of determination of 0.892, and a classification accuracy of 0.865, which are significantly higher than traditional statistical and machine learning models. The corresponding experiments demonstrate the key role of cross-modal fusion and fuzzy reasoning modules in improving the results. Feature importance analysis shows that factors such as internship duration, GPA, and research participation have a key impact on high-quality employment, which is consistent with expert consensus.

Background Assessing dance performance in physical education teacher education (PETE) requires judges to integrate technical and aesthetic qualities. Analytic rubrics make relevant dimensions explicit, whereas Adaptive Comparative Judgment (ACJ) elicits holistic pairwise decisions and produces a relative performance scale. This study compared the outcomes and student perceptions associated with the two approaches. Methods Seventeen pre-service teachers assessed 46 video-recorded student dance performances using an analytic rubric and ACJ. An expert reference was derived from five PETE lecturers who had assessed the performances live using the rubric and subsequently evaluated the video recordings through ACJ. Analyses examined rubric reliability and ACJ precision, convergence between method-derived rankings, student–expert rank alignment, cross-condition rubric score differences, and students’ perceptions of ease of use, usefulness, transparency, and fairness. Results Rubric- and ACJ-derived rankings were strongly associated within the student panel (Spearman's ρ = .863) and the expert reference panel ( ρ = .879). Aggregated student rankings also aligned closely with the expert-reference rankings under rubric scoring ( ρ = .890) and ACJ ( ρ = .888), with no clear difference in alignment between the methods. Aggregated student rubric scores showed high reliability, ICC(2,17) = .955, while ACJ scale separation reliability was .780 for students and .836 for experts. Student video-based rubric scores were, on average, 0.77 points higher than the archived live expert consensus scores. Students perceived ACJ as easier to use but the analytic rubric as more transparent. Perceived usefulness did not differ clearly, while fairness ratings descriptively favored the rubric. Conclusion Analytic rubric scoring and ACJ produced strongly convergent aggregate rankings, and the aggregated student panel differentiated relative performance quality in ways closely aligned with the expert reference under both approaches. However, the methods differed in score interpretation and perceived transparency. The results support considering ACJ as a complementary approach for digitally supported assessment and moderation rather than as a direct replacement for criterion-based summative assessment.