Educational Psychology
Faculty 05

The International Association for Computerized Adaptive Testing (IACAT) invites you to submit a proposal for a presentation or a symposium at its biannual conference, which will take place on October 10 –12, 2021 at the Goethe University in Frankfurt am Main, Germany. We will accept the submission of proposals up until April 30, 2021, 11:59 PM CET. You can download the detailed call for papers by clicking here.

We welcome both methodological research and practical applications of computerized adaptive testing (CAT) and artificial intelligence (AI) in assessment. Click HERE for the IACAT 2021 Homepage.

December 9th, 2020

Prof. Dr. Andreas Frey, Dr. Thomas Dreisörner and Dr. Christoph König have been granted start-up funding for educational disadvantage due to the coronavirus. The project “Identifying and reducing educational disadvantage due to the coronavirus (CoBi)” will be funded by the Goethe-Corona-Fund of the Goethe University Frankfurt.

December 7th, 2020

Prof. Dr. Andreas Frey and Aron Fink, together with Prof. Dr. Christian Spoden and Dr. Sebastian Born have written a short version of the KAT-HS concept, which is now available. Their paper describes four major challenges when it comes to the development of high-quality exams in higher education and specifies how to meet them on the basis of a novel combination of methods, quality standards and approaches stemming from psychometrics and educational and psychological measurement. The concept’s feasibility is classified according to examination law and the general conditions for its implementation in the regular operation of higher education institutions are discussed.

The paper is available here: https://eleed.campussource.de/archive/13/5119

Frey, A., Spoden, C., Fink, A., & Born, S. (2020). Kompetenzorientierte individualisierte Hochschulklausuren und deren prüfungsrechtliche Einordnung [Competence-oriented adaptive exams in higher education and its classification according to examination law]. eleed, 13. Available at: urn:nbn:de:0009 5-51197.

November 13th, 2020

The research data of the project "Criterion referenced adaptive testing in higher education (KAT-HS)" are now available on request to other researchers for secondary analysis. The research data are available on the website of the research data center at the Institute for Educational Quality Improvement (IQB). Researchers interested in using the data can apply for access to the data using an online form.

The full literature references of the published data sets are as follows:

Frey, A., Spoden, C., & Fink, A. (2020). Kriteriumsorientiertes adaptives Testen in der Hochschule (KAT-HS) (Version 1) [Criterion-referenced Adaptive Testing in Higher Education (KAT-HS)] [Data set]. Berlin: IQB – Institut zur Qualitätsentwicklung im Bildungswesen. http://doi.org/10.5159/IQB_KAT-HS_v1

Frey, A., Spoden, C., & Fink, A. (2020). Kriteriumsorientiertes adaptives Testen in der Hochschule (KAT-HS). Kalibrierung [Criterion-referenced Adaptive Testing in Higher Education (KAT-HS). Calibration] - Campus File (Version 1) [Data set]. Berlin: IQB – Institut zur Qualitätsentwicklung im Bildungswesen. http://doi.org/10.5159/IQB_KAT-HS_CF_v1

October 26th, 2020

We will work together with the Otto-von-Guericke-University Magdeburg in the development of a computerized adaptive test to assess competencies in the field of Education for Sustainable Development (ESD). The work will take place within the BMBF project OIT-BNE (Use-oriented development of an Outcome-Indicator-Test for assessment and operationalisation of student's competencies in Education for Sustainable Development in Germany). The order was received today. We will support the OIT-BNE-project by providing expertise, adapting adaptive test software of our department to the project context and integrating the continuous calibration strategy proposed by Fink, Born, Spoden and Frey (2018) which was developed in the KAT-HS project.

October 23rd, 2020

Prof. Dr. Christian Spoden (DIE Bonn), Prof. Dr. Andreas Frey, Aron Fink and Patrick Naumann have written an article on competence-oriented computer-based university Exams in teacher training, which has now been published in the Waxmann anthology “Bildung, Schule, Digitalisierung” (Eds.: Kaspar, Becker-Mrotzek, Hofhues, König, & Schmeinck).

The chapter presents a concept for adaptive, competence-oriented computer-based exams (KAT-HS; see website) and describes their implementation in university teaching. With the KAT-HS concept, problems of common paper-based exams can be overcome and requirements from the Bologna reform can be implemented.

The chapter is available here:

Spoden, C., Frey, A., Fink, A. & Naumann, P. (2020). Kompetenzorientierte elektronische Hochschulklausuren im Studium des Lehramts [Competence-oriented computer-based university exams in teacher training]. In K. Kaspar, M. Becker-Mrotzeck, J. Hofhues, J. König & D. Schmeinck (Eds.), Bildung, Schule und Digitalisierung. Waxmann.

October 22nd, 2020

Prof. Dr. Andreas Frey investigated in cooperation with Prof. Dr. Christian Spoden and Dr. Jens Fleischer the effect of test anxiety and test-taking motivation on person fit by using large-scale data from a german mathematics proficiency test for (self-) evaluation. Person fit statistics make it possible to identify students whose ability estimates might be compromised by estimation bias due to the misfit of their response patterns. The results showed an effect of the test anxiety on the person fit in specific ability levels (e.g., a negative effect for students with high math ability). Considering this result, the usage of credibility functions as weights for individual response patterns is discussed as a possibility to increase the validity of test scores.

The paper is available here:

Spoden, C., Fleischer, J., & Frey, A. (2020). Person misfit, test anxiety, and test-taking motivation in a large-scale mathematics proficiency test for self-evaluation. Studies in Educational Evaluation, 67. https://doi.org/10.1016/j.stueduc.2020.100910

October 22nd, 2020

In cooperation with Dr. Lale Khorramdel and Dr. Kentaro Yamamoto of the Educational Testing Service in Princeton, NJ, USA, Dr. Christoph König and Prof. Dr. Andreas Frey investigated the benefits of fixed item parameter calibration (FIPC) for small-sample situations in large-scale assessments. In this study, recently published in Educational Measurement: Issues and Practice, they found that applying FIPC to PISA’s field trial data considerably reduced the sample size requirements for an accurate item calibration. This is beneficial for the flexibility of large-scale assessments, because a greater proportion of the field trial sample can be used to test novel features regarding their feasibility for the PISA main surveys. Consequently, it is easier for large-scale assessments to keep up with current developments in the field of psychometrics and to meet stakeholders’ needs while, at the same time, keeping costs reasonable.

The paper is available here:

König, C., Khorramdel, L., Yamamoto, K., & Frey, A. (2020). The Benefits of Fixed Item Calibration for Parameter Accuracy in Small Sample Situations in Large-Scale Assessments. Educational Measurement: Issues and Practice. Advance Online Publication. https://doi.org/10.1111/emip.12381

September 3rd, 2020

The recently published third edition of the textbook “Testtheorie und Fragebogenkonstruktion” [Test theory and questionnaire construction] (Eds.: Moosbrugger & Kelava) contains two chapters with the participation of Prof. Dr. Andreas Frey. In the chapter “Computerized adaptive testing” (Frey), basic ideas of adaptive testing, elementary components of adaptive test algorithms and the effects of adaptive testing are presented. Prof. Dr. Andreas Frey is also co-author of the chapter “Validity of test value interpretations” (Hartig, Frey & Jude), in which the concept of validity is described in detail.

The chapters can be found here:

Frey, A. (2020). Computerisiertes adaptives Testen [Computerized adaptive testing]. In H. Moosbrugger & A. Kelava (Eds.), Testtheorie und Fragebogenkonstruktion (3rd ed., pp. 501-525). Berlin, Heidelberg: Springer. doi: 10.1007/978-3-662-61532-4_20

Hartig, J., Frey, A., & Jude, N. (2020). Validität von Testwertinterpretationen [Validity of test score interpretations]. In H. Moosbrugger & A. Kelava (Eds.), Testtheorie und Fragebogenkonstruktion (3rd ed., pp. 529-545). Berlin, Heidelberg: Springer. doi: 10.1007/978-3-662-61532-4_21

August 28th, 2020

In cooperation with the Goethe-Institut e.V., Aron Fink and Prof. Dr. Andreas Frey examined the psychometric quality of the online German test Goethe-Test PRO: German for professionals. The test is designed as an adaptive online test and evaluates the listening and reading competencies at the workplace at levels A1-C2 of the Common European Framework of Reference for Languages. The results were published as a white paper.

The paper is available via following link:

Fink, A., & Frey, A. (2020). Goethe-Test PRO: Deutsch für den Beruf – Untersuchung der psychometrischen Güte [Goethe-Test PRO: German for professionals - Evaluation of the psychometric properties]. PsyArchives. https://doi.org/10.23668/psycharchives.3150

August 4th, 2020

The journal 'Educational Measurement: Issues and Practice' (EM:IP) of the National Council on Measurement in Education (NCME) called for creative graphical submissions for their cover competition in its 7th year. The best submissions that 'illustrate a fundamental statistical/psychometric concept or aid in analyzing or visualizing educational data' will be used as the cover of the following issues. Aron Fink was chosen as one of the four winners with his submission 'The Musical Relative of the Boxplot - The Violin Plot'. His data visualization appeared as the cover of the current Volume 39, Issue 2. His visualization showcases that violin plots 'provide an advanced visualization of the distributions in a compact and attractive format that has several advantages over common approaches like classical box or density plots'.

The corresponding article can be found under the following link:

Cui, Zhongmin (2020): On the Cover: Violin Plot. In: Educational Measurement: Issues and Practice 39 (2), S. 7. DOI: 10.1111/emip.12339.

June 12th, 2020

A press release was issued today at the Goethe University on the advisory services offered by "Mainkind" in Corona times. The counselling centre Mainkind provides support for children and adolescents with learning difficulties, ADHD and intellectual giftedness. Mainkind is attached to the professorship of Prof. Dr. Andreas Frey and is directed by Dr. Thomas Dreisörner. The press release can be found here (in German): https://www.muk.uni-frankfurt.de/89044948/Telefonischer_Rat_in_Sachen__ Homeschooling

June 9th, 2020

Once again the annual colloquium "Methods of Empirical Educational Research" was held from the 8th to 13th of March 2020 in Hirschegg, Austria. In the course of the colloquium, 13 experts discussed contributions on current methodological developments within the scope of educational research.

On behalf of the work unit, talks were held by:

• Prof. Dr. Andreas Frey on taking item position effects into account in the continuous calibration of large-scale assessments
• Dr. Christoph König on the behavior of Bayesian hierarchical IRT models
• Aron Fink on the consideration of the uncertainty in item parameter estimates in computerized adaptive testing

We look once again back at a very stimulating time in Kleinwalsertal and give our thanks to all participants!

 March 23th, 2020

Prof. Dr. Andreas Frey was appointed as expert for research methods to the advisory board of the research project "Promotion of social and emotional competences in vocational education and training: A study on medical assistants in training" at the Federal Institute for Vocational Education and Training (Bonn).

December, 17th, 2019

The workshop on the KAT-HS concept is now available free of charge via our website https://kat-hs.uni-frankfurt.de as video-on-demand.

Our workshop on the improvement of university exams (Introduction to the construction, administration and evaluation of competence-oriented e-exams) took place on 8th of October 2019 and was recorded on video. We have cut, edited and supplemented the videos to create a compact and informative online workshop on the construction of fair exams.

The video workshop provides a comprehensive overview of directly implementable procedures for construction, administration and evaluation of high-quality competence-oriented electronic exams. It covers all the steps necessary for a scientifically proven measurement of competence using written examinations: specification of learning objectives, transfer of learning objectives into competence requirements, item construction, setting of grades and passing thresholds, exam evaluation, identification of problematic items, maintenance of a constant evaluation scale over multiple examinations, and reporting of examination results. Additionally, insights into the computer-based exam application KAT-HS and its configuration is given.

The video workshop was created by Patrick Naumann, Prof. Dr. Andreas Frey, Prof. Dr. Christian Spoden and Aron Fink. Registered users of the website https://kat-hs.uni-frankfurt.de can watch the videos free of charge in the download section of that page.

December, 16th, 2019

This year, our annual Christmas get-together was spent in the Escape Room in “Cooky's”. Split up into two groups, each group was determined to solve the riddles of the “laboratory”, resulting in a tie. The second portion of the night was spent with Äppler (cider) and regional specialities in the traditional cider restaurant “Fichtekränzi” in Sachsenhausen, Frankfurt.

We spent a great evening together to ring in the festive season. The entire department wishes on this occasion everyone dropping by our page a happy, festive season and a successful new year! (A.S.)

December, 10th, 2019

Our workshop on the improvement of university exams (Introduction to the construction, administration and evaluation of competence-oriented e-exams) took place on 8^th of October 2019 at the Interdisciplinary College for University Teaching at the Goethe University Frankfurt. In the workshop, a comprehensive overview of procedures for construction, administration and evaluation of high-quality competence-oriented electronic exams was provided.

The workshop covered all the steps necessary for a scientifically proven measurement of competence using written examinations: specification of learning objectives, transfer of learning objectives into competence requirements, item construction, setting of grades and passing thresholds, exam evaluation, identification of problematic items, maintenance of a constant evaluation scale over multiple examinations, and reporting of examination results. Additionally, insights into the computer-based exam application KAT-HS and its configuration were given. The participants had the opportunity to carry out steps of the KAT-HS concept practically for an exam from their own field of activity.

The workshop was organized by Patrick Naumann and Prof. Dr. Andreas Frey and was supported by Prof. Dr. Christian Spoden, Aron Fink and Irina Kuzmina. Registered users of the website https://kat-hs.uni-frankfurt.de/ can download workshop materials such as the presentation slides.

October, 08th, 2019

On the 18^th of September 2019 the symposium “Applications of the Item-Response-Theory (IRT)” took place during the 14^th Conference of the Subdivision “Methods and Evaluation” of the German Psychological Society (DGPs) in Kiel, Germany. The symposium gave insight into the applications of IRT in different diagnostic application contexts. Patrick Naumann presented the results of a simulation study entitled “Item Parameter Drift in Small Samples”, which compared the performances of three methods for the detection of item parameter drift using the Continuous Calibration Strategy (CCS).  In his talk “Consideration of Item Position Effects in Computerized Adaptive Testing with the Continuous Calibration Strategy” Prof. Dr. Andreas Frey presented comprehensive results on the effectiveness of the CCS considering item position effects. Prof. Dr. Christian Spoden gave a talk entitled “Adaptive and Partially Adaptive Measurement of Individual Change”, in which he reported on the use of CAT for measuring change of learning processes. Further information about the FGME can be found here.

September, 18th, 2019

From 12^th to 16^th August the 18^th conference of the European Association for Research on Learning and Instruction (EARLI) took place in Aachen. The conference (“Thinking Tomorrow's Education: Learning from the past, in the present and for the future”) focussed on results and ideas on state-of-the-art research in learning and instruction. Aron Fink held a talk entitled “Testing an explanatory model for the intention to use adaptive and non-adaptive e-exams by the university teaching staff”. This presentation focused on the factors influencing the intention of university lecturers to use (adaptive) e-examinations. Dr. Christoph König “Reducing sample size requirements in large-scale assessments for improving test design flexibility” presented an approach for increasing the flexibility of international large-scale assessments with Fixed Item Parameter Calibration (FIPC). In this approach the parameters of already previously calibrated items (such as the trend items in PISA) are fixed to their known estimates, and only the new item parameters (such as the newly developed items in PISA) are estimated.

August, 16th, 2019

The 50^th Meeting of the European Mathematical Psychology Group (EMPG) took place in Heidelberg from 5^th to 7^th August 2019. Dr. Christoph König had a talk entitled “Alternatives to the Inverse Wishart distribution in Bayesian hierarchical IRT models”. Based on results of a simulation study, the standard Inverse Wishart distribution (the most popular prior distribution for covariance matrices in Bayesian hierarchical Item Response Theory models) was compared to the performance of three alternative specifications (scaled IW, hierarchical IW, and a separation strategy based on the Cauchy distribution and the Cholesky factor of the correlation matrix) in terms of parameter recovery (Bias and RMSE) in the hierarchical two-parameter logistic (2PL) IRT model. The results show a superior performance of the alternative specifications.

August, 8th, 2019

The work unit Educational Psychology with focus on Counselling, Measurement, and Evaluation welcomes Nicola Hericks as its newest member!

July, 1st, 2019

The conference of the International Association for Computerized Adaptive Testing (IACAT) took place in Minneapolis from 10th to 13th June 2019. Prof. Dr. Andreas Frey presented results on item position effects in the continuous calibration strategy, which he examined together with Aron Fink and Prof. Dr. Christian Spoden. The presentation was entitled “Consideration of Item Position Effects in CAT with the Continuous Calibration Strategy”. The continuous calibration strategy (Fink et al., 2018) allows the use of computerized adaptive tests even in small samples. At the IACAT conference, the results of a Monte Carlo simulation study were presented. In this study, the suitability of the continuous calibration strategy was examined in the presence of item position effects.

Further information on the IACAT conference can be found at http://iacat.org/.

June, 14th, 2019

Dr. Sebastian Born, Aron Fink, Prof. Dr. Christian Spoden and Prof. Dr. Andreas Frey have published an article on equating methods in the continuous calibration strategy in the journal “Frontiers in Psychology”. They investigated different methods for scale transformation using the continuous calibration strategy (Fink et al., 2018) in computerized adaptive tests regarding the estimation accuracy of the item parameters (mean/mean, mean/sigma, Haebara and Stocking-Lord). The characteristic curve methods, in particular Stocking-Lord, turned out to be suitable methods for scale transformation in the continuous itempool calibration for computerized adaptive testing. The article can be found by clicking on the link in the doi.

Born, S., Fink, A., Spoden, C., & Frey, A. (2019). Evaluating Different Equating Setups in the Continuous Item Pool Calibration for Computerized Adaptive Testing. Frontiers in Psychology, 10, 1277. doi: 10.3389/fpsyg.2019.01277.

June, 6th, 2019

The German Research Foundation (DFG) is funding the project "Online Calibration of Item Response Time Models" with their program “Initiation of International Collaborations”. As part of the project, Prof. Dr. Alison Cheng and Prof. Dr. Cheng Liu from the University of Notre Dame (USA) will be visiting the department Educational Psychology with focus on Counselling, Measurement, and Evaluation in June and July 2019. We are looking forward to our guests!

May 13th, 2019

The work unit Educational Psychology with focus on Counselling, Measurement, and Evaluation welcomes Patrick Naumann as its newest member!

April 2nd, 2019

From 10.3.-15.3.2019 the annual colloquium "Methods of Empirical Educational Research" was held in Hirschegg, Austria. During the colloquium, 14 experts discussed contributions on current methodological developments. Our department was represented by contributions from Prof. Dr. Andreas Frey, Dr. Christoph König and Aron Fink. We look back again on a very stimulating time in Kleinwalsertal!

March 26th, 2019

The work unit Educational Psychology with its focus on Counselling, Measurement, and Evaluation contributed three talks to this year’s Conference of the Society for Empirical Educational Research (GEBF) in Cologne. Prof. Dr. Andreas Frey gave a talk about the use of computerized adaptive testing in repeated small sample surveys, Shirin Esmaeili-Bijarsari gave a talk about the acceptance of criterion-referenced computerized adaptive university exams from students, and Aron Fink gave a talk about influencing factors on the intention of lecturers to use (adaptive) computerized university exams.

February 28th, 2019

A paper on how to consider multiple item context effects in computerized multidimensional tests was published recently in Frontiers in Psychology. Dr. Norman Rose, Prof. Dr. Gabriel Nagy, Prof. Dr. Benjamin Nagengast, Prof. Dr. Andreas Frey and Prof. Dr. Michael Becker describe a generalized linear mixed model (GLMM) parameterization of the multidimensional Rasch model that allows considering multiple item position effects. They illustrate this approach with data from an item calibration study of a multidimensional computerized test assessing student competencies in mathematics, science, and reading. The paper is available as a download on the homepage of the journal: https://doi.org/10.3389/fpsyg.2019.00248.

February 25th, 2019

Prof. Dr. Andreas Frey was appointed to the advisory board of the KERMIT-Online project of the Free and Hanseatic City of Hamburg and the Hanseatic City of Bremen.

February 20th, 2019

Prof. Dr. Andreas Frey, in cooperation with Prof. Dr. Johannes Hartig, contributed a chapter on competence diagnostics to the new Handbook of School Education, edited by Dr. Marius Harring, Prof. Dr. Carsten Rohlfs and Prof. Dr. Michaela Gläser-Zikuda. The authors introduce the basic questions of competence diagnostics and illustrate the typical planning and administration of a competence test. The Handbook of School Education is available for purchase at http://www.utb-shop.de/handbuch-schulpadagogik-9518.html.

December 5th, 2018

A paper on the illustration of a novel response-time based method for modeling omitted responses in computer-based testing was published recently in Behaviormetrika. Prof. Dr. Andreas Frey, Prof. Dr. Christian Spoden, Prof. Dr. Frank Goldhammer und Dr. S. Franziska C. Wenzel derive the method from the theory of missing data problems of Rubin and colleagues and embed it in an item response theory framework. They illustrate the novel method with responses given by students to items of a computer-based ICT skills test. The paper is available as a download on the homepage of the journal: https://rdcu.be/bczWd.

November 20th, 2018

The second part of the Special Topic “Advances in Educational Measurement”, edited by Prof. Dr. Andreas Frey, Dr. Christoph König and Prof. Dr. Christian Spoden, has been published in Psychological Test and Assessment Modeling. The Special Topic presents a series of studies investigating means and possibilities to make educational measurement, and its statistical modeling approaches, more personalized and flexible. The increasing use of digital devices in educational processes requires educational measurement to be able to adapt accordingly. The Special Topic is available at https://tinyurl.com/y7m77jk9. The first part, published in June, is available at https://tinyurl.com/ychg9p3y.

October 18, 2018

At the 51. Congress of the German Psychological Society (DGPs) in Frankfurt am Main, Prof. Dr. Andreas Frey presented a position paper concerning “University Exams in the 21st Century”.

October 2nd, 2018

A paper on the development of a computerized adaptive test was published recently in the Journal of Computerized Adaptive Testing. Prof. Dr. Christian Spoden, Prof. Dr. Andreas Frey, and Dr. Raphael Bernhard show that the development of such tests, which is widely regarded as time-consuming and resource-intensive, can be accomplished in a relatively short timeframe. They illustrate this with three tests (measuring student competencies in reading, mathematics, and science), which had to be developed and piloted within an 18-month period, and were used directly afterward in a large nationwide research initiative. The paper is available as a download on the homepage of the journal: http://iacat.org/jcat/index.php/jcat/article/view/70.

October 2nd, 2018