Analysis and results
The area of analysis is managed by CITO’s psychometrician Gunter Maris.
A brief overview
Prior to the Main Study in 2011, SurveyLang conducted a Field Trial: the analysis that now follows provides information on the quality of the test items (both language tests and questionnaires) and in-country translations. This allowed any arising issues to be resolved before the Main Study.
After the conclusion of the Main Study, the Language Tests completed by the students sampled in participating countries were marked and the Questionnaires’ results analysed. This process not only examined students’ responses, but also the way the tests themselves functioned. At this stage, particular attention was paid to any differences found across languages and countries.
The data from the Language Tests and Questionnaires were also analysed together to study the relationships between performance and concepts covered by the Questionnaires. A key focus was looking at the relationships, both similarities and differences across languages and countries. These are presented in the Final Report. The European Commission may use the analysis in the report to make future policy recommendations. Countries participating in the survey are also able to examine their own data and report on the findings relevant to their country and context.
Marking of writing
Unlike the listening and reading tests and the Questionnaires, which could be easily marked as responses were predominantly multiple-choice, the marking of writing required a different approach. SurveyLang developed a markscheme specific to its assessments and provided exemplar tasks and training on use of this markscheme. Marking was completed in-country with a sub-sample quality controlled centrally by SurveyLang.
Data analysis methodology
The language testing data were analysed using a One Parameter Logistic Model (OPLM; Verhelst & Glas, 1995; Verhelst, Glas & Verstralen, 1995). The basic approach is that the data sets for each of these three skills represent a unidimensional scale. Exploratory and confirmatory factor analysis is used to examine the relationships in more detail.
Further advanced data analysis techniques such as cluster analysis and multidimensional scaling are applied to detect and analyse systematic differences between countries.
The most important part of the analysis is devoted to studying the relationships between the Questionnaire data and the language test data. The basic technique used is called latent regression analysis.
The standard setting section provides details on how SurveyLang assessments are related to the CEFR.