Setting the Standard
An interview with Neil Jones, ESLC Director
What’s been happening with SurveyLang in the last few months?
It’s been a long but rewarding process from start to finish. Last year we successfully completed the SurveyLang Field Trial in all the participating countries. This was a really worthwhile exercise as it allowed us to iron out any problems before the Main Study this year.
Did the Main Study throw up any surprises or challenges?
There certainly weren’t any unpleasant surprises. The Field Trial acted as a ‘first try’ which put us in good stead for the Main Study. The feedback we’ve received so far has been really positive which is great news.
This September, you reached quite a milestone with SurveyLang – what happened?
We held a big event in Cambridge which brought together experts from all over the world to set the standards for the Main Study. We’ve put a tremendous amount of work into getting what are good results, but we now have to interpret these in line with the Common European Framework of Reference for Languages (CEFR). It’s quite significant that the terms of reference of the study state that we have to report in terms of the CEFR as it shows how much prominence the framework is gaining in Europe and beyond. The work of the SurveyLang team will raise this profile further.
What happened at the standard setting exercise?
We brought in experts from as many participating countries around Europe as possible. We felt it was important to have broad participation in a study of this kind. For this reason, we invited prominent names connected with the CEFR, teachers and other stakeholders with an interest in the CEFR.
So what did the Standard Setting event involve?
The event lasted a week and we started with a series of inductions which covered familiarisation, test construction and the different methods of standard setting. We then got down to the nitty-gritty of setting the actual standards for the study which was a complex task.
And what did that involve?
We carried out individual sessions which focused on each of the tests. Each participant looked at the task, made a judgement on the difficulty and then came up with a cut off for each level. What they were effectively doing was setting pass rates which is important for any type of assessment. We rigorously documented opinions and then built in revision and replication to ensure consistency and reliability of results.
What was the role of the SurveyLang team at the Cambridge event?
Our role was to give guidance. We set the day up to work independently and in groups defined by language. They went through a set process where they analysed difficulty, reported back to the wider group and then revised if necessary. The challenge we have is that different countries have different local understandings about what the CEFR levels are and this is something we have to take into account when setting standards.
Why is standard setting so important?
Standard Setting determines the results we report, so the data we collect and our approach to interpreting it are what ensure SurveyLang results are valid, credible and of high quality.
When the results of the full study are published in 2012, who do you think the survey will impact the most?
The project will impact policy makers and strengthen the rationale for focusing on communication skills for language. It will also be good for the multilingualism programme which is gaining more prominence in Europe.
What’s next for the SurveyLang team?
There is still quite a lot of work to be done. After the standard setting event there will be some careful analysis and interpretation. Shortly we’ll begin applying results to the database and then we can generate the final report. It’s an exciting time.