A study on construct validity of science items in pisa 2006: the case of Turkey

Abstract

This study was designed for three major goals. The first goal is to investigate construct validity in the Turkish version of the PISA 2006 science units including stimuli (e.g. text, graphs, and pictures) and items in terms of positive and negative entities embedded in them. The second goal is to understand the effect of the negative entities detected in the science units on total test and at item level by achievement scores of 15 year-old students. The third goal that emerged through the study aimed to explore the 15 year-old students’ familiarity with PISA science units in terms of the students’ unique learning experiences. The study consists of three phases. The first phase is the exploration of eight science units including eight stimuli and 25 items. In order to analyze these science units, content analysis was implemented for all of the stimuli and items in the science units. An Item Rating Form was developed and used in order to compose categories of the content analysis from the revisions of the teachers (80 secondary school science teachers in total) working at different regions in Istanbul. Content analysis results showed that there were five main categories defined for the negative entities (content, language, typicality, presentation and structure) and four main categories formed for the positive entities (context, content, science process and composition). The number of the thematic units of the negative entities was more than the thematic units of the positive entities. The second phase of the study based on the results of the content analysis from first phase. For the second phase the revisions on the science units were made for recovery of the negative entities described in the science units. Hence, two tests were present, one of them is PISA-Original Turkish test (PISA-OT test) which includes Turkish science units used in PISA 2006 study and the other is PISA-Revised Turkish (PISA-RT test) which contains revised versions of science units made at the second phase of the present study. At the second phase the effect of the negative entities on the achievement scores of the students examined by using two instruments (PISA-OT and PISA- RT). The results of the second phase showed that there was a statistically significant effect of the negative entities included in the revision process on the total achievement levels of the students. Although the results of analysis at the item level showed that there were significant differences between the two comparison groups for eleven of the items, there was no significant difference for the remaining eleven items. In addition, it is found that the mean scores for the group who answered the PISA- Revised Turkish test were higher than the means of the group who took the PISA-Original Turkish test on each of the 22 items. The third phase is the examination for the answer of the third research question related with the familiarity of students with the stimuli of PISA science units in terms of language, lay-out, school knowledge and daily life experiences. It is found that students tend to be moderately familiar with the PISA stimuli. Based on the results of the three phases, it can be concluded that the aim of selection and formation of the items for the science literacy test is important from the point of construct validity. However, released PISA stimuli and items in Turkish form achieved this aim partially.