Development of a chemistry critical thinking test: initial reliability and validity studies

Abstract

Background There are a myriad of commercially available tools which test critical thinking. These tests are designed for recruitment purposes and the context of these tests are often generic or abstract. A more accurate representation of student critical thinking may be obtained when students attach significance to the context of the test. Aims The aim of this investigation is to develop an instrument which can evaluate a chemistry student’s critical thinking ability at any stage of their study. This paper describes the development of a pilot test and reliability and validity studies associated with the test. Design and methods A chemistry critical thinking test was developed with thirty multiple choice items designed to assess ‘making assumptions’, ‘assessing arguments’, ‘making hypotheses’, ‘testing hypotheses’ and ‘drawing conclusions’. This test was administered to 1200 first year Monash University undergraduate chemistry students at the beginning of semester 1, 2016. 746 students completed the test which was analysed for internal reliability by using Cronbach’s alpha, t-tests and a correlation matrix of all test items. In conjunction with this, qualitative data has been collected from an academic focus group with respect to the content and construct validity of the test. Results A bell curve distribution of student test scores was obtained. Each sub-section of the test, for example ‘making assumptions’, reflected a similar distribution of scores. The internal reliability studies are still in progress. Students from this cohort have been invited to take part in a test-retest reliability study, a construct validity study against commercially available tools, and provide demographic data. Students will also participate in focus groups to collect qualitative data to further assess the content and construct validity of the test. These finding will be presented as part of this paper. Qualitative data from the academic focus group suggests that the some test items are dependent on chemistry knowledge. However, the focus group agree with how the constructs are assessed, and the relevance of the thought processes required to obtain the appropriate answer. Conclusions A chemistry critical thinking test has been developed and piloted. 746 first year participants obtained a bell curve distribution of scores and internal reliability data is pending. Qualitative data suggests the pilot test is dependent on chemistry knowledge in some areas. However the constructs of critical thinking are in alignment with the intention of the test.