A validity study of the SSP-School Inclusion Questionnaire

Abstract

The paper is based on the results of a research that aimed to identify the factors influencing school inclusion among disadvantaged students. The main outcome of the research was the development of the SSP-School Inclusion Questionnaire, intended to: (1) measure school inclusion levels among students; (2) identify students with significantly lower levels of school inclusion, who need to be included in intervention programs; and (3) measure the impact of intervention programs using subjective indicators (beneficiaries' perception of school and teachers, feelings of safety in school, school performance). The questionnaire was completed by 480 vocational school students aged 15-19 from a large developed urban area (Cluj-Napoca) in Romania. The purpose of the research was to test the validity and reliability of the instrument and to propose it to professionals working in education (school psychologists, school counsellors etc.) for efficient data collection and for measuring the impact of educational intervention. [PUBLICATION ABSTRACT]

Headnote

Abstract

The paper is based on the results of a research that aimed to identify the factors influencing school inclusion among disadvantaged students. The main outcome of the research was the development of the SSP-School Inclusion Questionnaire, intended to: (1) measure school inclusion levels among students; (2) identify students with significantly lower levels of school inclusion, who need to be included in intervention programs; and (3) measure the impact of intervention programs using subjective indicators (beneficiaries' perception of school and teachers, feelings of safety in school, school performance). The questionnaire was completed by 480 vocational school students aged 15-19 from a large developed urban area (Cluj-Napoca) in Romania. The purpose of the research was to test the validity and reliability of the instrument and to propose it to professionals working in education (school psychologists, school counsellors etc.) for efficient data collection and for measuring the impact of educational intervention.

Keywords: SSP-School Inclusion Questionnaire, School Inclusion, Disadvantaged students, Vocational high-schools, Cluj- Napoca.

Theoretical background

Integration, as a psychological process of assimilation, can be achieved only in conjunction with social integration and is a process of incorporation of the individual into social systems: family, groups, classes, schools, community, society (OECD, 2007). Integrating a disadvantaged group of children refers to the ability of a group, class, school to assimilate new members who need support for adaptation, integration, socialization (OECD, 2007). On the other hand, inclusive education refers to the elimination of all learning barriers and ensures participation of all those vulnerable to exclusion and marginalization (UNESCO, 2000). It is a strategic approach designated to facilitate successful learning for all children. The first requirement of inclusive education is to decrease all forms of educational exclusion, until elimination. It proposes the provision of access, participation and successful learning for all children. The inclusive schools are open and friendly schools with a flexible curriculum and quality teaching practices that promote continuous assessment and educational partnerships (OECD, 2007).

In what concerns the choice of terminology, the term 'integration' is usually used to describe the process of the assimilation of students with learning difficulties, while the key aspect of 'inclusion' is that students who are at a disadvantage for any reason are not excluded from the mainstream education (Thomas, Walker & Webb, 1998, p.14). Therefore, 'school inclusion' is preferred to 'school integration' (Thomazet, 2009). Although some inclusive education studies may focus exclusively on disabled children (Miles & Singal, 2010), the present study will target the disadvantaged students. The term 'disadvantaged', as it is used in this paper, extends beyond special needs and includes other sources of marginalization, such as ethnicity and poverty (Badescu, 2010; Topping & Maloney, 2005).

Despite the wide literature on school inclusion and participation, the research methods employed in order to measure and assess inclusive practices within schools are mainly qualitative, such as interviews, focus-groups and observations (Kugelmass, 2001; Dyson & Millward, 2000; Hunt et al, 2000; Kratzer, 1997; Deering, 1996; Pickett, 1994). While qualitative methods can facilitate an in-depth exploration of inclusion practices, a survey instrument would allow a more precise measurement of various school inclusion dimensions, the possibility to make comparisons between (groups of ) students, as well as of measuring progress over time.

The purpose of the current research is to develop, test and validate a survey instrument to assess school inclusion. The theoretical framework for instrument development is informed by Skidmore's (1998) functionalist approach of educational institutions, stating that inclusion difficulties arise from deficiencies in the ways in which schools are organized (in terms of personnel, regulations, activities etc.). After reviewing literature (Badescu, 2010; Ainscow, & Miles, 2008; Ainscow, 2007; Booth, & Ainscow, 2007; Ainscow, Booth, & Dyson, 2006; Bowen, & Richman, 2005; Ainscow, Booth, & Dyson, 2004; Booth, & Ainscow, 2002; Lindsay, 2003) and examining several instruments, The School Success Profile-RO2 (SSP) was selected. The SSP is a questionnaire that measures students' school performance and their perception of their school, family, peer group and neighbourhood (Bowen & Richman, 2005; Haragus, Roth & Damean, 2010; Mezei, Damean & Dégi, 2010; Haragus, Damean & Roth, 2009). A short version of this instrument was adapted for measuring dimensions of school inclusion such as positive perceptions of school and teachers, feelings of safety and non-discrimination, satisfactory school attendance, behaviour and performance. The instrument collects data directly from students and it reflects their own perceptions.

The new instrument (SSP-School Inclusion Questionnaire) has been piloted on a sample of vocational high-school students (aged 15-19) from a large urban area (Cluj-Napoca) in Romania. The students in vocational high-schools have significantly lower socio-economic status than other high-school students (Damean & Todea, 2011), thus justifying the sample choice. Dyson, Howes & Roberts (2002) highlighted that the studies on school inclusion tend to be located in schools hich have been identified as inclusive and that direct report of outcome data are rare. The sample was designed to include all vocational high-schools in Cluj-Napoca, regardless of their inclusion policies. The article describes the methodology used for instrument validation.

The SSP-School Inclusion Questionnaire can respond to the following needs: (1) to identify the students who need intervention for school inclusion; (2) to evaluate whether the intervention has been successful; (3) to collect data on intervention efficacy (if completed pre- and postintervention) and to provide an appropriate measurement for program evaluation.

The SSP-School Inclusion Questionnaire is designed for the professionals in the field of education (school psychologists, school social workers etc.) who want to: evaluate students' levels of school inclusion; identify the risk of school drop-out among disadvantaged students in order to include them into intervention programs; measure quantitatively the impact of their intervention programs having thus the opportunity to demonstrate the efficacy of the programs; use the instrument in different contexts and environments (Roma communities, isolated rural communities etc.); make recommendations for the local or regional policy makers.

Method

The objective of the research was to develop and test an instrument to measure the needs, the results and the impact of the intervention programs that aim to improve educational inclusion and are targeted to students who belong to disadvantaged or marginalized communities. Therefore, the focus of this study is instrument assessment.

For the current study several SSP scales3 were used in order to measure school inclusion by taking into consideration 4 key dimensions of school inclusion: perception of school as an educational environment institution, perception of teachers, feelings of safety, and performance. A number of Likert scales subscribes to each of these dimensions, as it follows: school - learning climate and school satisfaction; teachers - teacher support, academic relevancy, and academic rigour; safety - school safety, personal safety in school, and non-victim status; performance - school engagement, trouble avoidance, and grades.

The instrument was tested on a pilot sample of 44 respondents and was further amended (the items or scales that did not prove very reliable have been replaced with more suitable ones or eliminated). The final instrument used for data collection was a 172-item questionnaire that took about 30 minutes to complete (for vocational high-school students). The quantitative data was collected4 during April - June 2011.

The research sample was composed of 480 students from 7 vocational high-schools in Cluj- Napoca. The sample included all vocational high-schools in Cluj-Napoca (a developed urban area from Transylvania), but the number of respondents in each school varied according to respondents' presence in the classroom and availability. Table 1 displays the demographic characteristics of the sample. Although data on parents' educational and occupational status was collected, the results are not presented due to the high non-response rate (around 50%). Respondents' reports regarding their income were not very conclusive, as over one third rated their income as very high, whilst the poverty rate is highest among vocational schools. However, the answer choices of the 5-step scale measuring the income level were not appropriately formulated (1 - We do not have enough money; 2 - We hardly have enough money; 3 - We have just about enough money to not need to borrow; 4 - We manage to buy some more expensive things; 5 - We manage to have everything we need). Respondents who receive a form of social aid often answered 'We manage to have everything we need', which may be true, but does not reflect their income level accurately. Therefore, the answer choices will need to be rephrased into more precise ones. Also, data on ethnicity must be treated with caution, as the percentage of observed Roma students was much higher than the percentage of self-reported Roma students, meaning that many of them declared a different ethnicity (Romanian or Hungarian).

For instrument reliability and validity assessment 3 different methods were used: (1) Cronbach's alpha; (2) bivariate Pearson correlations; and (3) Principal Component Analysis.

Results

Descriptive analyses for each school inclusion scale are presented in Table 2. All scales report good internal consistency (with Cronbach's alpha values between 0.7 and 0.9).

The scales were positively coded, so that minimum values indicate a negative attitude or perception, and maximum values indicate a positive attitude or perception. The mean scores indicate an overall positive tendency. However, all scales, except for Academic relevancy and Grades, are negatively skewed (left-skewed), meaning that most values (including the median) concentrate on the right of the mean, with extreme values to the left. Therefore, even though most respondents report rather positive attitudes toward the school environment, the sample contains respondents who scored very low on most school inclusion scales, suggesting a risk situation and a need for intervention5

Next, bivariate Pearson correlations were run between all scales (Table 3) as well as between all items of each scale. Since the 4 dimensions of the instrument are inter-related by default, significant positive correlations were expected between scales. It is worth noting here how respondents' school performance relates to their perception of their school, of their teachers, and of school safety.

Respondents who have a higher school satisfaction and perceive a more positive learning climate are also more engaged in school. The same applies for students who have a positive perception of their teachers. No significant relation was found between school safety and school engagement. School safety is strongly related to respondents' behaviour in school (trouble avoidance), and so are teachers' academic rigour and teacher support. Higher grades do not correlate with feeling safe in school, but with higher teacher support and higher academic rigour. Also, the items of each scale were positively inter-correlated, indicating that they all measure the same factor.

In order to validate the instrument, principal component analysis was used. All items in each scale were expected to load satisfactory on one underlying factor. The extraction method used was Principal Component Analysis and the rotation method used was Varimax with Kaiser Normalization. For easier reading of the matrices, values under .40 have been suppressed (see Appendix).

For each indicator, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is displayed. The KMO must have values above 0.50 for satisfactory factor analysis to proceed. The KMO values on all scales show that the sample size is adequate.

First, the components were extracted based on the Eigenvalue (criteria: Eigenvalue > 1). Following this procedure, in 5 scales more than 1 component has been extracted (Eigenvalue > 1): School satisfaction (2 components), Academic rigour (2 components), School safety (2 components), Non-victim status (3 components), and Trouble avoidance (2 components).

The Eigenvalue for components 2 and 3 was very close to 1, suggesting that these components were rather weak. Therefore, a fixed number of components (1) was extracted on all scales, forcing the items to load on the component with the highest Eigenvalue. The items of each indicator loaded satisfactory on one component, indicating the scales are valid and measure what they were designed to measure.

Conclusions

The outcome of the present study was the development of an efficient and reliable instrument to measure the school inclusion of disadvantaged students with the purpose of identifying the ones who need to benefit from intervention programs. This instrument is offered to school professionals and to anyone working in the field of education. The instrument is useful not only in terms of identifying the school inclusion problems or identifying the respondents who may need to be included in intervention programs, but also for drafting evidence based recommendations for school policies. Although validity tests indicated a good internal consistency of the scales, the use of additional objective measurements of school inclusion is encouraged.

A limit of the study is that the survey conducted in schools provided data only for the students who were present and willing to complete the questionnaire. It is possible that the mean scores would have been lower if the students who are very often absent had completed the questionnaire. In addition, the measurement used for poverty (self-reported income size) was not reliable. For future studies, the use of consumption is recommended as the basis for calculating financial indicators (Badescu, 2010, p.9).

In the course of this study, various areas where further research was needed were identified: the development of a methodology for evaluating how inclusive schools and teachers are; the use of other theoretical perspectives in the study of school inclusion; the development of recommendations for school policies.

Acknowledgement

This work was possible with the financial support of the Sectoral Operational Programme for Human Resources Development 2007-2013, co-financed by the European Social Fund, under the project number POSDRU 89/1.5/S/60189 with the title 'Postdoctoral Programs for Sustainable Development in a Knowledge Based Society'

Footnote

2 The School Success Profile-RO was translated and adapted as part of the research project PN-II 91063/18.10.2007-2010 financed by CNMP, coordinated by dr. Maria Roth, Babes-Bolyai University (www.successcolar.ro), after The School Success Profile, created by G. L. Bowen and J. M. Richman, Jordan Institute for Families, University of North Carolina at Chapel Hill (www.schoolsuccessprofile.org).

3 The permission for using any of the the School Success Profile-RO scales must be requested from the Centre for Evaluating the Socio- Educational Profiles (CESEP), contact: [email protected].

4 Data was collected in partnership with the Romanian Foundation for Children, Community and Family (FRCCF) by Julia Todea Várhegyi and Diana Damean as part of the Strengthening the Evidence Based Practice of educational CSOs' Initiative funded by the Education Support Program - Open Society Foundation.

5 Permission for instrument use and details regarding score interpretation can be requested from the Centre for Evaluating Socio-Educational Profiles (CESEP), [email protected].

References

References

1. Ainscow, M. (2007). Taking an inclusive turn. Journal of Research in Special Educational Needs, 7(1), pp.3-7. http://dx.doi.org/10.1111/j.1471-3802.2007.00075.x

2. Ainscow, M., Booth, T., & Dyson, A. (2006). Improving schools, developing inclusion. London: Routledge.

3. Ainscow, M., Booth, T., & Dyson, A. (2004). Understanding and developing inclusive practices in schools: a collaborative action research network. International Journal of Inclusive Education, 8(2), pp.125-140. http://dx.doi.org/10.1080/1360311032000158015

4. Ainscow, M., & Miles, S. (2008). Making Education for All Inclusive: Where Next? Prospects, 38(1), pp.15-34. http://dx.doi.org/10.1007/s11125-008-9055-0

5. Badescu, G. (2010). Social inclusion through education: a literature review. ESP Working Paper Series, No.8.

6. Booth, T., & Ainscow, M. (2007). Breaking down the barriers: the index for inclusion. Bristol: Centre for Studies on Inclusive Education.

7. Booth, T., & Ainscow, M. (2002). Index for inclusion: Developing learning and participation in schools. Bristol: Centre for Studies on Inclusive Education.

8. Bowen, G. L., & Richman, J. M. (2005). The School Success Profile. University of North Carolina at Chapel Hill: Jordan Institute for Families.

9. Damean, D., & Todea, J. (2011). Ghid metodologic pentru implementarea proiectelor de integrare scolara a elevilor dezavantajati. Cluj-Napoca: FRCCF.

10. Deering, P. (1996). An ethnographic study of norms of inclusion and cooperation in a multiethnic middle school. Urban Review 28(1), pp.21-40. http://dx.doi.org/10.1007/BF02354376

11. Dyson, A., Millward, A. (2000). Schools and special needs: issues of innovation and inclusion. London: Paul Chapman.

12. Dyson, A., Howes, A., & Roberts, B. (2002). A Systematic Review of the Effectiveness of schoollevel actions for promoting participation by all students. Retrieved March 23, 2011, from http:// eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=juIcqkP5Q8U%3d&tabid=276&mid=1096

13. Haragus, T.P., Damean, D., & Roth, M. (2009). Proprietati psihometrice ale unui nou instrument de evaluare a performantelor scolare: Profilul Succesului Scolar. In M. Roth, D. Damean & M. Iovu (Eds.), Succesul scolar la intersectia factorilor sociali (pp.23-40). Cluj- Napoca: Presa Universitara Clujeana.

14. Haragus, T.P., Roth, M., & Damean, D. (2010). The Measurement of the Social Dimensions of School Success - A Validity Study of the Romanian Version of the School Success Profile. Studia Universitatis Babes-Bolyai Sociologia, 1/2010, pp.31-56.

15. Haragus, T.P., Roth, M., & Mezei, E. (2010). Validitatea si fidelitatea Profilului Succesului Scolar În România. Cluj-Napoca: Presa Universitara Clujeana.

16. Hunt, P., Hirose-Hatae, A., Doering, K., Karasoff, P., Goetz, L. (2000). 'Community' is what I think everyone is talking about. Remedial & Special Education, 21(5), pp.305-317. http://dx.doi.org/10.1177/074193250002100507

17. Kratzer, C. (1997). Community and diversity in an urban school: co-existence or conflict? Report by the Graduate School of Education & Information Studies, University of California, Los Angeles.

18. Kugelmass, J. (2001). Collaboration and compromise in creating and sustaining an inclusive school. International Journal of Inclusive Education, 5(1), pp.47-65. http://dx.doi.org/10.1080/13603110121498

19. Lindsay, G. (2003). Inclusive education: a critical perspective. British Journal of Special Education, 30(1), pp.3-12. http://dx.doi.org/10.1111/1467-8527.00275

20. Mezei, E., Damean, D., & Dégi, C. (2010). The School Success Profile: confirmatory factor analysis on a representative Romanian sample. In D. Damean, M. Roth, C. Dégi & L. Vaetisi (Eds.), The Social Ecology of School Success: Implications for Policy and Practice (pp.21-66). Cluj-Napoca: Cluj University Press.

21. Miles, S., & Singal, N. (2010). The education for all and inclusive education debate: conflict, contradiction or opportunity? International Journal of Inclusive Education, 14(1), pp.1-15. http://dx.doi.org/10.1080/13603110802265125

22. OECD (2007). Education Policies for Students at Risk and those with Disabilities in South Eastern Europe: Bosnia-Herzegovina, Bulgaria, Croatia, Kosovo, FYR of Macedonia, Moldova, Montenegro, Romania and Serbia. Paris: OECD.

23. Pickett, R. (1994). The relationship between school structure and culture and student views of diversity and inclusive education: a comparative case study of two middle schools. Unpublished PhD thesis, University of Wisconsin.

24. Skidmore, D. (1998). Continuities and developments in research into the education of pupils with learning difficulties. British Journal of Educational Studies, 47(1), pp.3-16. http://dx.doi.org/10.1111/1467-8527.00099

25. Thomas, G., Walker, D., & Webb, J. (1998). The making of the inclusive school. London: Routledge.

26. Thomazet, S. (2009). From integration to inclusive education: does changing the terms improve practice? International Journal of Inclusive Education, 13(6), pp.553-563. http://dx.doi.org/10.1080/13603110801923476

27. Topping, K., & Maloney, S. (Eds.). (2005). The Routledge Falmer Reader in Inclusive Education, Abingdon: Routledge Falmer.

28. UNESCO (2000). Education for All 2000 Assessment. France: UNESCO.

Appendix

(ProQuest: Appendix omitted.)