Psychometric properties of the Persian version of the Ambulatory Care Learning Educational Environment Measure (ACLEEM) questionnaire, Shiraz, Iran

Abstract

Purpose: Evaluation is the main component in design and implementation of educational activities and rapid growth of educational institution programs. Outpatient medical education and clinical training environment is one of the most important parts of training of medical residents. This study aimed to determine the validity and reliability of the Persian version of Ambulatory Care Learning Educational Environment Measure (ACLEEM) questionnaire, as an instrument for assessment of educational environments in residency medical clinics.

Materials and methods: This study was performed on 180 residents in Shiraz University of Medical Sciences, Shiraz, Iran, in 2014–2015. The questionnaire designers’ electronic permission (by email) and the residents’ verbal consent were obtained before distributing the questionnaires. The study data were gathered using ACLEEM questionnaire developed by Arnoldo Riquelme in 2013. The data were analyzed using the SPSS statistical software, version 14, and MedCalc^® software. Then, the construct validity, including convergent and discriminant validities, of the Persian version of ACLEEM questionnaire was assessed. Its internal consistency was also checked by Cronbach’s alpha coefficient.

Results: Five team members who were experts in medical education were consulted to test the cultural adaptation, linguistic equivalency, and content validity of the Persian version of the questionnaire. Content validity indexes were >0.9 in all items. In factor analysis of the instrument, the ­Kaiser–Meyer–Olkin index was 0.928 and Barlett’s sphericity test yielded the following results: X ²=6,717.551, df =1,225, and P ≤0.001. Besides, Cronbach’s alpha coefficient of ACLEEM questionnaire was 0.964. Cronbach’s alpha coefficients were also >0.80 in all the three domains of the questionnaire. Overall, the Persian version of ACLEEM showed excellent convergent validity and acceptable discriminant validity, except for the clinical training domain.

Conclusion: According to the results, the Persian version of ACLEEM questionnaire was a valid and reliable instrument for Iranian residents to assess specialized clinics and residency ambulatory settings.

Introduction

As one of the key contributors to educational development, educational environment is definitely one of the most valuable assets of an academic setting.¹ Based on its quality, this environment can either support students or impede them from obtaining the desirable outcome of education, which is learning.² This impact is probably more pronounced in medical education settings, where the environment that engulfs students plays an important role in their skill acquirement and, consequently, patient care. Therefore, to guarantee a high-quality medical education, careful and continuous observation, assessment, and revision of the existing educational environment are mandated.^1,3

Although many use the words teaching and learning environments synonymously and interchangeably,⁴ learning environment applies mostly to the physical locations, contexts, and cultures in which students learn. In other words, it usually refers to the quality of individuals' interactions with one another as well as the educational settings designed to facilitate learning.⁵ These also include policies and governance structures. Overall, a standard learning and teaching environment provides students with sufficient opportunities to acquire theoretical and practical skills as well as social responsibility that in turn prepares them for their future professional lives.^5–8

Nowadays, the main environment influencing medical students is the clinical learning environment, which helps them merge their theoretical training with clinical skills.^9,10 Since the learning environment can only be assessed through its outcomes, the learning process, as the most important outcome, seems to be the perfect choice for its evaluation.^11,12 This process is influenced by a wide variety of factors. Therefore, all the effective factors must be considered for its complete evaluation.¹³ These factors themselves can be categorized into internal and external factors. The first refers to mental qualities, objective and intrinsic motivation, and sentiment, while the second includes educational facilities, methods, and external stimulators of the educational environment.^6,11

In the recent years, several studies have been conducted to develop and validate new instruments for assessment of learning environments in medical education, such as Dundee Ready Education Environment Measure (DREEM), Postgraduate Hospital Educational Environment Measure (PHEEM), Clinical Learning Environment Scale, and Dutch Residency Education Environments Test questionnaire.^14–21 Review of the literature indicated that all these tools were more suitable for inpatient medical learning.^{16,18,21–24}

Ambulatory or outpatient care is the medical care given to the patients who do not need hospital admission. In addition to diagnosis, follow-up, consultation, interventional treatments, and rehabilitation services, the ambulatory care may also include medical procedures.^25–28 As suggested by World Health Organization in 2008, the ambulatory care has been considered as an important part of medical residency programs.²⁹ Thus, more attention has been paid to outpatient medical education and its learning environment in the recent years. Ambulatory Care Learning Education Environment Measure (ACLEEM) questionnaire was designed by Riquelme et al²³ in 2013 to assess the educational environment of residency medical clinics in Pontifical Catholic University of Chile in Latin America.

Considering the importance of utilization of new and improved evaluation methods, such as ACLEEM, in outpatient medical education enhancement and lack of studies based on this method in Iran, an attempt was made to adapt a valid and reliable Persian translation of this questionnaire in Shiraz University of Medical Sciences (SUMS).

Materials and methods

Design and setting of the study

This cross-sectional study was conducted on Persian-speaking medical residents of SUMS, Shiraz, Iran, in 2014–2015.

Sample size and participants

Several researchers have provided conflicting viewpoints and rules of thumb for sampling. Most of the researchers have recommended that the sample size ranging from 100 to 400 is adequate for testing the construct validity. On the other hand, others have suggested five or ten times the number of the items.³⁰ Therefore, because of the limited number of residents in SUMS, all of them were invited to participate in the study. The inclusion criteria of the study were being Iranian, speaking Persian, and having experience of teaching and learning in the ambulatory setting in different specialty fields, including psychiatry, pediatrics, surgery, orthopedics, dermatology, otorhinolaryngology, ophthalmology, neurology, internal medicine, obstetrics and gynecology, anesthesia, pathology, traditional Persian medicine, cardiology, and rehabilitation and physical medicine, in their residency period at least for one course (up to 5 months). Overall, 180 residents aged 25–46 years participated in the study.

Instrument

ACLEEM questionnaire was administered by the researcher and members of the Education Development Office (EDO) in the educational groups with the support of the Education Developmental Center, Quality Improvement in Clinical Education Research Center, special departments, and heads of each educational group of SUMS.

This instrument contains 50 questions scored based on a five-point Likert scale ranging from 4 (strongly agree) to 0 (strongly disagree). The 50 items were embedded in three domains, including clinical teaching (items 1–16), clinical training (items 17–38), and support (items 39–50). Because items 24 and 27 contained negative statements, we had to reverse code the scores for these questions. The maximum score of the questionnaire was 200.²³

Development of the Persian version of ACLEEM questionnaire

After taking consent from Arnold Riquelme, the author of the original version of ACLEEM questionnaire, forward–backward method was used to translate this questionnaire from English into Persian. First, the questionnaire was translated into Persian by two experts in English language at EDO of SUMS. Then, backward translation was done by another linguist. After that, the final English version was compared to the original version of the questionnaire and confirmed by an expert linguist. After all, a professional bilingual translator made the final Persian version of the questionnaire. The last confirmed version of ACLEEM questionnaire was used for evaluation of the psychometric properties.

Data collection

The obtained questionnaire was administered in printed format. Before distributing the questionnaires, the objectives of the project were explained to the residents. In addition, the residents were ascertained that they were free to leave the study at any time. It should be noted that each questionnaire took ~10–15 minutes to be filled.

Research ethics

Arnoldo Riquelme's license to use this tool was received by email. This study was approved by the ethics committee of SUMS. The main idea, purpose, and benefits of the study were described for the medical residents. Then, the participants announced their verbal consent to take part in the study on a voluntary basis. Additionally, all the residents filled the questionnaires anonymously.

Reliability and validity assessment and statistical analysis

In order to assess and confirm the cultural adaptation, linguistic equivalency, and content validity of the final Persian version of ACLEEM questionnaire, the instrument was given to expert team members involving five faculty members of Education Developmental Center who were proficient in medical education. Moreover, the reliability of the ACLEEM domains was tested using Cronbach's alpha coefficient. Internal consistency was considered to be acceptable if the coefficient was at least 0.7.³¹ Furthermore, appropriate statistical analyses, including minimum and maximum scores, mean ± SD, corrected item-total correlation, items' scores, and Cronbach's alpha with and without any specific items, were applied for determination of the properties of the items.

Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and Bartlett's test of sphericity were used to specify whether factor analysis could be useful with our data due to proportion of variance in the variables.

Moreover, exploratory factor analysis with Varimax rotation was utilized to determine the construct validity of the Persian version of ACLEEM questionnaire. Convergent and discriminant validities, the two aspects of construct validity, were examined using Spearman's correlation. Accordingly, values ≥0.40 represented appropriate convergent validity. Besides, discriminant validity was supported whenever the correlation between an item and its hypothesized domain was higher than its correlation with the other domains. Convergent and discriminant validities were computed using the following formulas:³¹

(1)

(2)

All the statistical analyses were performed using the SPSS statistical software, version 14 and MedCalc^® statistical software.

Results

In total, 174 of the 180 residents (96.66%) returned the questionnaires. Six questionnaires were excluded (3.33%) since they were incomplete. The participants included 98 (56.3%) males and 76 (43.7%) females. The mean ± SD of the residents' age was 31.89±4.08 years. Besides, the mean ± SD of the male and female residents' age was 32.43±4.45 years and 31.04±3.27 years, respectively, and the difference was not statistically significant (P=0.06). The characteristics of the residents have been presented in Table 1.

Table 1 The characteristics of the residents who completed ACLEEM questionnaire Abbreviations: ACLEEM, Ambulatory Care Learning Educational Environment Measure; SD, standard deviation.

Cultural adaptation, linguistic equivalency, and content validity

Cultural adaptation, linguistic equivalency, and content validity of the Persian version of ACLEEM questionnaire were confirmed by all the five faculty members who were expert in medical education in EDO. The content validity indexes of all items in the scale were >0.90, and none of the questions were removed.

Reliability

The reliability of the 50-item questionnaire was confirmed with Cronbach's alpha of 0.964. All the three domains of the Persian version of ACLEEM questionnaire also showed sufficient reliability (Cronbach's alpha coefficients >0.7). Minimum and maximum scores, mean ± SD scores, and Cronbach's alpha coefficients of each domain have been shown in Table 2.

Table 2 Minimum and maximum scores, mean scores, and Cronbach's alpha of the dimensions of the Persian version of ACLEEM questionnaire Notes: aThe number in parentheses indicates the percentage of the minimum score. bThe number in parentheses indicates the percentage of the maximum score. cSum of the scores in each dimensions/number of questions of each dimensions. Abbreviations: ACLEEM, Ambulatory Care Learning Educational Environment Measure; SD, standard deviation.

Construct validity

Using factor analysis for assessment of sample adequacy, KMO index was found to be 0.928 and Bartlett's test yielded the following results: X ²=6,717.551, df =1,225, and P≤0.001. All the initial communalities were equal to 1, all the extracted communalities were ≥0.4, and only two questions (24 and 27) had extracted communalities <0.3. Moreover, all the questions of the clinical teaching domain, 63.63% of the questions of the clinical training domain, and 75% of those of the support domain showed strong correlation coefficients (>0.4) and had clear loading for the main domains. The results of the factor analysis with Varimax rotation used to test the construct validity of the Persian version of ACLEEM questionnaire have been presented in the three last columns of Table 3.

Table 3 Items analysis and factor loading (Varimax rotation) of the three-factor solutions for the Persian version of ACLEEM questionnaire Notes: Bold values show items belonging to subscales in the original version of ACLEEM questionnaire. *These items have coefficient less than 0.40 in own subscales. Abbreviations: ACLEEM, Ambulatory Care Learning Educational Environment Measure; SD, standard deviation.

Our findings showed that the scaling success rate for convergent validity was 100% for teaching and support domains and 95% for the training domain. Additionally, the success rates for discriminant validity were 87.5% (28/32), 54.54% (24/44), and 79.16% (19/24) for teaching, training, and support domains, respectively (Table 4).

Table 4 Item scaling test: convergent and discriminant validities of the Persian version of ACLEEM questionnaire Notes: aNumber of correlations between the items and the hypothesized scale corrected for overlap ≥0.4/total number of convergent validity tests. bNumber of convergent correlations significantly higher than discriminant correlations/total number of correlations. Abbreviation: ACLEEM, Ambulatory Care Learning Educational Environment Measure.

Items analysis

The mean score of the total instrument was 2.34±0.64. The subscale mean scores for clinical teaching, clinical training, and support were 2.48±0.77, 2.40±0.65, and 2.01±0.75, respectively (Table 2).

The lowest mean score of the items was 1.20 (for item 48: "there are adequate bathroom facilities in the outpatient clinic [OPC]"), while the highest mean score was 2.96 (for item 1: "working in the OPC enables me to develop my problem solving skill"). Overall, none of the mean scores of the items were >2.96, and the mean scores of the seven items (23, 24, 27, 46, 47, 48, and 49) were <2. The results of items analysis indicated that removing or retaining a special item did not change the Cronbach's alpha. The items analysis for the Persian version of ACLEEM questionnaire has been presented in Table 3.

Discussion

Learning environment is one of the curtail factors, which has a significant impact on medical students' success and satisfaction.^9,32 Therefore, educational systems need a certain instrument to assess the quality of the medical environment given for this goal.³³ The ambulatory medicine program is suggested as an important section of the educational course for development of medical residents' clinical skills by providing focused teaching with increased exposure to patients with a wide range of chronic medical conditions in a dedicated learning environment.³⁴ ACLEEM questionnaire was the first specialized instrument developed to evaluate various aspects of the learning environment in outpatient clinics. It is also a unique instrument in this respect.²³ Therefore, this study was designed to assay the validity and reliability of the Persian version of ACLEEM questionnaire to determine whether this version is appropriate for measurement of the residency medical ambulatory environments in Iran.

Reliability and repeatability discuss the casual variability associated with measurements.³⁵ In this study, the Cronbach's alpha coefficient of the instrument was 0.964, which is in line with the results of the study by Riquelme et al³⁶ in 2015. Additionally, Cronbach's alpha coefficients were >0.70 for all the three subscales. Thus, the reliability of the Persian version of ACLEEM questionnaire was confirmed.

Content validity is related to the proportion to which the items are sensible and reflect the conscious domain of interest.³⁰ None of the items in the Persian version of ACLEEM instrument were removed due to confirmation of content validity by an expert five-member team with excellent content validity indexes.

Construct validity illustrates the ability of the scale to measure what it is assumed to measure. Convergent and discriminant validities are the two important aspects of construct validity.³⁰ The acceptable values of KMO and Bartlett's sphericity test indicated that our study sample was sufficient and factor analysis could be done for evaluating the construct validity. In this study, scaling success of convergent validity was excellent for all the three dimensions, as shown in Table 4. Besides, discriminant validity was >70% in clinical teaching and support domains, but not in the clinical training domain. This implies that the questions of clinical teaching and support domains, but not those of the clinical training domain, were well understood by the residents to assess the relevant aspects. Up to now, no studies have assessed the convergent and discriminant validities of this questionnaire to compare the results.

In our study, the mean score of total questionnaire and all three subscales were >2, which reveals a more positive than negative environment in all dimensions. Clinical teaching dimension had the highest mean score among the others. The mean scores of seven items (23, 24, 27, 46, 47, 48, and 49) were <2, the expected mean score of each item. This can be attributed to inadequate supplies and instruments at the clinics of SUMS as well as to Iranian culture. For instance, "opportunity for visiting and following up the patients" in items 23 and 24, "adequacy of the number of supervisors in clinics" in item 27, "computer access" in item 46, and "provide lockers to keep the residents' belongings safe" in item 47 were the components focused on instruments and supplies, and "adequate bathroom facilities" in item 48 might refer to Iranian culture and religious practices related to bathing in private homes rather than clinics, in addition to instrument inadequacy.

The results of factor analysis revealed that the majority of the questions were included in the questionnaire.³⁰ Riquelme et al²³ divided the original ACLEEM questionnaire into three domains, including clinical teaching, clinical training, and support. These three domains were also used in factor analysis in this study. However, the items were distributed differently in the domains of the Persian questionnaire. The results of factor analysis also indicated that the Persian version of ACLEEM questionnaire did not exactly evaluate the three underlying constructs reported in the original version.²³ Accordingly, all the 16 items (100%) of the clinical teaching domain were clearly loaded in the expected domain. All the items of the support domain, except for items 43 "I feel part of the team in the OPC" and 44 "I receive support from other OPC residents when I need it", were also clearly loaded in the appropriate territory in factor analysis. Items 43 and 44 were loaded for the clinical training domain in factor analysis with Varimax rotation. However, eight of the 22 items (36%) of the clinical training domain (21, 22, 23, 24, 25, 27, 28, and 31) had no clear loading for this domain. On the other hand, ten of the 22 items (45.45%) of the clinical training domain (17, 19, 20, 22, 23, 25, 26, 28, 29, and 31) had higher coefficients in the clinical teaching domain in factor analysis. This may imply that the study residents had a different perception and attitude toward some items. In other words, the Persian version of this questionnaire could have a different structure. Thus, another factor analysis could be performed for finding more domains in the Persian version of this instrument. The results also indicated that items 24 and 27 had a low coefficient in all the domains in factor analysis with Varimax rotation. Therefore, these items had a little capability for assessment of the medical ambulatory setting in Iranian residents.

Jalili et al¹⁸ conducted a study in 2014 for validating the Persian version of PHEEM questionnaire for measurement of the educational environment in academic emergency department. The results revealed that this instrument had a good reliability with Cronbach's alpha of 0.86. However, the Persian version of PHEEM had a different structure in factor analysis. The results of this study are also approximately similar to those obtained by Jalili et al. In another study performed by Khan et al³⁷ in Pakistan in 2011, the internal consistency of DREEM instrument was confirmed with Cronbach's alpha of 0.91. However, a large mismatch in structure was observed compared to the original version.³⁷ This similarity regarding structural differences in ACLEEM, PHEEM, and DREEM in different nationalities could be associated with the participants' cultural diversity.

Our study had some limitations, the first of which being the small sample size, which was due to the small number of individuals passing the residency examination for SUMS each year as well as to the diversity at the time of beginning the residency training in ambulatory care setting in different specialty fields. Moreover, in order to have a more uniform sample, the residents had to have at least 5 months of experience upon entrance and all the residents who met this inclusion criterion were entered into the study. Another study limitation was that the residents filled the questionnaires anonymously. Therefore, it was not possible to assess all the residents for the second time, and consequently, the temporal stability of the instrument could not be assayed. Third, because ACLEEM questionnaire is the only available instrument for evaluating the ambulatory care learning educational environment,²³ it is not possible to test the criterion validity to assess whether it has empirical association with external criteria, such as other established instruments. Finally, our participants were distributed in different levels of residency course from first to fourth years. Therefore, the results might have been affected by the heterogeneity in their medical experiences and skills.

Overall, our study findings revealed excellent internal consistency, excellent convergent validity, and good discriminant validity of the Persian version of ACLEEM questionnaire for evaluation of the residency outpatient clinics, environments. Therefore, this valid and reliable instrument can be useful to improve the medical ambulatory settings, eventually leading to progress of teaching and training practices. This instrument can also help managers monitor the performance of the educational systems, faculty members of medical schools, and residents. Furthermore, this tool lets educational systems identify the weak points of medical ambulatory settings and improve processing of the goals. Future studies are recommended to assess the quality of clinical teaching and training and support the residents of SUMS to detect and resolve the gaps in learning programs and medical ambulatory settings.

Conclusion

Based on our findings, the Persian version of ACLEEM questionnaire was a multidimensional, simple, practical, valid, and reliable instrument for assessing the environment of outpatient clinics of medical residents. It also provided a useful index of the priorities for curriculum improvement. Yet, ACLEEM questionnaire had a different structure from the original version and measured more than three aspects of outpatient clinics for Persian residents.

Acknowledgments

This article was extracted from the thesis written by Mohammad Mahdi Parvizi as the partial fulfilment of the requirements for obtaining Master's degree in Medical Education in Shiraz University of Medical Sciences (proposal number 93-6964). Hereby, the authors wish to thank the residents who participated in this study and all the individuals who helped in performing the project. They are also grateful for Ms A. Keivanshekouh at the Research Improvement Center of Shiraz University of Medical Sciences for improving the use of English in the manuscript.

Author contributions

MMP, MA, MRD: collected the data, wrote the manuscript, conceived, designed, and did statistical analysis, and revised the manuscript. MA, PJ: conceived, designed, and did statistical analysis and revised the manuscript. ZP: collected the data and wrote the manuscript.

Disclosure

The authors declare that they have no conflicts of interest in this work.

References

1. Fitzpatrick J, Worthen B, Sanders J. Program Evaluation: Alternative Approaches and Practical Guidelines. 3 ed. Boston: Pearson: Allyn and Bacon; 2004.

2. Richmond M, Canavan C, Holtman MC, Katsufrakis PJ. Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment. J Grad Med Educ. 2011;3(4):511–516.

3. Hostler SL, Gressard RP. Perceptions of the gender fairness of the medical education environment. J Am Med Womens Assoc. 1993;48(2):51–54.

4. Franzone JM, Kennedy BC, Merritt H, Casey JT, Austin MC, Daskivich TJ. Progressive independence in clinical training: perspectives of a national, multispecialty panel of residents and fellows. J Grad Med Educ. 2015;7(4):700–704.

5. Shahidi F, Saqeb MM, Amini M, Avand A, Dowlakham HR. Qualitative evaluation of general practitioner training program as viewed by graduates from Shiraz, Fasa and Jahrom Medical Universities. J Adv Med Educ Prof. 2015;3(3):142.

6. Divaris K, Barlow PJ, Chendea SA, et al. The academic environment: the students’ perspective. Eur J Dent Educ. 2008;12(suppl 1):120–130.

7. Giddon DB. Why dentists should be called oral physicians now. J Dent Educ. 2006;70(2):111–114.

8. Henzi D, Davis E, Jasinevicius R, Hendricson W, Cintron L, Isaacs M. Appraisal of the dental school learning environment: the students’ view. J Dent Educ. 2005;69(10):1137–1147.

9. Hasani H, Bahrami M, Malekpour A, et al. Evaluation of teaching methods in mass CPCR training in different groups of the society, an observational study. Medicine. 2015;94(21).

10. Yamani N, Liaghatdar M, Changiz T, Adibi P. How do medical students learn professionalism during clinical education? A qualitative study of faculty members’ and interns’ experiences. Iran J Med Educ. 2010;9(4):382–394.

11. Goldie J. AMEE education guide no. 29: evaluating educational programmes. Med Teach. 2006;28(3):210–224.

12. Ghanbarzadeh K. The Correct Way to Study and Learn. 1 ed. Tehran: Publishing Company Publication; 1380.

13. Gooneratne IK, Munasinghe SR, Siriwardena C, Olupeliyawa AM, Karunathilake I. Assessment of psychometric properties of a modified PHEEM questionnaire. Ann Acad Med Singapore. 2008;37(12):993–997.

14. Riquelme A, Oporto M, Oporto J, et al. Measuring students’ perceptions of the educational climate of the new curriculum at the Pontificia Universidad Catolica de Chile: performance of the Spanish translation of the Dundee Ready Education Environment Measure (DREEM). Educ Health (Abingdon). 2009;22(1):112.

15. de Oliveira Filho GR, Vieira JE, Schonhorst L. Psychometric properties of the Dundee Ready Educational Environment Measure (DREEM) applied to medical residents. Med Teach. 2005;27(4):343–347.

16. Roff S. The Dundee Ready Educational Environment Measure (DREEM) – a generic instrument for measuring students’ perceptions of undergraduate health professions curricula. Med Teach. 2005;27(4):322–325.

17. Clapham M, Wall D, Batchelor A. Educational environment in intensive care medicine – use of Postgraduate Hospital Educational Environment Measure (PHEEM). Med Teach. 2007;29(6):e184–e191.

18. Jalili M, Hejri SM, Ghalandari M, Moradi-Lakeh M, Mirzazadeh A, Roff S. Validating modified PHEEM questionnaire for measuring educational environment in academic emergency departments. Arch Iran Med. 2014;17(5):372–377.

19. Tomietto M, Saiani L, Saarikoski M, et al. Assessing quality in clinical educational setting: Italian validation of the Clinical Learning Environment and Supervision (CLES) scale. G Ital Med Lav Ergon. 2009;31(3 suppl B):B49–B55.

20. Saarikoski M, Leino-Kilpi H. Association between quality of ward nursing care and students’ assessment of the ward as a learning environment. J Res Nurs. 1999;4(6):467–474.

21. Boor K, Van Der Vleuten C, Teunissen P, Scherpbier A, Scheele F. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach. 2011;33(10):820–827.

22. Aspegren K, Bastholt L, Bested KM, et al. Validation of the PHEEM instrument in a Danish hospital setting. Med Teach. 2007;29(5):498–500.

23. Riquelme A, Padilla O, Herrera C, et al. Development of ACLEEM questionnaire, an instrument measuring residents’ educational environment in postgraduate ambulatory setting. Med Teach. 2013;35(1):e861–e866.

24. Takmil F, Amini M, Nabeiei P, et al. A brief description of Medical Education Master Program in Shiraz University of Medical Sciences. J Adv Med Educ Prof. 2015;3(4):153–153.

25. Carney PA, Eliassen MS, Pipas CF, Genereaux SH, Nierenberg DW. Ambulatory care education: how do academic medical centers, affiliated residency teaching sites, and community-based practices compare? Acad Med. 2004;79(1):69–77.

26. Whitcomb ME. Ambulatory care education: what we know and what we don’t. Acad Med. 2002;77(7):591–592.

27. Hunt CE, Kallenberg GA, Whitcomb ME. Medical students’ education in the ambulatory care setting: background paper 1 of the Medical School Objectives Project. Acad Med. 1999;74(3):289–296.

28. Lenaz M. Clinical education in the ambulatory care setting. Conn Med. 1997;61(10):673–674.

29. Neidlinger N, Singh N, Klein C, et al. Incidence of and risk factors for posttransplant diabetes mellitus after pancreas transplantation. Am J Transplant. 2010;10(2):398–406.

30. Fayers PM, Machin D. Quality of Life: The Assessment, Analysis and Interpretation of Patient-Reported Outcomes. West Sussex: John Wiley & Sons; 2013.

31. Jafari P, Forouzandeh E, Bagheri Z, Karamizadeh Z, Shalileh K. Health related quality of life of Iranian children with type 1 diabetes: reliability and validity of the Persian version of the PedsQL Generic Core Scales and Diabetes Module. Health Qual Life Outcomes. 2011;9:104.

32. Gagné RM. The Conditions of Learning. 1st ed. New York, NY: Holt, Rinehart and Winston; 1965.

33. Yildiz E, Güngörmüş Z. The validity and reliability study of the Turkish version of the evidence based practice evaluation competence questionnaire. Nurse Educ Today. 2016;45:91–95.

34. Latta L, Tordoff D, Manning P, Dent J. Enhancing clinical skill development through an Ambulatory Medicine Teaching Programme: an evaluation study. Med Teach. 2013;35(8):648–654.

35. Csikszentmihalyi M, Larson R. Validity and reliability of the experience-sampling method. In: Csikszentmihalyi M, editor. Flow and the Foundations of Positive Psychology. Dordrecht: Springer Netherlands; 2014:35–54.

36. Riquelme A, Padilla O, Herrera C, et al. Measuring the educational environment in ambulatory settings. Educ Med. 2015;16(2):131–140.

37. Khan JS, Tabasum S, Yousafzai UK, Fatima M. DREEM on: validation of the Dundee ready education environment measure in Pakistan. J Pak Med Assoc. 2011;61(9):885.