Peter Edwards and I are very eager to help our students be ready for study abroad. Our JALT 2015 poster grows out of previous work we have done, and is informing a new project we are working on now.
Since 2004 I have been trying to help my students better understand the reality of what study abroad at the university level will be like.
[read more=”Read more” less=”Read less”]Part of this process has been asking my students through the years what they believe they are getting ready for – their expectations about study abroad. Over the years I have informally collected information about this both in conversations with students and through written responses, but in more recent years I have tried to collect larger amounts of data about what my students believe.
In 2010 I presented at JALT on this work, and in 2013 wrote an article about it (available here). This article describes some of the things that came before the project we are working on now.
In talking about this research with colleagues, I met Peter, who taught me more about survey design and the kind of quantitative analysis that could be run on the data. We worked together to develop a new survey, and conducted our new version in 2013.
We presented at JALT in 2013 on some interesting basic frequencies from this data set, but since then we have done more with it. This year we are looking at subgroups of students through crosstabs.
Our survey was conducted using both a paper-based survey and the software package LimeSurvey. Our questionnaire was in Japanese, and asked students to respond on a six point scale to a set of Likert items. We also had a few items about background: which program the student was in, past study abroad experience, year in university, and whether the student had talked with someone about what study abroad would be like (with the answer giving some indication of how much experience that informant had).
As part of another project we are working on now we wanted to look at this data set again from scratch, in addition to the new tests we would be running on it. We originally had 494 responses, but some of these were unusable. We removed responses which were incomplete. Attempting to find those respondents that might have answered randomly, we also removed some respondents that gave inconsistent answers more than once in the survey (for example, if a student said they agreed or strongly agreed that lecture is common, but then also said they felt strongly that US students would be surprised by a lecturer, I marked this as inconsistent; if a student had multiple inconsistencies like this, we considered them for removal).
Going through these steps we decided on cuts that brought our working data set to 468 respondents.
We then took the six point responses to the Likert items and made two new recoded versions. In one we divided the students into disagree vs. agree (combining the strongly disagrees, disagrees, and somewhat disagrees into one category, vs. the somewhat agrees, agrees, and strongly agrees).
We also made another version to compare students with stronger opinions on each item. We did this by recoding the strongly disagrees and disagrees as one category, with the agrees and strongly agrees as the other. The more neutral responses were coded as “unknown” for this, allowing us to ignore them in the crosstabs comparing the students with non-neutral opinions.
With this data set I also conducted exploratory factor analysis (EFA) on the Likert items using principal axis factoring (PAF) with GNU/PSPP (a free and open source alternative to SPSS) on the Likert items. I made this choice after reading http://pareonline.net/pdf/v10n7.pdf This advises 1) EFA is more useful than Principal Component Analysis (PCA) since it is better able to identify factors without conflating things that probably do not go together, 2) PAF is a good choice where there may be non-normal data). Following advice in the same article I looked at the scree plot and the factors it produced. I then followed the advice in the article to “After rotation… compare the item loading tables; the one with the ‘cleanest’ factor structure – item loadings above .30, no or few item crossloadings, no factors with fewer than three items – has the best fit to the data.” They do advise in this article that oblique rotation methods will provide more useful results than orthogonal methods, but PSPP does not support oblique methods, so at the time I did not have oblique rotation methods at my disposal. I used varimax and ran it with both correlation and covariance as the method to compare the results.
This allowed us to produce a new item in the data set to tag students with/without a study abroad focus. The factor was produced based on how students responded to these items (Cronbach’s Alpha = 0.79). For this survey we summed their responses to them to produce the factor:
- I think I have realistic expectations about what classes abroad are like.
- I expect my classes in Japan to help prepare me for study abroad.
- I want to study abroad no matter what.
- If there were an elective class about what classes abroad would be like, I (think) I would like to take it.
- Whenever I think of my future career, I imagine myself using English.
- I am likely to study abroad.
We also produced a factor for WTC (willingness to communicate) (Cronbach’s Alpha = 0.77) based on these items:
- The reverse-coded form of this item: “I often feel anxiety about my English ability, and this anxiety hinders me from speaking in class. ”
- I would approach lost foreigners on the street, and help them using English.
- In class when something is unclear, I usually raise my hand to ask the teacher to explain.
- I often volunteer my opinions in discussion classes, without a teacher calling on me.
- In English class I speak up if I disagree with the teacher or a classmate.
Finally, we had a factor for whether students are oriented toward Japanese media and values, or overseas media and values based on responses to these items. Chronbach’s Alpha for this one is only 0.56, though:
- I prefer the values and customs of Japan over others I know about.
- I prefer watching films and TV programs made in English-speaking countries.
- One reason I want to study abroad is to escape life in Japan.
After doing all this, I used R with the software package R Commander to run crosstabs on the binary forms of all the questions related to study abroad expectations. I checked to see if there was a significant difference on the binary form of each statement if we compared…
- the population items
- the students that had the non-neutral scores on the factors (the top and bottom quartiles)
- the binary form of all the Likert items (looking at all 468 students each time)
- the students with non-neutral views on the Likert items (students who answered 1 or 2 or 5 or 6, ignoring the 3 and 4s)
For this poster session, we have selected only those crosstabs where we have at least 30 respondents in each subgroup being compared and a p-value under 0.01. We did not have any very large effect sizes here, but we are still interested in researching and sharing the results, especially since this is exploratory research.
Reflecting on the results of this project, and using the results of this research, we are now working on analyzing the results of a new survey we designed and conducted earlier this year, with a grant from our university’s Intercultural Research Institute. We are currently examining the data we have collected from this new survey, and intend to share the results of this research in the near future.[/read]
