RealELT

(Although this post appears as having been published on Feb 11, 2017, like the others below it was originally posted here in November of 2015, and accompanied our JALT presentation at that time. I have kept most of the original phrasing.)

This post is about some of the things we were most interested in sharing at JALT 2015, and focuses only on those findings. Further below you can also find pdf versions of the charts on our poster (with more detailed numbers than you’ll find in this post), a discussion of what we did to get these results, and links to previous work we did as well as a brief introduction to a project we are working on now.

[read more=”Read more” less=”Read less”]

In 2013 we (Stephen Shrader and Peter Edwards) presented the results of a survey we conducted with the help of colleagues at our university regarding what our students believe about study abroad. At that time we were primarily looking at whether students fell on the side of agreeing/disagreeing with statements about what they thought they would experience in classes overseas.

Since then we have been working on a new survey, and as part of the process we were looking at the 2013 dataset using some new tests. This time we are looking at how different subgroups of students in our sample responded when we examined their responses with cross tabulations (crosstabs). For this poster we are only discussing differences where we had at least 30 respondents in each subgroup and a statistically significant p-value (p < 0.01).

We tested for differences based on a wide range of things such as which program the student was in, their year, past study abroad experience (prior to university), and how they responded on other survey items (to see if some beliefs/attitudes corresponded with others). Interestingly, we found few correlations based on student background – most things that emerged seemed to be in how students answered certain survey items on beliefs/attitudes.

All our questions were about what students thought US university students do in regular content courses – not language courses.

Some of the things we found interesting in this data set:

Informal Debate

It seems that a majority of our students (79.1%) believe that informal debate (where individuals disagree and argue with each other, but without set teams or times) is common in US university classes.

With regard to this item, we found a correlation between student beliefs about how they think US students would feel about a professor that lectures. We found that our respondents who believe US students would think of a professor that lectures as strange/unusual are more likely to think informal debate is common.

Full Class Discussion Format

One of our questions asked about whether students believe full class discussion is more or less common than other discussion formats such as pair work or small group work. This item was trying to discover if students realize they might have to jump into discussions overseas without any kind of pair work as a warm-up, and participate in a discussion with everyone in the room listening. We found that 71.6 of our respondents think that full class discussion is more common than other discussion formats.

With regard to this item, one interesting thing was a difference in student attitudes about willingness to ask a teacher for clarification. We found that students who have a stronger tendency to be willing to ask a teacher for clarification in class were more likely to think full class discussion format is common, while students that say they would not be likely to raise their hands and ask for help were also more likely to think other discussion formats would be more common than full class discussion. (Perhaps a bit of wishful thinking here?)

Pair Work

One thing that concerned us is the large number of students that seemed to think pair work is common in US university content courses. Only 18.8% of our respondents seemed to think pair work is not something US university students do (again, we are not talking about language classes here, but content courses).

On this item we found a significant correlation between student beliefs about lecture. Unsurprisingly, students that think lecture is common are less likely to think of pair work as common. (The problem here, and what makes it interesting to us, is that many of the students do seem to think of pair work as common, while the majority think lecture is uncommon.)

Small Group Work

75.4% of our respondents fell on the side of saying small group work (in teams of 4-6) is common.

Again we found a correlation with student beliefs about lecture. Students that think US students would find a lecturer to be unusual/strange were more likely to think small group work would be common.

Reading/Text Use

We had a couple of items related to textbooks and reading, and the results were similar for those two items. We found that about while about 60% of the students do think they will be using textbooks overseas (and have more reading-based homework than in Japan), about 40% do not, and tend to think US professors in America avoid the use of texts rather than having them as part of the course. We were happy to find the majority here understands they will be doing a lot of reading (though we are not sure if they realize just how much it involves).

For this item, we found that students in our Center for International Education (our CIE is where students have courses just before going overseas, with international students and taught similarly to courses in our partner schools overseas) were more likely to know that US students have more reading than students in Japan – unfortunately we only had data from 64 students in that program, though. We also find that sophomores were more likely than first-year students to know they’d have more reading in the US.

Once again, we also found a correlation between students’ beliefs about lecture, too. There seems to be a correlation, at least in this sample of students, between assumptions about textbook use and lecture. Students that were more likely to think of lecture as common were more likely to also think of reading as important. Students that thought less of the importance of texts and reading were less likely to think of lecture as a common experience in US classes.

We also had a set of items on the survey related to willingness to communicate (WTC). We found that, at least in this sample, students with a higher WTC were more likely to think US professors make use of texts, while students with a lower WTC were more likely to think US professors avoid the use of texts.

Academic Writing vs. Kansobun

We were happy to find that most (72%) of our students seem to understand that university-level writing would be about data and theories rather than kansobun-type writing (which our survey item described as being more like a personal opinion/experience-type essay, in addition to using the Japanese term kansobun). While still concerned about the 28% that think kansobun-style writing is what they would be doing, we were happy to see the majority of students know the writing would not be like a kansobun, we did get some interesting results that also concerned us when we ran crosstabs for this item.

Our survey had a set of items that together indicated whether a student is oriented toward Japanese values and media, or instead is looking outside of Japan for values, and is seeking to “escape” this culture. We found that in this sample, and somewhat surprisingly, that students who prefer Japan and may be more domestically oriented were more likely to think of the writing done in US universities as about data and theories, while students more likely to be oriented toward overseas media, looking outside Japan for values, and seeking to escape this culture were more likely to think of the writing to be kansobun-type writing.

We also had a set of items that sought to identify which students would be oriented toward actually wanting to study abroad and actively preparing for it, vs. students that said they had less interest in study abroad and fewer plans to do it. We were concerned to find that the students with stronger plans to study abroad to be more likely to believe the writing would be like a kansobun. The students with fewer plans to go were more likely to have an accurate understanding of what the writing would be like.

This item also produced a couple of other interesting results. We found that students who thought US students would be unsurprised by a lecturer were also more likely to know the writing would be about data and theories rather than personal experience.

We also found students who said they would be more likely to raise their hand and ask a teacher to clarify something in class were also more likely to think of the writing as a kansobun. The students who were less likely to raise their hands and ask for help were more likely to think of the writing as being about data and theories.

Lecture

Despite the fact that our students do experience a lot of lecture in our partner schools overseas, this may not be the image our students have of what they will experience in a US classroom. It seems we have a good number of students that do not associate the technique of lecture with classes conducted in English-speaking countries. We had two items on the survey related to our students’ assumptions about lectures. On one of them (not one we are reporting on at JALT), we found that over half of our students think that lecture is uncommon in the US. One item we are reporting at JALT produced an almost even split between our students on whether they think US students would think a professor that teaches primarily using lectures is strange. We had hoped a larger number of students would realize that while the way lectures might be conducted is not necessarily the same as in Japan, it is still a widely-used teaching technique.

As mentioned above, we found a correlation between student beliefs on whether or not texts are commonly used in the US with how students responded to this item on surprise at a lecturer. Students that think texts will be an important part of their overseas experience were more likely to think US students would be unsurprised by a lecturer. This concerns us since 40% of our respondents did not think of texts/reading as common.

Office Hour Visits

We were happy to see that most (84.4%) respondents think US students visit their professors during office hours at least a few times each semester.

One interesting result from the crosstabs on this item was that students with strong opinions on the lecture item discussed above (surprise at a lecturer) had different opinions on the office hour visit issue. Respondents that think US students would be surprised by a lecturer were also more likely to think of office hour visits as common. Might this reflect a perception of a lecturing professor as inapproachable?

We also found that students who think informal debate is a common class activity were more likely to think of office hour visits as common.

In Closing

We decided to focus our attention in this poster on results that came back as statistically significant, and also where we had at least 30 respondents in each subgroup to compare. As mentioned in the introduction, more details are available in the introduction. This research is very exploratory in nature.

The thing that surprised me (Stephen) is that most of the items related to background or past experience did not produce statistically significant results. Even though we had a pretty even split of first-year vs. second-year students, and also intensive English students vs. regular English majors, these differences in program/year did not seem to consistently relate with the students’ preconceptions about study abroad. We also had questions to identify which students had prior study abroad experience before coming to university, as well as which students had taken the time to talk with people about what study abroad would be like. Interestingly, though, these items did not produce strong enough results to report here. As we continue our research with new surveys we might find these things are actually important, but in this dataset the students’ beliefs about what study abroad would be like seemed to mostly correlate with certain other preconceptions (text and lecture as connected), or in some cases differences in attitude (such as WTC).[/read]

The files at the bottom of this post are pdf versions of the information we presented at JALT 2015.

[read more=”Read more” less=”Read less”]Each of these documents takes one of the items from our survey related to study abroad expectations and lists that item at the top of the page.

The first bar chart shows how the results came out when we look at all 468 respondents. This is followed in the subsequent bar charts by comparisons of how different subgroups responded to that item. This allows you to compare how a particular subgroup answered vs. the subgroup they were compared with in crosstabs, and also how that subgroup different from all respondents.

At the top of the page you can see a text box, and in the middle it has a line like this | indicating a division between students that disagree with the proposition vs. agree. The bar charts on a given page are organized so this center line marks the dividing point between disagree|agree for every subgroup.

A further discussion of our survey is provided in this post below.

1. bar_chart_example_version_1
2. d38_to_present_p_1_of_2
3. d37_to_present_p_1_of_1
4. d36_p_to_present_1_of_1
5. d34_to_present_p_1_of_1
6. d33_to_present_p_1_of_1
7. d32_to_present_p_1_of_1
8. d30_to_present_p_1_of_1
9. d29_to_present_p_1_of_1
10. d22_to_present_p_1_of_2[/read]

Peter Edwards and I are very eager to help our students be ready for study abroad. Our JALT 2015 poster grows out of previous work we have done, and is informing a new project we are working on now.

Since 2004 I have been trying to help my students better understand the reality of what study abroad at the university level will be like.

[read more=”Read more” less=”Read less”]Part of this process has been asking my students through the years what they believe they are getting ready for – their expectations about study abroad. Over the years I have informally collected information about this both in conversations with students and through written responses, but in more recent years I have tried to collect larger amounts of data about what my students believe.

In 2010 I presented at JALT on this work, and in 2013 wrote an article about it (available here). This article describes some of the things that came before the project we are working on now.

In talking about this research with colleagues, I met Peter, who taught me more about survey design and the kind of quantitative analysis that could be run on the data. We worked together to develop a new survey, and conducted our new version in 2013.

We presented at JALT in 2013 on some interesting basic frequencies from this data set, but since then we have done more with it. This year we are looking at subgroups of students through crosstabs.

Our survey was conducted using both a paper-based survey and the software package LimeSurvey. Our questionnaire was in Japanese, and asked students to respond on a six point scale to a set of Likert items. We also had a few items about background: which program the student was in, past study abroad experience, year in university, and whether the student had talked with someone about what study abroad would be like (with the answer giving some indication of how much experience that informant had).

As part of another project we are working on now we wanted to look at this data set again from scratch, in addition to the new tests we would be running on it. We originally had 494 responses, but some of these were unusable. We removed responses which were incomplete. Attempting to find those respondents that might have answered randomly, we also removed some respondents that gave inconsistent answers more than once in the survey (for example, if a student said they agreed or strongly agreed that lecture is common, but then also said they felt strongly that US students would be surprised by a lecturer, I marked this as inconsistent; if a student had multiple inconsistencies like this, we considered them for removal).

Going through these steps we decided on cuts that brought our working data set to 468 respondents.

We then took the six point responses to the Likert items and made two new recoded versions. In one we divided the students into disagree vs. agree (combining the strongly disagrees, disagrees, and somewhat disagrees into one category, vs. the somewhat agrees, agrees, and strongly agrees).

We also made another version to compare students with stronger opinions on each item. We did this by recoding the strongly disagrees and disagrees as one category, with the agrees and strongly agrees as the other. The more neutral responses were coded as “unknown” for this, allowing us to ignore them in the crosstabs comparing the students with non-neutral opinions.

With this data set I also conducted exploratory factor analysis (EFA) on the Likert items using principal axis factoring (PAF) with GNU/PSPP (a free and open source alternative to SPSS) on the Likert items. I made this choice after reading http://pareonline.net/pdf/v10n7.pdf This advises 1) EFA is more useful than Principal Component Analysis (PCA) since it is better able to identify factors without conflating things that probably do not go together, 2) PAF is a good choice where there may be non-normal data). Following advice in the same article I looked at the scree plot and the factors it produced. I then followed the advice in the article to “After rotation… compare the item loading tables; the one with the ‘cleanest’ factor structure – item loadings above .30, no or few item crossloadings, no factors with fewer than three items – has the best fit to the data.” They do advise in this article that oblique rotation methods will provide more useful results than orthogonal methods, but PSPP does not support oblique methods, so at the time I did not have oblique rotation methods at my disposal. I used varimax and ran it with both correlation and covariance as the method to compare the results.

This allowed us to produce a new item in the data set to tag students with/without a study abroad focus. The factor was produced based on how students responded to these items (Cronbach’s Alpha = 0.79). For this survey we summed their responses to them to produce the factor:

• I think I have realistic expectations about what classes abroad are like.
• I expect my classes in Japan to help prepare me for study abroad.
• I want to study abroad no matter what.
• If there were an elective class about what classes abroad would be like, I (think) I would like to take it.
• Whenever I think of my future career, I imagine myself using English.
• I am likely to study abroad.

We also produced a factor for WTC (willingness to communicate) (Cronbach’s Alpha = 0.77) based on these items:

• The reverse-coded form of this item: “I often feel anxiety about my English ability, and this anxiety hinders me from speaking in class. ”
• I would approach lost foreigners on the street, and help them using English.
• In class when something is unclear, I usually raise my hand to ask the teacher to explain.
• I often volunteer my opinions in discussion classes, without a teacher calling on me.
• In English class I speak up if I disagree with the teacher or a classmate.

Finally, we had a factor for whether students are oriented toward Japanese media and values, or overseas media and values based on responses to these items. Chronbach’s Alpha for this one is only 0.56, though:

• I prefer the values and customs of Japan over others I know about.
• I prefer watching films and TV programs made in English-speaking countries.
• One reason I want to study abroad is to escape life in Japan.

After doing all this, I used R with the software package R Commander to run crosstabs on the binary forms of all the questions related to study abroad expectations. I checked to see if there was a significant difference on the binary form of each statement if we compared…

• the population items
• the students that had the non-neutral scores on the factors (the top and bottom quartiles)
• the binary form of all the Likert items (looking at all 468 students each time)
• the students with non-neutral views on the Likert items (students who answered 1 or 2 or 5 or 6, ignoring the 3 and 4s)

For this poster session, we have selected only those crosstabs where we have at least 30 respondents in each subgroup being compared and a p-value under 0.01. We did not have any very large effect sizes here, but we are still interested in researching and sharing the results, especially since this is exploratory research.

Reflecting on the results of this project, and using the results of this research, we are now working on analyzing the results of a new survey we designed and conducted earlier this year, with a grant from our university’s Intercultural Research Institute. We are currently examining the data we have collected from this new survey, and intend to share the results of this research in the near future.[/read]