Student Peer Review Process? Here’s My Version

When I tweeted yesterday that I was creating a peer review process for my students to assess each other’s writing and provided a sample of what I was up to, I had many requests to share the whole process.

I am happy to do that and will do so here in this post with two caveats:

  1. This process is untested. This means, I can’t even give you anecdotal feedback as to if it works. So for now I will share the materials but it is probably wise to warn you it may backfire. Conversely, it may also be brilliant. I have never tried peer review before but I’ve spent about a year dreaming up this process. I really wanted to do double peer review, meaning I really wanted students to peer review two of their classmate’s papers but as I tried to schedule it, I recognized in our compressed term, there just isn’t enough time to do that and still allow for the time to do research, the synthesis exercise, and the peer review process. I’ll write a follow up blog after I have trialed the process.
  2. Remember that all writing is contextual. I’ve created this process specific to the needs of my assignment, my course content, and my teaching style. If you want to model what you do after my process, you won’t be able to simply lift my points and my language choices and have a neat fit. Make it your own.

Here is the document describing the process:

Qualitative Paper Assignment

In the document you will find:

  1. My detailed assignment guidelines for the qualitative synthesis paper.
  2. A description of the peer review process.
  3. My content rubric for the paper (out of 33)
  4. The peer reviewed checklist for the students to help them assess their classmate’s work.

As this is an assignment structured using scaffolding theory, these are the basic activities to scaffold the process:

  1. As the course I teach is a research course, I start the term introducing them to the principles of qualitative research. As I teach them about qualitative methods, I am simultaneously connecting the theory content to what they need to write about. They also get a class on searching the literature and what needs to be included in the background to a research study.
  2. I have the students find 5 primary qualitative studies on a topic of their choice. They have to show me these 5 primary studies. There are numerous benefits to this requirement, most of them, selfishly benefit me: a) I ensure that they write this paper using the literature I asked for and not just any old literature. It saves me so much grading time to check this in advance; b) I get to look each one of them in the eyes at least once during this process. For some students, the article checks are the only conversation I’ll have with them all term; c) the students can write their paper knowing they won’t lose marks for incorrectly identifying qualitative studies.
  3. I provide them with an article identification guide to help them distinguish qualitative, quantitative, and discussion papers. Article Type Identification Guide
  4. I have them read their articles next and then we come together and do a synthesis exercise over one class.  I wrote a tweet thread about that exercise a while back (see below).
  5. The students will go off and write and before the planned peer reviewed process they would hand it in to me and I would grade it. The next steps are steps that are untrialed.
  6. Now students will hand in a preliminary paper and share that preliminary paper with a pre-selected classmate.
  7. Each will use the peer review checklist to independently comment on the other’s paper. They will also be welcome to correct grammar and APA.
  8. I will hold a peer review class where they can ask me questions but the goal is also for them to sit down with the rubric for the paper and jointly come up with a score for the content portion of each of their papers. Two chances to practice with the rubric.
  9. Notice that the rubric and the peer review checklist are colour coded. That’s because the colours that match on both documents are the items on the peer review checklist that they will be considering when scoring that section on the rubric. The rubric is broad areas of writing (content, synthesis, research, mechanics) the checklist is ordered by the required heading sections of the paper. Hopefully this will help them realize what expectations affect what sections of the rubric.
  10. After the peer review process, the students will go off and revise their papers based on their discussions. They will hand in a final draft of their paper to me along with a self-scored rubric for their own content.  Their grade will be the score they give themselves — unless they score themselves too low then they will get my score. If they score themselves too high? Well that’s a bridge I’ll cross when and if it happens.
  11. I am not having the students grade each other or assess themselves on APA. They’ll obsess over it and it will take away from what I want them to focus on, which is their content. I’m going to grade their APA.
  12. 50% of their entire course grade goes to the little pieces that make up this assignment. The breakdown of the assignment in terms of value is:
    1. Content of paper 33/
    2. APA 3/
    3. Participating in synthesis class and making progress 3/
    4. Article checks 1/
    5. Peer review 10/

 

How I will score their peer review exactly is still a work in progress and I’m still reflecting on it as I don’t have to “grade it” for a couple months yet. It will be a combination of participating in the peer review, collegiality with peer and being a good citizen and getting documents in on time, a thorough effort at feedback to their peer, the degree to which they pay attention to the feedback they were given in their next draft, a one paragraph response to that feedback, and realistic self-assessments on the rubric.

I’m blessed this term with an abnormally small class which will be extremely useful for me to trial this process and discover the weak spots in my pedagogy.  I hope, those of you who were interested also find this useful. Please feel free to comment below with questions or tweet at me to discuss, or share your own peer review processes.

Part II: The Study is Done. What’s Next?

If you are coming to this blog first, this is part II of a three part series describing the story behind my research for this study:

Exploring Self-Efficacy and Anxiety in First-Year Nursing Students Enrolled in a Discipline-Specific Scholarly Writing Course

Part I is here.

Part III is here.

Dissemination and All that FUN Stuff 

Tom did the stats and, dammit, we ended up with a p = .051 significance for change in self-efficacy from pre to post course. Every researcher’s worst nightmare. You’ll note I do what most researchers do with p = .051, I call it non significant in the abstract but proceed to talk about it like it is significant. Take that as you wish.

I went to a conference in February 2013 and presented just a basic version of my findings. I focused the presentation on the course design because I knew that was what my audience would be most interested in. I was swamped after the presentation ended. Writing is a huge problem in nursing programs. Everyone is frustrated. After that conference, we almost “sold” (I wouldn’t have made a cent) the scholarly writing course to be used by another institution but they deemed it then too expensive for the size of their student group. It had taken me 7 years to perfect the design of that course which began when I taught it not-for-credit prior to implementing it as a credit course. A detailed description of the course can be found in the appendix of the publication I am discussing and is linked above. The party that had looked into buying it thought why re-invent the wheel? However, the decision makers at their institution said no.

Related, but incidental, the process the finance people use to calculate how much a course design is worth unto itself is pretty fascinating. They need to pay me more. My intellectual property is worth a lot of money to them.

I finished data collection in February 2012, but it took me till fall of 2012 before I did anything with the data. I presented at that February 2013 conference in Edmonton and then in the spring of 2013 for our faculty, and then I sat on it. I started another study in the fall of 2013 to replicate and improve upon what I had done in this study. So I started to read what had been published about undergrad writing in nursing because it was really time I did a more thorough job of reading. It was a sad state of affairs the writing research in nursing, and very limited or anecdotal. For the most part, what I was reading taught me that I had an instinctive knack for figuring out what kinds of questions researchers and educators had about student writing. Next thing I knew it was 2014 and I needed to do more than present, I needed to write the study up and see if someone would publish it.

And I was terrified. Yes me. Terrified of writing. The last thing I had published was the main study for my Master’s thesis. That one seemed to take a long time to go from thesis to draft. It went through (as I recall) 3 peer reviewed processes at the first journal of submission — 2 revise and resubmit requests and then a final smaller revision. This was in what I now call the “olden days” of publishing when you had to mail hard copies of your blinded paper to be reviewed and they snail mailed your reviews back to you. I had graduated with my Masters in 2002. I sent the paper out for publication in 2003. I had a kid in there somewhere because I remember being on mat leave, coming in for my baby shower, and picking up an envelope with my reviews in it. I don’t know when I finally dealt with them. I think it took me a long time because the paper didn’t reach publication until January/February of 2006. In fairness, it went to a good journal so I was pleased. But that was my last academic publication before the one I write about now. By the time I got that article published, I had written a novel. My marriage was on its way to falling apart. My marriage did fall apart. I wrote a second (shitty) novel. I was about to have a midlife crisis and spend about six years just riding my bike ridiculous distances. I didn’t want to write academically. The next thing I knew I woke up and it was 2011. I had done a stint on anti-anxiety meds, calmed the F down, and realized life didn’t need to be lived with so much agitation and competition and I should be enjoying its beauty not tackling it with a vengence. Ironically enough, I came back to my academic sensibilities through that epiphany. Yes, going back to academia actually mellowed me.  I had changed my name back to my maiden name. I decided to conduct a research study and it took me 3 years to get my ass in gear and write it up.

Getting my Undergraduate Students Involved

What’s the quickest way to get a lit review done? Assign your undergraduates to do it for you for credit. This is where authors three and four, Torrie and Holly got involved. I had this idea. My students wrote this paper in my research course where they had to find 5 primary studies on a topic. Not the same paper I use now but its predecessor. I knew enough at this point to know that there were barely 5 articles on my topic but there were some. I threw out an offer to the class in the fall of 2014. If anyone would explore writing research and writing self-efficacy, I would use their work to structure my review of the literature and I would give them authorship on the published work. Three students stepped forward with interest. One eventually dropped out. I helped them hone their topic. Torrie looked at the studies which had developed interventions to improve writing. Holly wrote about self-efficacy and anxiety. The additional articles that Torrie and Holly found began pushing my reading into the social science literature which is where my ideas on writing instruction and research really began to develop.

Fast forward into the future and when I was finally prepping the first version of the manuscript for publication in spring of 2015, the journal we first sent to required more than just manuscript involvement for publication credit so Torrie and Holly, the troupers that they are, agreed to explore the qualitative data I had for this study. I asked them to read through that data with a lens toward statements made that supported the quantitative findings. They did a wonderful job and identified some similar points to each other and learned the value of parallel data assessment and trustworthiness of qualitative analysis and I could now tick off “contributed to analysis” beside their names for project involvement. The findings of their analysis are mentioned briefly in the paper in the discussion section at the bottom of the first paragraph on p. 12.

I think of everything I did within this project, getting Torrie and Holly involved in this work is the part I am most proud of. These were two bright students (now nurses) who sacrificed researching something more glamorous and clinically based (the stuff students value) to gain a really important academic credential that they can talk about and use on their CVs for years. I was happy to give them this opportunity.

The final stage of this process was the publication experience which can be found in Part III

The Story Behind the Research #1: On Naivety and Perseverance

One of the things we don’t do so well in academia is to talk about the stories behind the the structured, impersonal research articles we write. So think of this post as the commentary on the DVD for a movie you watched, or in this case the academic article you just read, or should read, or will eventually read at some point in the very distant future.

Exploring Self-Efficacy and Anxiety in First-Year Nursing Students Enrolled in a Discipline-Specific Scholarly Writing Course

The first draft of this blog was really long so I’ve broken it into three parts. The first part talks about how I naively plunged into designing this study with little mentorship or guidance. Part II talks about how long it took me to do much of anything with the data and how I got my undergraduate students involved. Part III is the perseverance part because getting this study published was an adventure.

The Naivety: Planning and Conducing the Study

It was about October 2011.  It was the second year of our “new” Baccalaureate Program and I was in the second year of teaching my newly minted “for credit” Scholarly Writing course offered to our first-year students. The idea had been festering in the back of my mind for a year to study my Scholarly Writing course and its effectiveness. My credentials at that point in time: I had been teaching research to undergraduates for about 4 years at this point; I had done a research based thesis for my Master’s; I sat about 5 feet away from one of our psych instructors who was also teaching stats for the program (my second author Dr. Tom Harrigan, who would never let me call him Doctor anything in real life); and, perhaps the most critical factor, there was no one around who would tell me not to do it.

So what the hell, lets do a research study.

I had read very little writing research literature. Actually it would be more honest to say   I had read nothing about writing research. I knew nothing about Self-Efficacy Theory other than that it existed. I had one meeting with our Research and Planning Director who was also the chair of the research ethics board at the College. I knew him from cycling before I knew him as the research guy. He gave me a few study design tips and tips on how to ensure ethical approval and off I was writing an ethics proposal.

I think maybe it worked in my favour to walk into planning and implementing a study on my own. I didn’t have anyone around me asking questions I’m glad I didn’t think of myself like, “shouldn’t you have a PhD to conduct a research study?” Knowing myself as I do, it is highly likely someone did question my credentials and experience in some way and I just didn’t hear it or process it. Being the girl who went out and rode a 300km bike ride last spring on almost no training, it is safe to say that when I get an idea in my head I’m pretty hard to stop no matter how foolhardy that idea might be. I suppose, I had Tom, but Tom’s main involvement didn’t come until I had the data and the stats needed to be run.

The course started at the end of November 2011 and I am pretty sure that I didn’t get ethical approval until just before Christmas. My manuscript says I collected data for the pre-test 4-classes into the term which would be about three weeks into the term because the class was held for 2 hours once a week.

I used Google to find a questionnaire. That’s how I found the General Self-Efficacy Questionnaire. I didn’t know Writing Self-Efficacy (or that other people had made questionnaires for it) existed. I developed the writing self-efficacy questionnaire following the pattern of that General Self-Efficacy Questionnaire. I consider myself lucky (now three studies deep and some reliability and validity data in my pocket) that the questionnaire works. It is probably best that I don’t go into too many details about how very naive I was to go out and just MAKE my own questionnaire. It didn’t occur to me not to. I had developed a questionnaire for my Master’s thesis too. I have a publication describing it. (Yes, I’ve changed my name. We’ll get to that). So of course I was going to develop a questionnaire. I was an experienced writer, writing teacher, and I knew the things that students sat down and talked to me about while clearly overwrought and distressed in the midst of the writing process.

So I plunged forward and ran the study and made oodles and oodles of mistakes. For example:

  1. I wasn’t really pretesting these students I was early testing them. They were 3 weeks into the course before they got their pretest.
  2. I had them develop a personal identification code so I could keep the students anonymous. That code asked them to use their Mother’s initials and date of birth.  Well… some have more than one mother, and their mother has had more than 2 names in her life, and sometimes the birth date was used and sometimes the birth month. And some of them still forgot their code. Lesson learned, nothing is fool proof.
  3. I did the post-test measure into the following term in someone else’s class. The person who was teaching was not willing to give me much of her class time for the students to complete the questionnaire. Some of them took the questionnaire home and I never saw it again. When I got beautiful written reflections about their writing experience in the early part of the course, in post test, the qualitative questions were left blank or were one or two sentences long.
  4. I didn’t think about program attrition. I’m not really sure we realized the degree of our own program attrition at that time but there were many students due to multiple course failures, that didn’t even make it to that third term. I lost all my low grade students from the sample and they are an important voice in writing self-efficacy.
  5. Anxiety — I asked them to rate their anxiety based on their “next” scholarly paper which worked well for the first data collection period because they were about to write the paper assigned in my course. But I made a stupid and naive decision for the post-test. These students didn’t have a known scholarly paper coming up at the end of the course. What exactly were they rating or imagining when I asked them to rate their anxiety for their next scholarly paper?

Nevertheless, when I went forth and analyzed the data some interesting things emerged. And I realized that since I’d last written a research study paper, that all the rules had changed. Two things I will discuss in future parts.

Part II

Part III

 

 

Can I Have an Extension, Please?

Where is the line in the sand when it comes to granting or refusing extensions on writing assignments? The image is a bike tire track on sand but it was the closest I had to a personal image representing that metaphor.

I asked for my first extension as a PhD student a couple of weeks ago. In fairness, my entire class also asked for an extension. It had been strongly suggested to us that we participate as much as possible in the Research Symposium organized by the faculty. The symposium was wonderful so it wasn’t difficult to want to participate, but it was also two full days and two evenings of symposium activities, and one evening of being exhausted taken away from completing an intense 25-page assignment on measurement. We were lucky. (Or charming?) Our prof said yes and it was a like releasing the death grip on our sanity.

Student requests for extensions became a hot and controversial topic on Twitter earlier this week. Extensions are a writing issue that all writers encounter a need for. They are linked to my primary area of study in academic writing: Writing Self-Efficacy. They are also linked to the social inequalities present in academia among students that we may not recognize when we criticize students for their inability to get their work done on time.

The Debate

  1. All students should be given extensions on request regardless of reason.
  2. No student should be given an extension because in the real world, deadlines  are not extended.

It is obviously much more complicated than that but that is the essence of the dichotomy as it comes out of the mouths of black and white thinkers. It might be easier to talk about the shades of grey in between by telling you my process of dealing (or rather not dealing with) extensions.

A number of years ago when I was a much more junior college instructor I observed very quickly that in a class size of 70 students (or more) that it was a time consuming pain in the ass to receive 20 or more emails asking for extensions for various reasons. Not only that, but our department made them fill out a form which was then to go in their file so they could keep track of a pattern of behaviour.  For each student who requested an extension, I would have to make an independent decision to grant or not grant their request.

… but what, after all,  defines a good enough reason?

Illness? A death in the family? Hospitalization? My kid is sick? My parent is sick? I’m too busy with other assignments and clinical? All of the above? Some of the above? None of the above?

I am bound by some department policies. Extensions are allowed, to a maximum of one week, at the discretion of an instructor, and they must be requested 24-hours in advance of a due date (which, think about this, means 72 hours in advance if the due date is a Monday). Something about tracking requests for extensions and making a decision about what was a valid reason for needing an extension based on my own personal view point didn’t sit well with me.

So I began pre-announcing, in the first class of a term, a due date and an automatic extension date for the entire class. No questions asked. AND — this might be the controversial part — I give those who had in their paper “on time” 2 bonus marks.

Here is my rationale for my personal policy:

  1. The students are adults. They should be allowed some control over their life.
  2. No matter how hard we try to say we are preparing students for real life, being a student is NOT real life. Hence why I teach at the college level. It’s been my extended way of avoiding real life for a very long time now. So the argument that no extensions are granted in the real world, doesn’t wash well with me. And it’s pretty much flat out untrue more often than it is true.
  3. I believe that rewards are far more motivating than punishments.
  4. It prevents students from being put in the place of needing to lie to you, or having to accept a punishment such as grade loss.  Having restrictive extension policies such as you can only have an extension if you are sick or someone close to you dies, basically puts students who need extensions for other reasons in a place where they have to make a choice that isn’t a choice.  So they are punished, or they lie to you. Talk about stress and anxiety. I don’t see how either is a good option when you are the less powerful party just trying to survive.
  5. Lets not even go down the road of discussing those who demand doctor’s notes. What a waste of a doctor’s time (and the student’s money — they don’t hand out those notes for free) to send a sick student to sit in a waiting room and spread germs just to humour a teacher’s fear of being lied to and fear being unfair to all the others who were not as unlucky and didn’t get sick.
  6. Sometimes the shit hits the fan within that final 24-hour period.
  7. It demonstrates that I trust my students to make their own judgement call about their lives and what they can manage. It also protects their privacy. They no longer have to tell me their often very personal stories that might be embarrassing to them. And, frankly, I don’t need to know.

How has this worked out for me? I don’t always use this policy. When I am using true scaffolding writing strategies, for example, where students are completing their assignment in small stages across the term, it is often not needed. But when I use it, I find it very successful. Here are some of the discoveries I have made along the way:

  1. Students need a reminder to psychologically view the first due date as the due date because if they start thinking of the extension date as the due date, some of them will run into big troubles with procrastination. I structure the pre-assignment expectations as such that no student should be in the position of not having something started by the time the first due date rolls around.
  2. I tell them to not make a decision until the last minute so the extension is used only if they really need it. I also tell them not hand in crap just for bonus marks. They can often earn far more than the 2 marks just with taking a little time to better edit their paper. (Problem I acknowledge: it requires that they understand what kinds of editing will get them more marks. Some of them don’t.)
  3. 100% of terms over the last 5 academic years has seen more students taking the extension than not. In fact the number of students handing in the paper “on time” ranges from 0 to 15 out of an average of 60 students per class.
  4. The majority of students handing in the paper “on time” are the A-grade students anyway. So the bonus may not be necessary (but I’ve never tested that theory).
  5. Most students know that putting off their school work just makes subsequent school work harder to attend to. There is no down time in  nursing school.
  6. With the very rare and extreme exception, further extensions are not granted.

I’ve heard the arguments about being limiting with extensions. That it isn’t fair to the students who hand in their work on time. That it is laziness and procrastination that makes them request extensions. In my experience, it is rarely laziness or poor time management that leads to a student wanting an extension. It’s the very complicated lives our students lead. Arguments of fairness assume all the students are starting off on a level playing field. They aren’t. You cannot compare the responsibilities of a student who is single parenting three children to the responsibilities of a 22 year old, living at home with her parents and has no job but school work. And that 22-year old may be being abused by her boyfriend. You simply don’t know.

And here is a crazy but anecdotal observation — those with the strictest extension policies for students are also the first to demand concessions for themselves when they need more time.

There are students in our classrooms who are working just so they can feed their family and have many other disadvantages both visible and invisible. It’s family demands, especially in cultures where you cannot say no to your family if they show up at your door at 8 PM expecting to be entertained. It’s being educationally disadvantaged your whole life and being behind in understanding the lingo and behaviours that define being “a good student.” It’s low self-efficacy that can paralyze thinking and emotions. It is an exhausted brain demanding that it be shut off for the night even though the assignment is due in 6 hours. It’s anxiety, which is really a legitimate illness, except it isn’t a socially acceptable illness so how do you describe your “illness” when you aren’t barfing or feverish, so you aren’t “really” sick, but you can’t seem to get anything done anyway?

Having said that, there are limits. There are lines in the sand. Extensions cannot drag on and on or be unlimited. We have due dates to manage our own workloads and personal circumstances too.  The college has a generous 3-month post end of a course policy for outstanding assignments and exams which can be activated in exceptional circumstances. We’ve put it into effect numerous times with students in need (a student whose house burned down, another having gastrointestinal hospitalizations, another hospitalized for mental health issues, another with cancer).

So there is willingness to allow extensions on due dates, but sometimes a student needs to consider their status as a student. Students under stress often have limited insight into when they need to ask for help or need to take the load of school off their shoulders. They think they can do their school work despite the world crashing down around them. They make bad decisions. Sometimes a one week extension isn’t enough to get through. Fortunately I am in a program with diligent and excellent student advisors (who are also nurses) who can coach students through this decision making process. Sometimes an extension is not what a student needs. What they need is permission to stop, take a breath, fix their lives, and come back healthy. School will still be there.*

*mini side rant about scholarships and funders who take student’s money away for failures or time off for illness. That BS needs to stop too. Not all students live lives, or have the academic naturalness, to handle the full time course loads required by some funders.

Let’s Chat about our English as a Second Language (ESL) Students and Writing

Warning:  I’m about to climb up on top of a giant soap box. Maybe you’ll climb up with me, or maybe you won’t. But I would like to hear your experiences with what I am about to talk about if you are willing to share.

Disclosure: In Canada we have two official languages, French and English, but I am fluent in only one (I’ll let you guess which one). I have also always said that I was gifted enough talent in one language to make me forever incompetent in all other languages. I have tried to learn French but it is a struggle. My kids are in French Emersion for school and beyond about grade 3, I lost my ability to help with homework because I can’t read the instructions.  I have tried to visualize myself going to a foreign country and having to learn in a language that was not second nature for me. I have great admiration for anyone who can speak fluently in more than just English. I expect it is a great sign of intelligence. These are my biases. But I have to assess student writing ability nearly daily in my work and I’ve put a lot of thought in how I should address the writing problems of all students.

ESL students, EAL student, L2 students, whatever we are calling them these days are NOT worse writers than our domestic students. It almost feels like a big sigh of relief to get that out.

I’ve been doing a lot of reading lately since I started my series of research studies looking at writing self-efficacy in nursing students. The body of literature on writing ability in undergraduate students focuses mostly on generic undergraduate groups, or L2 specific groups, and it is rare for researchers to separate out their ESL students from their domestic students and make a comparison. I’m not sure if the lack of comparative studies is because researchers think the comparison is unnecessary or if it is because there may be an inherent assumption that ESL students are worse writers than our domestic students so no one bothers to actually test and see if it is true. In the writing self-efficacy research literature I’ve only come across one study (Williams & Takaku, 2011) that overtly compares these groups.

Certainly in my own teaching experiences I’ve witnessed a number of things that I find disturbing. These experiences span my entire career, and not just my teaching career but also my student life and have been witnessed at more than one institution; many instructors believe that ESL students in English language programs have poor language skills.

I’ve never considered myself an ESL instruction expert and I once considered this a deficit in my ability to teach writing. So one year, after I’d been teaching academic writing to nursing students for about 6 or 7 years, I attended a workshop geared toward understanding the needs of ESL students, in particular with regards to issues of plagiarism.  It was a wonderful workshop. I learned a lot and we had some great discussions as a group. It changed my perspective on how plagiarism should be viewed.

But I also walked away from that workshop wondering why it was labeled specific to ESL students. Everything that was discussed, I had also seen with our Canadian born students.

So in my studies, I decided to ask the question. Do self-identified ESL students have lower writing grades, lower writing self-efficacy and higher anxiety than non-ESL students? Here is what I’ve found so far, bearing in mind this is the experience of one educational institution, a college in a prairie province in Canada that has, up to this point, admitted nursing students with a minimum 60% average and tests students for language ability prior to admission for a minimum standard (Degrees of Reading Power score of 75 minimum).

And yes, I have ethical approval for this work.

Here is a table from the first set of data, that I’ve clipped from my notes about my findings. Comparisons were made between ESL and Not ESL students for paper grade for the first year course (Paper %), final writing course grade (Final %), Writing self-efficacy at three time points (WSES), an APA and grammar knowledge test (APA/G), and state and trait anxiety . The findings were so unremarkable that I didn’t even fill in all the slots on the table:

screen-shot-2016-11-10-at-2-55-21-pm

Two years later I did a follow-up study with the same group. Yesterday I ran these results. The sample sizes are smaller and the means more variable as a result, but the results are again showing no difference between ESL and non ESL students on the following measures:  Paper grades for year 1, 2, 3, three clinical practice scores, GPA, the Degrees of reading power admission score, writing self-efficacy measures using two different measures, my own — which will be published in the spring — and the Post Secondary Writerly Self-Efficacy Scale, and anxiety using a visual analog scale.

In addition I ran a chi square examine ESL and year in nursing program. You see, in the follow up study, all these students were supposed to be in third year but many of them had fallen out of synch. I thought it might be worthwhile assessing if there was a difference in progression between self-identified ESL and non ESL students.

screen-shot-2016-11-10-at-3-38-34-pm

The numbers are small so view with caution and the second year ESL count violates the assumption of at least 5 observations per category, but it appears that there is no difference in proportions of ESL and non-ESL students who fail to progress on schedule within the program (X2 = .253, p = .71).

Do ESL students struggle with writing and language? Yes they do, but so do all the other students as well. Williams and Takaku (2011) had a similar finding — in fact, they found that over time and with help seeking as a mediator, ESL students would out perform the domestic students. Domestic born students, in qualitative research, also acknowledge difficulties understanding academic and research language. I hear complaints about difficulties with language every day in my classroom from students of all shapes, sizes, colours, genders, ages, and voices.

So what does this mean for how we approach students and writing in groups of mixed ESL and Canadian born students:

  1. Many students struggle with writing. The elements of writing they struggle with and the reasons for their perceived struggle may be slightly different but they struggle, and many also approach writing anxiously and question their ability to write — some more than others, but it has nothing to do with first or second language status.
  2. Having an accent and a foreign name does not mean one is a poor writer or that that individual has low writing self-efficacy. Not having an accent and having a North American name does not mean one will write well.
  3. Treat each student who asks for help as an individual and address their individual needs and concerns.
  4. Be conscious of unfounded assumptions or biases about any particular student for any reason.

In our program before admission, every student completes a language test and has to meet a minimum standard. It is OK to acknowledge that you have trouble understanding a student’s accent. I have a colleague who is researching internationally educated nurses and those nurses acknowledge they have trouble with accents as well. It is not OK to hear an accent and assume that is a signal for a weak student and a weak writer.

Oh, and if you ever need to give an example of when statistically non-significant findings are important…..

 

 

On Using the Media to Teach Research Writing and Critical Analysis

When I first started teaching research methods, I would have told you that having an academic writing assignment in my course was a bad fit. The only writing assignments that I had ever heard of being done were an article critique or a research proposal.  A research proposal was above an undergraduate level and an article critique got them to think about the strengths and weaknesses of research but only apply that knowledge to one isolated study.

However when our program was redesigned, my research methods course was updated to  include teaching the concept of scholarship in nursing. While I wasn’t looking to steer undergraduates toward publication by asking them to write, I did want to get them deeper into the scholarship of research. I had been having them analyze portions of research articles since I began teaching the course, but wanted to push it a step further and have them begin to critically analyze a small body of research on the same topic.  Imagine that! I wanted them to actually read research in a research course.

The First Assignment Plan

My first academic writing assignment plan asked them to find 5 primary studies on a researchable topic of their choice. I told them it was preferable if they chose 5 qualitative or 5 quantitative studies and not try and mix and match the paradigms, although I would make exceptions depending on topic. Hint: it really only works if their quantitative studies are purely descriptive. Undergraduates trying to compare the results of 4 randomized trials and a qualitative study is disastrous… one of these thing is not like the other.  They had to provide a background, nursing significance and explain the gap in knowledge on their topic as indicated by their 5 studies (and other sources as required). A compare and contrast of the findings of the 5 studies and a discussion of study limitations as indicated by the authors of those studies, was the bulk of the 5-7 page limit requirement. Their conclusion to their paper was the purpose statement for a future “hypothetical” study.

What followed, and ultimately what got me to modify the assignment, were 4 appendixes describing portions of a possible future study. Replicating a study in their pile was allowed but “difficulty points” (ultimately meaning I was more forgiving if they made mistakes) were awarded to students who developed the next steps in the body of the work, corrected the flaws in the studies in their pile, or developed a new intervention. I told them they had unlimited funds which resulted in more than a few brand new exercise facilities built.  The appendixes were to describe their study method (A), their sample (B), procedures for ethical protection of their participants (C), and to find and discuss the characteristics of one questionnaire  to measure a one variable, if their study was quantitative, or develop 4 open ended research questions if qualitative (D).

What happened was amazing. The students developed a great understanding of the methodologies they were writing about and using for their hypothetical study design. But the flaws were also huge. The ethics section became repetitive from student to student and the only students who did well on the questionnaire section were the ones who came to see me about their paper and we sat together to identify its characteristics and find reliability and validity information.

And there was one other, not insignificant problem — the assignment was labour intensive to grade. Each paper was 10-15 pages long including the appendixes.

Modifying the Plan: Research Versus the Internet

The decision to modify the paper was as a result of reading this Deitering and Gronemyer (2011) article describing the importance of getting students involved in the public discussion about research. Their arguments about why students should be examining more than scholarly work were compelling and I made them discipline specific by recognizing the following: Students have a need to understand the literature that influences public debate and the opinions of their patients. They are also in need of understanding the difference between a published study and a blog post/newspaper article related to the published study and be able to explain why the former is a more reliable source of evidence.

Let me state the obvious — obvious to us academic-types anyhow — the media gets health information wrong much of the time.  Media articles are rarely written by knowledgeable health care professionals or researchers and are instead written by journalists with no medical training who may never read the full research study but produces an article for their publication based off a press release which may give an incomplete picture. For example, this popular media story on wine drinking and its equivalence to an hour worth of exercise was conducted on rats who drank the human equivalence of 100 bottles of wine to show the effects the media article claimed.  The media report was so much fun it was emailed among my colleagues and we giggled about it wishing it were true (and later that term, a student tackled this topic for my scholarly paper assignment). Sigh… back to the treadmill.

I didn’t want to lose what I had observed to be the best part of the assignment — sending students off to search for 5 primary studies on the same research topic. While settling on 5 as the appropriate number of studies to locate was somewhat of a guess, it proved to be exactly the right number to be challenging for undergraduates, yet possible for most every topic. Some students were able to find more than 5 and have the flexibility to exclude an article that didn’t fit well with the others. Other students struggled to find that 5th article and developed some clever research skills as a result.  The requirement that they show me their 5 primary studies to ensure they were using the correct sources got me face to face with every student. I heard their stories of why the topic interested them. We had conversations on making the articles fit together for easier critical analysis. To this day it surprises me when grading the papers, how much I’ve learned about each student simply from these 5 minute conversations.

While the option to choose their topic based on the inspiration from a media, blog or other internet source remains the most popular option, two other routes to inspiration have also been observed as successful adaptations. Students can also describe a clinical experience they had where the practice they observed differed in some way from the practice they were taught — for example, alcohol swab versus no alcohol swab prior to glucose monitoring.  Personal experiences with the health care system have also been addressed such as a student who gave birth to a premature baby who felt her husband was ignored during kangaroo care. The point is to describe a practice, or relate what the media is saying to influence their readership, and see if the research literature matches the message. It works well with qualitative article sets as well, for example the media article of inspiration can be any first person narrative of their experience with a disease or other condition related to health.

The Assignment Guidelines

In brief, each student paper must contain a discussion of the following:

  1. Background to their research problem.
  2. Significance to nursing. Students are more than welcome to use research from other disciplines to support their research question but they have to explain (and cite) why nurses should care about that problem.
  3. A summary of their media article, clinical scenario or personal experience.
  4. Compare and contrast their 5 primary studies in terms of their research findings.
  5. Discuss the strengths and weaknesses of the research studies reviewed.
  6. Discuss how the research findings compare with the media description (I’ve had students integrate this requirement throughout their compare and contrast and that works also).
  7. Practice implications. In what way, if any, should practice change given what the research says?
  8. A table summarizing their research studies with author, title, independent variable, dependent variable, population, sample size  for quantitive studies, and phenomenon of interest replacing the IV and DV in qualitative sets.

Students are able to construct their paper in the order of their choosing and in whatever way feels logical for their topic.

Evaluation

This assignment has been successful beyond my wildest dreams. Student engagement has increased. I get the chance to speak to every student about their topic which meets my requirement of looking them all in the eyes during the writing process. I strongly believe that having every student connect with you during the writing of their paper reduces the likelihood that they will plagiarize. By having a quick review of their primary studies, I also save myself a tremendous amount of time while grading because I know I am reading a paper amalgamated from the correct type of literature.

In the second year of implementation, due to feeling I was doing a little too much handholding in helping them identify qualitative versus quantitative research, I created a document which outlined the characteristics of various published articles (primary quantitative, primary qualitative, research protocols, review articles, discussion articles, and other). The students are now required to review this document and fill in a log sheet which attempts the article identification prior to approaching me to approve their articles. I no longer identify what their articles are for them which allows me to spend more time exploring how well their articles fit together as a package.

Implementing this paper is time consuming. The mini meetings I have with them about their topic take up all my mid class break time and I usually require another 20-30 minutes immediately preceding or following class to attend to every student who wants my attention on a given day.  I have to teach them how to synthesize their results and thematically analyze the themes from qualitative studies. But the upfront time I put in prior to the paper due date, is time I don’t have to spend frustrated with poorly conceived assignments while grading.

If you are interested in modifying the assignment for use in your own courses, you have my permission. I believe this assignment can work in multidisciplinary contexts. Just send me a tweet to let me know @academicswrite and I’ll be happy to answer questions.

Let’s Chill Out About Plagiarism: Yes, I Just Said That.

How do you decide what is a serious enough offence to call plagiarism? Where do you draw that line?

The day Melania Trump plagiarized Michelle Obama’s speech at the Republican National Convention was one of the best days of the summer, and I say that, as a Canadian, watching the whole American election circus from afar.  It was a good day because the world was talking about plagiarism and where you draw the line — something I have to think about nearly every day as an academic. They were talking about what defines plagiarism, in particular, but more importantly they were talking about how plagiarism is about the context in which it is committed.  I spent my time on social media that day rubbing my hands together with glee.

My title to this blog is not in any way to suggest that plagiarism is not to be taken seriously. I have seen plagiarism in action. I’ve had students hand in annotated bibliographies that are word for word from the abstracts of the published articles.  I’ve seen students submit identical or nearly identical assignments. I’ve known of a student who stole another student’s work, unbeknownst to the student victim, by copying their paper from a computer when s/he left the lab to use the bathroom. I’ve marked countless papers with vast amounts of uncited information. I had two sisters hand in a paper on the same topic with the same reference list with identical errors, identical headings with different writing under each paragraph, but the citations in basically the same order. The latter was likely a case of inappropriate collaboration — or one sister wrote both papers — I will likely never know what went down there but it wasn’t honest writing.

This recent Chronicle of Higher Education article also concerns me. While I have no doubt that there are people making big money off student cheating, I refuse to believe that every student is doing it (otherwise these businesses could never submit an assignment on time, they’d be so busy). Maybe I’m naive. But for students who are hiring others to create custom assignments for them based on a course’s assignment guidelines, it would be nearly impossible, short of a confession, to catch it. I’m not going to waste time hunting these students down.  In a practice based profession, I’m hoping they’ll manage to fail themselves out in other ways.  A now retired colleague mentor of mine used to always say: Do we want spend our time educating, or policing?

Over 10 years ago when I was a junior faculty member, a couple of more senior instructors decided to become marking zealots. They began pulling students’ sources as they graded and searching for every point the student made in citation and low and behold they found many situations where students had copied word for word from sources.  Should I be marking this way too? I wondered at the time.  I could never bring myself to do it. It felt wrong, to me, to grade papers with this kind of mindset.  It felt like viewing students as guilty until proven innocent. It felt like a witch hunt.

My preference for identifying plagiarism was, and still is, to rely on the subtle signs: poor awkward writing and grammar in this paragraph, high level language in the next always sets my alarm bells off. But there are others too: font type or font size changes, hovering a cursor over the text and seeing web links, misuse of pronouns, change in verb tense, change in person voice from third to second, and the more obvious, finding another student’s name mistakenly left unedited somewhere in the document. I frequently, when grading, pull sentences out of papers and drop them into Google just to be sure. I’ve never used Turn It In. I believe there is a fee to a program to access it and in these times of fiscal restraint, my department has just said no.

Uncited material that should have been cited is the most common type of “plagiarism” I see. About 10 years ago, we, meaning a sub committee in my department, set rules about when we would call it plagiarism — 4 or more incidences of a missing citation would be considered plagiarism.  We would make the student take their paper back and add the citations. They would get a form filled out and placed in their file.

I’ve since had instructors, who weren’t around at the time we made the decision, say to me. Students will purposely leave three uncited sections in their paper because they know they can get away with it.  What!? Wouldn’t it be easier to put the citation? It is way too much work to be that deliberate in your purposeful non-citing.  Students who haven’t cited, really, haven’t thought much about it at all.

I would sit with these students, often facing their tears and soggy Kleenexes, in my office and point out the error of their ways and they would stare at me blankly not really sure what I was asking them to do when I asked them to fix their citation. I don’t think I took that from anyone, some of them would say in many varied ways, I just wrote that because I know it.  I heard it in class. 

And my thinking started to change. Most, if not all uncited work I saw in undergraduate student papers had nothing to do with intentionally or maliciously trying to steal the work of someone else to, I dunno, try and look smart? Most of it was as a result of poor research or not understanding the value in finding a source to support their rationale or argument they were presenting. I stopped asking them to go back and fill in their missing citations because most of them didn’t have a citation to fill in.

And besides, what is an “uncited section?” A sentence? A paragraph? What if they just put the citation at the end of the paragraph? Does that mean that everything earlier in the paragraph is uncited? Or did the one citation intend to cover the whole paragraph (insufficiently)? Now I just tell them it isn’t sufficiently cited and I move on.

And I started teaching my graders in first year to notice the signs of writing that was just a student rambling off the top of their head as if they were expert enough to make the point. Often those paragraphs were full of simplistic thinking, grand sweeping claims, and non specific statements:

Alcohol is a big part of our society. For young adults alcohol can be a big struggle when it is being introduced into their lives. It is very common for people to turn to heavy drinking, also known as binge drinking when trying to destress and have fun [sic].

This paragraph was the first 3 sentences of an introductory paragraph written by a first year student. There is so much wrong with it. It’s vague — how many people in society? What people? Who says there is a connection between binge drinking and de-stressing? And where are the citations to support the accuracy of these claims? The comment in the image to this blog, is what I had typed on this student’s paper.

Is this plagiarism? No. This is a student writing what she knows about binge drinking off the top of her head, maybe as a summary of what she read or from her own personal experiences with it.  Her following paragraph(s), detailed and cited many of the points made here. So my feedback to her was to delete the whole first paragraph. It wasn’t plagiarism. It was poorly written and redundant. A waste of words in a 3-page paper.

I once dealt with a suspected case of plagiarism where the student had copied one paragraph of her paper from a website. Her paper was about urinary incontinence and the one paragraph, maybe in the 50 to 100-word range, turned out to be a paragraph from a website about fecal incontinence. She didn’t copy the paragraph word for word. She did make some inconsequential changes so there is no doubt that what she left unchanged was somewhat deliberate, or at the very least, lazy.  On top of it all, she put an incorrect citation. The citation she put was not the website she copied from. She also had a couple other spots in her paper where she had used or poorly paraphrased 5 or 6 words in a row from one of her sources. Her grader had even highlighted spots where she had used two words in a row from a source.  The vast majority of her paper, however, was paraphrased. The intent to paraphrase was clear in every spot but this one place. From reading her work, my sense was simply that she didn’t give a damn about the assignment or the quality of what she was handing in. Maybe she spent a couple hours writing the paper and submitted it as it was. Maybe she wrote it at at 4 a.m.

Not academic misconduct, I told my team leader and the instructor.  Arguments ensued.  The one paragraph, the one from the fecal incontinence website: Bad. Stupid. Dumbass even.  The student deserved to be hauled in and given a good slap on the wrist. Certainly a reduced grade was in order.  She didn’t deserve to have academic misconduct stamped for life on her transcript.

To be honest, if I hadn’t been cued to check. I doubt I would have caught that case of plagiarism to give her that slap on the wrist. I’m not sure, and I’ve wondered since, if that matters. The paper was very poorly written and not cohesive. It was a difficult read. A poor grade would have been the outcome anyway. I’m not sure that the instructors, in this case, would have found that paragraph either except that conditions made them check. The instructors had found one incidence of plagiarism in one student and decided to go back and check the entire class, as if one case meant a giant conspiracy to pull the wool over their eyes, to take them as fools, was underway.

Here is the crux: if you go looking for something. You will find it. You may even begin to redefine it, to prove yourself right. To solidify your case. In order to indict someone, no stumble, no matter how minor, can be left unmentioned. Finding one sentence that is 90% copied from a source suddenly then means that every time the student uses the same terminology as an original source, even medical terminology which shouldn’t be changed, you will mark it as plagiarism. Suddenly, one word, two words, three words in a row, can count as plagiarism — if you have declared it should be so and if it justifies your anger over that one sentence. You may even feel a little giddy while doing this because you’ve caught them. Now you can solidify your case that plagiarism is rampant among students and the student with the one sentence or the one paragraph is now a villain. Call yourself a hero now, maybe.

But I call it witch hunting. Policing. Confirmation bias.

I found this sentence in one of my student’s papers last year: how powerful losing weight can be. Six words in a row from the original. Common words, common idea. I highlighted it in the student’s paper and left it. The rest of her paragraph was well paraphrased. Put that phrase into google with quotes around it and the student’s source pops up as the first item along with 2 other media sources that picked up the article. Not plagiarism.

This scenario, which came under discussion on Twitter the other day, however, is very much plagiarism.

Melania Trump likely used less words of Michelle Obama’s 2008 speech than my fecal incontinence student used in that one paragraph. Did Melania Trump plagiarize? The bulk of her speech was “her own” or written by hired speech writers, so at the very least original in wording, if not original thinking. You bet your buckets she plagiarized.  Melania’s words were being delivered to millions with the intention of influencing them. She stole the words (and one can argue, the values) of another woman making a speech in the same context 8 years earlier and those words had helped Michelle Obama’s husband get elected as President of the United States, something Melania, perhaps, also desires for her husband.

My student wrote her paper for an audience of one in a learning environment. Her teacher. Context is everything.

There are so many reasons why a student may paraphrase poorly or not cite.  Is it really helping the student with this occasional mishap or bouts of lazy writing to crucify them with the label of academic misconduct?  Or would talking to them and understanding why it happened help you understand their writing fears and their feelings of low self-efficacy and the fact that they are just trying to survive a gruelling nursing program? Maybe what you have instead is a teachable moment? I’m a great believer that finding ways to look students in the eye while they’re in their paper preparation phase is the best plagiarism prevention of all. Some “plagiarism” is just bad writing that needs to be corrected. We need to be sure we know the difference.