Student Peer Review Process? Here’s My Version

When I tweeted yesterday that I was creating a peer review process for my students to assess each other’s writing and provided a sample of what I was up to, I had many requests to share the whole process.

I am happy to do that and will do so here in this post with two caveats:

  1. This process is untested. This means, I can’t even give you anecdotal feedback as to if it works. So for now I will share the materials but it is probably wise to warn you it may backfire. Conversely, it may also be brilliant. I have never tried peer review before but I’ve spent about a year dreaming up this process. I really wanted to do double peer review, meaning I really wanted students to peer review two of their classmate’s papers but as I tried to schedule it, I recognized in our compressed term, there just isn’t enough time to do that and still allow for the time to do research, the synthesis exercise, and the peer review process. I’ll write a follow up blog after I have trialed the process.
  2. Remember that all writing is contextual. I’ve created this process specific to the needs of my assignment, my course content, and my teaching style. If you want to model what you do after my process, you won’t be able to simply lift my points and my language choices and have a neat fit. Make it your own.

Here is the document describing the process:

Qualitative Paper Assignment

In the document you will find:

  1. My detailed assignment guidelines for the qualitative synthesis paper.
  2. A description of the peer review process.
  3. My content rubric for the paper (out of 33)
  4. The peer reviewed checklist for the students to help them assess their classmate’s work.

As this is an assignment structured using scaffolding theory, these are the basic activities to scaffold the process:

  1. As the course I teach is a research course, I start the term introducing them to the principles of qualitative research. As I teach them about qualitative methods, I am simultaneously connecting the theory content to what they need to write about. They also get a class on searching the literature and what needs to be included in the background to a research study.
  2. I have the students find 5 primary qualitative studies on a topic of their choice. They have to show me these 5 primary studies. There are numerous benefits to this requirement, most of them, selfishly benefit me: a) I ensure that they write this paper using the literature I asked for and not just any old literature. It saves me so much grading time to check this in advance; b) I get to look each one of them in the eyes at least once during this process. For some students, the article checks are the only conversation I’ll have with them all term; c) the students can write their paper knowing they won’t lose marks for incorrectly identifying qualitative studies.
  3. I provide them with an article identification guide to help them distinguish qualitative, quantitative, and discussion papers. Article Type Identification Guide
  4. I have them read their articles next and then we come together and do a synthesis exercise over one class.  I wrote a tweet thread about that exercise a while back (see below).
  5. The students will go off and write and before the planned peer reviewed process they would hand it in to me and I would grade it. The next steps are steps that are untrialed.
  6. Now students will hand in a preliminary paper and share that preliminary paper with a pre-selected classmate.
  7. Each will use the peer review checklist to independently comment on the other’s paper. They will also be welcome to correct grammar and APA.
  8. I will hold a peer review class where they can ask me questions but the goal is also for them to sit down with the rubric for the paper and jointly come up with a score for the content portion of each of their papers. Two chances to practice with the rubric.
  9. Notice that the rubric and the peer review checklist are colour coded. That’s because the colours that match on both documents are the items on the peer review checklist that they will be considering when scoring that section on the rubric. The rubric is broad areas of writing (content, synthesis, research, mechanics) the checklist is ordered by the required heading sections of the paper. Hopefully this will help them realize what expectations affect what sections of the rubric.
  10. After the peer review process, the students will go off and revise their papers based on their discussions. They will hand in a final draft of their paper to me along with a self-scored rubric for their own content.  Their grade will be the score they give themselves — unless they score themselves too low then they will get my score. If they score themselves too high? Well that’s a bridge I’ll cross when and if it happens.
  11. I am not having the students grade each other or assess themselves on APA. They’ll obsess over it and it will take away from what I want them to focus on, which is their content. I’m going to grade their APA.
  12. 50% of their entire course grade goes to the little pieces that make up this assignment. The breakdown of the assignment in terms of value is:
    1. Content of paper 33/
    2. APA 3/
    3. Participating in synthesis class and making progress 3/
    4. Article checks 1/
    5. Peer review 10/

 

How I will score their peer review exactly is still a work in progress and I’m still reflecting on it as I don’t have to “grade it” for a couple months yet. It will be a combination of participating in the peer review, collegiality with peer and being a good citizen and getting documents in on time, a thorough effort at feedback to their peer, the degree to which they pay attention to the feedback they were given in their next draft, a one paragraph response to that feedback, and realistic self-assessments on the rubric.

I’m blessed this term with an abnormally small class which will be extremely useful for me to trial this process and discover the weak spots in my pedagogy.  I hope, those of you who were interested also find this useful. Please feel free to comment below with questions or tweet at me to discuss, or share your own peer review processes.

Part III: How Much Academic Rejection can a Girl Take?

If you have come across this blog first, this is Part III of a 3 part blog looking at the story behind the research which led to the publication of this study:

Exploring Self-Efficacy and Anxiety in First-Year Nursing Students Enrolled in a Discipline-Specific Scholarly Writing Course

Part I explores conducting the study.

Part II talks about what came after.

Part III (this part) will explore the perseverance required to get this thing published. The ending has already been spoiled but the journey is what matters. It was an adventure that turned out alright in the end.

I approached publication as go big or go home. We should always start that way shouldn’t we? I went to one of the most highly ranked nursing education journals and attempted to submit the paper as “original research” ignoring the fact that their author guidelines said they wouldn’t publish single-site studies in this category. In all fairness to my dumb decision, I had gone and flipped through the journal and it did appear that they regularly broke this rule. No shock, it was desk rejected. But they did ask me to revise the manuscript and resubmit it as a “research brief.” I cringed. To fit the “research brief” category, the manuscript, which was 15-pages, would need to be cut to 8-pages. I debated not doing it but I went forth anyway. I was ruthless. I gave myself five seconds to make a decision on a passage and if I hesitated it was gone. I actually think I ended up with a pretty decent short manuscript. I sent it back to the journal. It went to review and two months later it came back….. rejected. The feedback: manuscript didn’t go into enough depth in a very long list of areas.

I was pissed because I had been asked to resubmitted it in a category that denied me the space to give those details and it got rejected for not saying enough. No shit. Perhaps if I had been more experienced I could have foreseen that outcome and declined their resubmit request.

In the meantime, I had continued on with my reading. In 2013 before I started preparing this manuscript I had decided to run my study again, this time with a time-control period prior to the course start, so I was already working on another study.

(In fact, I got the requested minor revisions for the manuscript for this second study last week so that one will be coming soon as well.)

But all the new reading was developing my understanding of self-efficacy theory.  I decided to re-write some of the review of the literature and discussion. I liked some of the edits I made to shorten the manuscript so some of those stayed, in addition to expanding on some of the literature. So if you are keeping track, at this point in the process, I am re-writing this manuscript for the second time.  It was written, it got re-written to shorten it, and now I am re-writing it in preparation to go to another journal.

In September 2015, I submitted the paper to a second journal, where it sat. And sat. And sat in review forever and a day. I would email the editor every couple of months to find out how it was going. She was a lovely lady. Her sympathetic emails back to me always stated she couldn’t seem to get anyone to follow-through with reviewing the manuscript. At some point she had found one person. She really wanted three people to review it.  Seven months later when I was getting the same response (reviewers not responding), I asked her if I should pull the manuscript and send it elsewhere. And she agreed that was appropriate. She sent me the reviews she had — there were 2 completed at this point, much to my surprise — and they were unenthusiastic reviews. They thought the review of the literature was not great. They thought the study was weak. They thought I was good at hiding the study weaknesses with editorializing (truth). I think she allowed me to pull it because she would have ultimately rejected it. It was now spring of 2016.

One of the reviewers told me that I needed to team up with a mid-career researcher to gain some experience.

What the hell? I wrote to Tom in an email. How can they tell?

They looked you up. He wrote back. And if they had looked me up they would have found no publications. All my previous publications were under my former married name.

It was supposed to be a BLIND review. 

I was still reading. In 2015-16 I had been working on a third big writing self-efficacy project exploring all the measurements instruments assessing the concept.  By the time I pulled this study article from the second journal, I had written that manuscript, I had written the manuscript for the replication study with the time control. I dove into re-writing THIS manuscript, especially the review of the literature for the third time. In fact, it seems to me I wrote the new review of the literature for this publication just after I wrote the first draft for the second study manuscript which in the end was probably fortuitous because it meant I was forced to write two different reviews of the literature. I used the same literature to write two different versions of the same background and review of the literature. Those of you who do a lot of research using the same concepts will know what I mean when I say, this is hard.

I sent this manuscript off to the new Canadian Association of Schools of Nursing (CASN) journal Quality Advancement in Nursing Education. It is open access and publicly available but there are no fees. It is peer reviewed but I doubt it yet has an impact factor as it is too new. I’m also not sure if it is indexed yet. I just tried to search for it in our library and it didn’t come up. The journal has been in existence for just over 2 years. I believe this is its 7th issue.

But they were wonderful. The reviews were wonderful and helpful. One reviewer asked me to write a section on scaffolding theory and include an appendix with the scholarly writing course structure and activities. I nearly screamed in excitement. I had already done a ton of reading about scaffolding theory because I had written it into the second study’s manuscript and allowing me space to publish my entire course structure was a gift I would have never gotten from any other well established journal with their strict rules and tight word counts. So now the final product comes in at a whopping 21 published pages. Not including references, tables, or the appendix it is 6500 words long — unheard of lengths for nursing journals.

Incidentally Torrie and Holly were at the beginning of the second year of their nursing program in October 2014 when they wrote their papers for my class on the topic of writing interventions and writing self-efficacy and anxiety. With all the revisions and re-writes this manuscript went through in the time since, there are very few of their actual words left. There is a spot here and there that I remember not being my writing but I can no longer remember what parts were theirs.  They graduated from our nursing program in the fall of 2015. They’ve been practicing nurses for about a year and a half now. They are both wonderful gifted students. It makes me very happy to have given them this opportunity even though I know the whole experience hardly feels like it belongs to them anymore. Funny thing is that I was involved in a project with my professors when I was an undergraduate student that was eventually published. I hardly feel like I played any role in that project either anymore but there it sits in my publication credits. So this paper for me was like giving back to Torrie and Holly  what one of my profs gave to me as an undergrad and I am forever grateful. It may be one reason why I am where I am today.

And so concludes the story behind this research. I look forward to your feedback and I encourage you, if you are a blog writer, to consider telling these stories of your work.

If you missed Part I and Part II they can be accessed at the hyperlinks.

The Evolution of My Writing

I used to write my papers on an old Hewlett-Packard word processor.  It had a little tiny memory disk that was about half the size of a hard “floppy” disk, although I don’t think those existed at the time. It was a Christmas gift or a birthday gift from my parents during my last year of high school. I used it through my first 5 years of university, so about 1989-1994.  It had a little tiny computer screen which allowed me to see about 6 lines at a time.  It could be used like an electric typewriter as well. But I liked that I was able to save my writing to these little disks and when I wanted to print something, it auto typed it out like a regular typewriter would at about 80 to 100 words per minute.

Back then I always hand-wrote the first drafts of my papers.  I liked to be able to see a page worth of work at a time and manually flip back and forth between those pages. One hand written page, roughly, worked out to be just short of one double spaced, 12pt font, typed page of writing.  After I finished the first handwritten draft (where I edited little), I used to type the paper into the word processor and then do all my editing there.  The act of typing out what I had written allowed me time to reflect upon some of that first draft uglies and where I needed to make global changes.  Thus, I’ve always believed that the real writing happens in editing.

I wrote my papers using this process through two university degrees and continued to handwrite rough drafts even when I retired the HP and began using Microsoft Word on a PC. Then sometime during the first year of my master’s degree I started writing papers directly into the computer. Papers got harder to write, at that level, and I found I couldn’t write anything in any planned order any more. Also, with 40 references to incorporate, I could no longer write an entire paper in a single day. It was easier to put a heading into a computer file and leave a blank than it was to guess how much of a blank to leave on looseleaf to accommodate what needed to be said.

I presented at a conference this year and while it really had nothing to do with my presentation, someone asked me if writing processes had changed in the era of computers.  I had been talking about a well-known writing process theory that was developed in the 70s that still seems very relevant today, even though fewer people likely handwrite their drafts than would have been common back then.

Now that I write solely on a computer, I still write a pretty rough first draft  but I edit as I write more than I used to.  Autocorrect helps with my clunky spelling skills. But it is still always my goal to get a sketch of what I want to say down in writing and then work my magic while editing. I consider my first drafts so rough, that I would never show a first draft to anyone…. not even co-authors. I always like to let a manuscript germinate for a while. A week is awesome, if I have it, but 2 days is often enough before I tackle the next draft.  I spend the time between reflecting on what I wrote and things I forgot to say.  I look for ways to solidify my argument or make connections between points I may not have seen the first time around.

I’ve heard of these strange creatures who obsess over every word in a sentence and write a clean rough draft that needs nothing more than a polish.  I am not one of them.  And to be honest, I don’t envy them. It would be painful writing that would make me feel like the end was so far away. I believe the key to being prolific is to spew it all out in one shot as quickly and as horribly (or not so horribly) as possible. If you aren’t solid on your thinking in some areas, what you write later may be a clue to how to fix it. I’ve written articulate phrases like ….. and blah blah blah prediction …. into my papers just to remind me what I need to learn more about when I come back and edit that section. Sometimes you have to stop and read more to figure out what comes next.

And that is one thing that has never changed about my writing process. I love revisions. I have always worked better building off an existing idea than staring at a blank page hoping  that words will come. A done first draft feels like a finished paper to me because I know how it begins, I know how it ends, and now I can fill in the holes with what I believe to be the best part of writing: rewriting.

Having said that, you need to write the way you write.  It isn’t wrong to be a multi draft writer. It also isn’t wrong to write each sentence as if you’ll never have to change it.  Just write.

*Image: some really bad fiction writing I did when I was about 20. I remember going back and doing the highlighting. It was the only parts I liked.  (My attempts at fiction, back then, never made it to the word processor phase of writing).