OIA Blog

We now have a Pre/Post Test Development tutorial available on our tutorial page.  OIA is happy to assist SCAD programs with redesigning their pre/post tests.

Currently, SCAD programs are required to implement pre/post tests in all the required courses of a BFA major.  Because OIA has focused its efforts on updating program goals and outcomes this year, it’s possible that the revisions to the goals and outcomes will encourage updates to the pre/post tests as well.

If you would like to update your pre/post tests, please review our tutorial for an outline of best practices with regard to both writing, designing, and implementing pre/post tests.  In the mean time, here is a short summary of pre/post test best practices:

Ideally,  pre/post tests……

• should measure essential features of the course content (do not use the pre/post test as an opportunity to measure trivial or specific course information)
• should NOT vary by instructor.  Because pre/post tests should measure essential content of a given course, it’s possible to rely on one version of a pre/post test for a course.
• should be designed for efficiency and accuracy.  There are strategies for easing the burden of the pre/post test process for faculty, and there are strategies for improving the reliability of the test for measuring student knowledge.

Here are some suggestions for improving efficiency and accuracy:

- Follow a multiple choice format (if appropriate for the content of the course).  Because there is only one correct answer to a multiple choice question, grading a multiple choice test takes less time than grading a format that allows for variations on the right answer.  In addition, because there is only one correct answer to a multiple choice question, a multiple choice format is reliable for measuring student knowledge (assuming the test question is written well – see multiplechoice_Tutorial_Final for tips on writing multiple choice questions).

- Rely on the pre/post test as a grading opportunity.  It is possible for a pre-test to be used for course participation points, and possible for a post-test to be used as a quiz or test opportunity.  I have even heard of faculty who write a pre/post test, and then instead of implementing the pre/post test separately from other grading opportunities, they incorporate half of the pre/post test questions into the course midterm and half into the course final.  This method is great because it 1) reduces the number of times a student has to take a test in the course, 2) ensures that students will take the pre/post test seriously, and 3) ensures  that the midterm and final reflect the essential features of the course.

-Redesign the test for electronic grading.  If students take the pre/post tests online or if students complete the tests using scannable answer sheets, faculty will not have to manually grade the tests.

-Pay careful attention to when the pre/post tests are implemented. Students might be more likely to take the pre-test seriously if it is implemented in the second week of the quarter compared to the first.  During the first week of the quarter, students are still adding/dropping courses and getting acclimated to the new quarter; thus, it might not be the most reliable time to implement the pre-test.  Plus, if you implement the pre-test during the second week of the quarter, you avoid testing students who drop the course after the first week.  On the same note, post-tests might be taken more seriously if they are implemented earlier than the tenth week of quarter.  Students are less likely to suffer from end-of-quarter-itis during the eighth or ninth week of the quarter.

 

The above suggestions will hopefully ameliorate concerns faculty have about the quality of their pre/post tests; however, OIA also understands there are concerns about the value of pre/post tests since many programs do not rely on the data to make evidence-based program decisions.  To ameliorate this concern, OIA will be happy to supplement a program’s annual assessment of capstone data with an evaluation of pre/post tests at the program-level.  Please contact your respective assessment coordinator to discuss this possibility in detail.

We now have a COURSE scoring guide tutorial available on our tutorial page.

OIA is happy to assist individual faculty members develop course scoring guides.   A scoring guide is a simplistic, visually organized way to measure performance on a set of tasks. Scoring guides can be designed to measure performance based on a Likert-scale, checklist, letter-grade, or other point-value system.

Faculty Benefits.  A course scoring guide may simplify and standardize the process of grading more time-consuming projects, such as research papers and design collections.  For instance, when I used to grade papers, I loved to provide students’ with a lot of written feedback – regardless if the paper was of ‘A’ quality or ‘C’ quality.  After taking a considerable amount of time to write out my feedback, I felt that I needed to read and contemplate the paper all over again for the purpose of grading it.  Plus, I was concerned about grading all my students based on the same criteria. This is where a scoring guide would have helped me out.  I could have created a scoring guide that assigned point values to the essential features of the paper.  That way I could keep track of the points accrued during my first reading of paper. Scoring guides are also beneficial for noticing patterns in students’ abilities to successfully complete assignments.  For instance, you may notice that students are consistently scoring lower on ‘layout techniques’ than on ‘content summaries.’  This may suggest to you that more class time should be devoted to teaching about professional layout techniques.  This pattern may have gone unnoticed without the visual organization of a scoring guide.

Student Benefits. Course scoring guides benefit students as well.   Having assignment expectations outlined on a scoring guide directs students to the essential features they should focus on when producing an assignment.  Providing students with the completed scoring guides upon returning their assignments may also reduce the number of questions students have about how their grades were determined.  Moreover,students will be able to pinpoint the features of an assignment they need to improve upon.  For instance, a professor may have a rule that he/she subtracts five points from a student’s paper for not following MLA guidelines.  A student may be struggling with the MLA guidelines, but is likely to remain unaware of this struggle if  he/she is only provided with the final grade on the assignment.  However, the issue may become clear to the student if he/she is provided a scoring guide  that demonstrates the loss of  five points due to not following the MLA guidelines.

When designing a course scoring guide:

1.  determine the product you want to evaluate.  In the example above, the product would be a a research paper.   

2. determine the essential features of the research paper (i.e. comprehensive summary of research, well-articulated research question, etc.)  To ensure you are simplifying the grading process, do not include minutia such as font style when creating a scoring guide.

3. determine the performance metric you want to follow. You have several options, and the decision is often based on personal preference.  Some people prefer to associate a point value to each essential feature (five points for this, ten points for that, etc.), others might prefer to assess each essential feature based on a Likert scale.  Choose the option that you are most comfortable with.

Here is an example of a course scoring guide for a writing assignment.

If you have any questions about course scoring guides, please feel free to contact us. **Please note that although program scoring guides are created using the same steps those used to create a course scoring guide, OIA enforces a few standardized guidelines for creating a program scoring guide. If your program is interested in revising your program scoring guides, please contact your respective assessment coordinator.**

Writing program/course goals and outcomes provide the foundation on which an assessment plan can be developed.  Without strongly defined goals and outcomes, the assessment plan will crumble.  An assessment plan cannot stand on its own – it must be linked to a greater purpose to have any meaning.  An assessment plan not linked to program goals and outcomes, will likely provide generic results that are not relevant to a program’s intentions or relevant to a program’s capacity for improvement.

Luckily SCAD values the use of goals and outcomes, and expects them to be written for every program and course.  That being said, writing goals and outcomes is not a one-time initiative.  Goals and outcomes should be reviewed on an annual basis to account for any curriculum changes or realization of program’s actual intent or impact.

The OIA team is available to help programs revise their goals and outcomes.  OIA will facilitate a meeting among faculty to brainstorm and edit goals and outcomes in real time.  Although OIA facilitates the meeting, the majority of the revisions come from the faculty.  It is important for faculty to contribute to this process because only they can validate the accuracy of the goals and outcomes based on course content.  An additional benefit to faculty involvement is that it allows faculty proper time to reflect on their program as it stands and respond to faculty members’ perspectives on what the program should achieve.

Although it’s important to collaborate with fellow faculty in creating/updating goals and outcomes, it’s also important that everyone is aware of some basic guidelines for writing goals and outcomes.  Here are some basic guidelines:

Differentiating between goals and outcomes:

Goals

• Broad and general
• Aims of the course
• Indicates what the course provides
• Describes content and learning opportunities

Outcomes

• Specific, observable, and measurable
• Products of the course
• Indicates what the student can do
• Addresses assessment methods and criteria

Example of a Goal:

Students will be exposed to the materials, practices, and strategies needed to operate successfully as a studio artist.

Example of an Outcome:

Students will develop business materials such as purchase orders, contracts, and pricing lists that are consistent with industry standards.

Important Note: Verbs used to describe an outcome must be measurable because they are used to determine the data collected for an assessment and used to create the assessment tools.  AVOID verbs such as understand, appreciate, and learn, which represent actions or concepts that are ambiguous and difficult to measure.  Instead use active verbs like create, produce, analyze, describe.

To learn more about writing goals and outcomes, please review the tutorial attachment below created by OIA.

SCAD OIA_Writing goals and outcomes_8.3.11

Multiple choice tests are one of the more practical ways to gauge student learning.  There are two common mistakes made by test developers when designing multiple choice tests  – the test is either 1) too easy or 2) too confusing.  If a multiple choice test is too easy then it is likely not measuring quality course content.  If a multiple choice test is too confusing than it likely is just measuring student ability to take a multiple choice test (rather than measuring student knowledge).  An effective multiple choice question should consist of content that challenges student’s minds while also being formatted in a clear, cohesive way as to not challenge student’s ability to correctly answer the question if they understand the content.  Here are a few tips that breakdown an effective multiple choice question.

1. Understandable. Questions and answer options should be clear and concise, so that  students should only have to read them once to understand them.  The goal is to test student knowledge – not to test student’s ability to decipher multiple choice questions and answers.

2. Relevant. Try to avoid testing students on trivial class information (example:  name of documentary watched in class) and instead focus on content that aligns with course goals and outcomes.  Ideally, every grading opportunity should align with course goals and outcomes so that it adds to the overall assessment of the course and program its serving.

3.  Sufficient. Provide more than two answer options to the question.  If there are only two answer options, then a student has a 50% chance of guessing the right answer.  I suggest using four answer options for every question on a multiple choice test.  This way a student only has a 25% chance of guessing the right answer and the student does not become distracted with a varying number of answer options from question to question.

4 .  Plausible. Ensure all answer options are plausible.  This is probably the most difficult step to ensuring an effective multiple choice question.  It can be challenging to produce three plausible distractors (incorrect answer options) to distract from the correct answer option. However, since you are providing the correct answer with the multiple choice format,the only way to test student’s knowledge with this approach is to include plausible distractors.

5.  Design Repetition. Follow a consistent format with your questions.  To start, always place the omitted section of the question at the end of the sentence  rather than at the beginning or middle to preserve the “flow” of the question.  as mentioned above, it’s a good idea to stick to the same number of answer options for each question.  take that one step further and follow the same ordering format with each set of answer options.  for instance, you can place answer options in alphabetical order or you can place them in order from shortest response to longest response.  This formatting technique allows the test to flow for the reader, and it prevents the test developer from accidentally placing the correct answer in the same position for every question.

Please review the examples below for an illustration of an effective multiple choice question.  OIA is available to present extended tutorials on this subject matter, as well as others, through brownbag lunches or webinars. If you are interested in more information,  please contact OIA at assessment@scad.edu.

Examples