Academic Assessment Workshop on Program Educational Objectives and Student Outcomes
Understanding the significance of Program Educational Objectives (PEO) and Student Outcomes (SO) in academic assessment. PEO outlines career achievements while SO specifies knowledge and skills expected from graduates. Explore the anatomy of PEO, ABET Student Outcomes, and Program Specific Outcomes to enhance academic planning and evaluation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
WORKSHOP ASESMEN PENCAPAIAN SO ABET II 1 OKTOBER 2015 Struktur dan Perencanaan Asesmen Akademik tingkat Prodi
PEO & SO Program Educational Objectives Program educational objectives are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve Student Outcomes Student outcomes are narrower statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire in their matriculation through the program. ABET SO a k + Program Specific Outcomes
Anatomy of PEO Program Educational Objectives Profession of Graduates Field of work Level of responsibility Profile of Graduates Specific competences Capability to improve Institutional vision / idealism
Student Outcomes No ABET Student Outcomes 1 (a) an ability to apply knowledge of mathematics, science, and engineering 2 (b) an ability to design and conduct experiments, as well as to analyze and interpret data 3 (c) an ability to design a system, component, or process to meet desired needs with realistic constraints 4 (d) an ability to function on multi-disciplinary teams 5 (e) an ability to identify, formulate, and solve engineering and physics problems 6 (f) an understanding of professional and ethical responsibility 4
Student Outcomes No ABET Student Outcomes 7 (g) an ability to communicate effectively 8 (h) the broad education necessary to understand the impact of engineering and physics solutions in a global, economic, environmental, and societal context 9 (i) a recognition of the need for and an ability to engage in life-long learning 10 (j) a knowledge of contemporary issues 11 (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering physics practice 5
Program Specific Outcomes Discipline specific criteria formulated by professional society Overlapping requirements need to be satisfied only once
Program Specific Outcomes Performed alongside the assessment of SO Example : Civil Engineering 1. Curriculum The program must prepare graduates to apply knowledge of mathematics through differential equations, calculus-based physics, chemistry, and at least one additional area of basic science, consistent with the program educational objectives; apply knowledge of four technical areas appropriate to civil engineering; conduct civil engineering experiments and analyze and interpret the resulting data; design a system, component, or process in more than one civil engineering context; explain basic concepts in management, business, public policy, and leadership; and explain the importance of professional licensure. Reflected on the Faculty s Curriculum Vitae 2. Faculty The program must demonstrate that faculty teaching courses that are primarily design in content are qualified to teach the subject matter by virtue of professional licensure, or by education and design experience. The program must demonstrate that it is not critically dependent on one individual.
Continuous Improvement Process Program Level Assessment Course Level assessmen t
Our Tasks Map the PEO to SO Map the Curriculum Path Map SO to Curriculum Remember: Student Learning is Accumulative Assess the SO in Course and Program Level Determine Performance Indicator Determine Rubric Collect the data Analyze the data Report the SO Attainment
Application (10 minutes) State the PEO State the SO Map PEO to SO
Mapping PEO SO PEO SO a b c d e f g h i j k add PEO I PEO II PEO III PEO IV
MAP THE CURRICULUM PATH
Application (15 minutes) Choose one SO Determine the courses map that build the capacity of the attainment of the chosen SO Choose the courses to be assessed for the attainments of the SO 1. 2. 3. Repeat 1-3
COURSE AND PROGRAM ASSESSMENT Similarities and Differences
Course Assessment Course Context Subject matter Faculty member Pedagogy Student Facilities CONCEPTS TOPICS Control Elements Laplace Transformation Systems Dynamics Analysis Stability Transient Performance Frequency Response Empirical Design Model Based Design SUBJECT Terminology Process Models Process Performance Controller Design Automatic Controls Assessment Focus Evaluate individual student performance Evaluate teaching effectiveness Assessment Timeline: One Semester
Changes to Blooms Taxonomy 1956 2001 Evaluation Create Synthesis Evaluate Analysis Analyze Application Apply Comprehension Understand Knowledge Noun Remember Verb
Course Assessment Cannot cover all topics related to subject Cannot cover all concepts related to each topic Decision made based on context of course and characteristics of students Not all concepts are at the same cognitive level Assessment data taken at the concept level Assumption related to performance on topics based on performances on concepts CONCEPTS TOPICS Control Elements Laplace Transformation Systems Dynamics Analysis Stability Transient Performance Frequency Response Empirical Design Model Based Design SUBJECT Terminology Process Models Process Performance Controller Design Automatic Controls
Program Assessment Performance Indicator Student Outcomes Ability to gather relevant information from various sources Program Educational Objective Appreciation for and ability to pursue life- long learning Ability to manage time and energy to learn a new knowledge / skill Graduates will be capable to conduct advanced studies or to engage in early career path by being effective long-life learners Assessment Timeline: Years
Course vs. Program Assessment Course Assessment Program Assessment Cannot cover all topics related to subject Cannot cover all concepts related to each topic Decision made based on context of course and characteristics of students Not all concepts are at the same cognitive level Assessment data taken at the concept level Assumption related to performance on topics based on performances on concepts Cannot cover all Outcomes related to Program Educational Objectives Cannot assess all Performance Indicators related to each Outcome Decision made based on context of course and characteristics of students Not all Performance Indicators are at the same cognitive level Assessment data taken at the Performance Performance Indicator level Assumption related to performance on Student Outcomes based on demonstration of Performance Indicators
Cumulative Effect of Student Learning Over Time 4 Satisfactory Performance 3.5 Cognitive Level * SO Score 3 Student Ability = 2.5 2 1.5 1 Unsatisfactory Performance 0.5 0 Year 1 TF1101 Year 2 TF20XX Year 3 TF30XX Time Year 4 TF40XX
Context for Program Level Assessment Environmental Factors Pre Uni trait of students Assessment Timeline: Years
Grades Assessment Grades have limited use for program assessment as they do not have diagnostic value Grades can be a flag, but do not point to specific strength and weaknesses of what students know or can do A student s grade in a course or on a project or exam represents the student s performance on a set of aggregates knowledge / skills
Difference between Course & Program Assessment Degree of Complexity Time Span Accountability for the assessment process Cost Level of faculty buy-in Level of precision of the measure
Course Assessment : Example WRITING CONCEPTS TOPICS Focus Supporting Details SUBJECT Content Coherence Transitions Academic Writing Organization Voices Word Choice Sentence Fluency Convention Style Assessment Timeline: One Semester
Program Assessment : Example Performance Indicator (Abbreviated) WRITING Student Outcomes Supporting Details Program Educational Objective Effective communication skills Graduates will demonstrate the ability to communicate effectively with various audiences in written, Coherence oral and Audience Convention Graphics graphical forms Assessment Timeline: Years
CONCEPTS TOPICS Focus Supporting Details SUBJECT Content Coherence Transitions Academic Writing Organization Style Voices Word Choice Sentence Fluency Convention Performance Indicator (Abbreviated) Student Outcomes Supporting Details Program Educational Objective Graduates will Effective commu- nication skills demonstrate the ability to communicate effectively with various audiences in written, oral and graphical forms Coherence Audience Convention Graphics
Performance Indicators Essential Parts Subject content Content that is the focus of instruction (e.g., steps of the design process, chemical reaction, scientific method) Action verb Direct students to specific performance (e.g., list, analyze, apply) What to avoid Verbs that describe feelings, emotions, thoughts that are not measurable Appreciate, believe, know, learn, think Description of what student will do Write a paper about . Demonstrate hot to use Remember: write performance indicators from the perspective of what the student should be able to demonstrate by means of assessment
Blooms Action Verbs REMEMBER UNDERSTAND APPLY ANALYZE EVALUATE CREATE Count, Define, Describe, Draw, Find, Identify, Label, List, Match, Name, Quote, Recall, Recite, Sequence, Tell, Write Conclude, Demonstrate, Discuss, Explain, Generalize, Identify, Illustrate, Interpret, Paraphrase, Predict, Report, Restate, Review, Summarize, Tell Apply, Change, Choose, Compute, Dramatize, Interview, Prepare, Produce, Role- play, Select, Show, Transfer, Use Analyze, Characterize, Classify, Compare, Contrast, Debate, Deduce, Diagram, Differentiate, Discriminate, Distinguish, Examine, Outline, Relate, Research, Separate, Appraise, Argue, Assess, Choose, Conclude, Critic, Decide, Evaluate, Judge, Justify, Predict, Prioritize, Prove, Rank, Rate, Select Compose, Construct, Create, Design, Develop, Integrate, Invent, Make, Organize, Perform, Plan, Produce, Propose, Rewrite
Performance Indicators Students should be able to: <<action verb>> <<something>> Learner centered Specific action oriented Measurable Cognitively appropriate for intended level Value free free from subjective values or standards as best as you can
Application (5 minutes) Choose an Outcome Without talking to anyone in your table, write as many performance indicators as possible for the chosen outcome Write only one performance indicator per post-it Performance Indicators have two essential parts: Essential Parts Subject content Content that is the focus of instruction (e.g., steps of the design process, chemical reaction, scientific method) Action verb Direct students to specific performance (e.g., list, analyze, apply)
Affinity Process (15 minutes) Place all your post-its on the flipchart paper Because you table team was working on the same outcome, many of the performance indicators will be similar Move post-its around and group all the ones with SIMILAR CONTENT together (do not group them by VERB) After that is done, each grouping should represent one performance indicator CONTENT Determine the appropriate level (action verb) for each grouping and label the grouping as one performance indicator Use blank sheet at your table the outcome and list the performance indicator (one per grouping) for the outcome
Application (20 minutes) Determine the cognitive level of the Performance Indicators Determine the level of cognitive level of the whole SO path
Importance of Well Stated Performance Indicators Provides faculty with clear understanding for implementation in the classroom Makes expectations explicit to students (great pedagogy) Focuses data collection
Review of Performance Indicators Think performance indicator levels as: Introduce (knowledge and comprehension) Reinforce (application and analysis) Emphasize (evaluation and synthesis) Application : Identify the level of each Performance Indicator for the outcome you critiques (5 minutes) Reevaluate the courses that will be assessed using the performance indicator
Creating & Using Rubrics Objectives: Define Rubric Rubric Features Construct a Rubric Aggregate Rubric Data
Definition & Rubric Purpose A rubric is a scoring instrument for evaluating performance along a task-specific set of criteria rather than a single numeric score. Purposes of rubrics: Describe what outstanding or satisfactory or inadequate performance looks like Articulate the criteria against which student learning is judged Provide an objective guide for assessment
Basic Rubric Features A Rubric: Basic Features Stated Objective or Purpose - title Scoring Criteria characteristics of good performance on the task Levels of Performance defined degrees of competency Descriptors briefly define what is expected at each level of performance
Presentation Rubric http://www.rcampus.com/rubricshowc.cfm?code=Z79XC8&sp=true& Peer Evaluation: Group Work http://www.rcampus.com/rubricshowc.cfm?code=K78BBC&sp=true& Self Evaluation for Group Participation http://www.rcampus.com/rubricshowc.cfm?code=B78BAA&sp=true& Individual Portfolio http://www.rcampus.com/rubricshowc.cfm?code=K7369C&sp=yes& Blog http://www.rcampus.com/rubricshowc.cfm?code=N44XXC&sp=yes& Student Participation http://www.siue.edu/~deder/partrub.html Team Work http://course1.winona.edu/shatfield/air/team%20work.pdf Journal http://course1.winona.edu/shatfield/air/reflectionrubric-1.pdf
Steps to Creating A Rubric Identify and define the assessment objective or purpose (title) Select and write the needed number of scoring criteria (key attributes) Select and write the desired levels of performance Select and write the descriptors (features of scoring criteria)
Application : Rubric Development Using the outcome and performance indicators you developed, create an analytic rubric (at least four rows) Determine four performance levels Describe each performance level Remember : How will the findings be used? Will findings enable you to make decisions about program improvement? Use the template provided
Creating A Rubric Exemplary 4 17-20 Accomplished 3 13-16 Developing 2 8-12 Beginning 1 0-8 Writing Quality & Organization Engagement Study Habits Resource Knowledge Self-Efficacy
Aggregate Rubric Data Exemplary 4 Accomplished 3 Developing 2 Beginning 1 N = 75 Writing Quality & Organization 15 (20%) 25 (34%) 30 (40%) 5 (6%) Engagement 10 (13%) 15 (20%) 35 (47%) 15 (20%) Study Habits 15 (20%) 15 (20%) 40 (29%) 5 (6%) Resource Knowledge 5 (6%) 5 (6%) 60 (57%) 5 (6%) Self-Efficacy 5 (6%) 10 (13%) 40 (54%) 20 (27%)
Program Rubric Exemplary Satisfactory Developing Unsatisfactory Provides ample supporting details Provides adequate supporting details Include some details, but may include extraneous or loosely materials Includes inconsistent or few details that may interfere with meaning of the text Supporting Details Content Organizational pattern is logical, convey completeness & wholeness Organizational pattern is logical, convey completeness & wholeness with few lapses Achieves little completeness & wholeness though organization attempted Little evidence of organization or any sense of completeness & wholeness Organization Coherence Uses effective language, makes engaging & appropriate word choice for audience & purposes Uses effective language & appropriate word choice for audience & purposes Limited & predictable vocabulary, perhaps not appropriate for intended audience & purposes Has a limited or inappropriate vocabulary for the intended audience & purposes Audience Consistently follows the rules of standard English for conventions Generally follows the rules of standard English for conventions Generally does not follows the rules of standard English for conventions Does not follow the rules of standard English for conventions Style Conventions Figures and chart are appropriate, clear and communicate well to the audience Figures and chart are clear and with few exceptions, communicate clearly Figures and chart are used to communicate, but lack consistency in format and style detracting from audience understanding Figures and chart are missing or have deficiencies in formatting and style which detract from understanding Graphics
Course Rubric Exemplary Satisfactory Developing Unsatisfactory Maintain exceptional focus on the topic Maintain consistent focus on the topic Provides inconsistent focus on the topic Demonstrate little or no focus Focus Content Provides ample supporting details Provides adequate supporting details Include some details, but may include extraneous or loosely materials Includes inconsistent or few details that may interfere with meaning of the text Supporting Details Organizational pattern is logical, convey completeness & wholeness Organizational pattern is logical, convey completeness & wholeness with few lapses Achieves little completeness & wholeness though organization attempted Little evidence of organization or any sense of completeness & wholeness Coherence Organization Provides transitions that eloquently serve to connect ideas Provides transitions that serve to connect ideas Provides transitions that is weak or inconsistent Uses poor transitions or fails to provide transitions Transition Allows the reader to sense the person behind the words Some sense of the person behind the words is evident Some sense of the person behind the words is attempted Little or no sense of the person behind the words is attempted Voice Uses effective language, makes engaging appropriate word choices for audience & purposes Uses effective language, appropriate word choices for audience & purposes Limited & predictable vocabulary, perhaps not appropriate for intended audience & purposes Has a limited or inappropriate vocabulary for the intended audience & purposes Word Choice Style Sentences / phrases appropriately varied in length & structure Sentences / phrases somewhat varied in length & structure Shows limited variety in sentence length & structure Has little or no variety in sentences length & structure Sentence Fluency Consistently follows the rules of standard English for conventions Generally follows the rules of standard English for conventions Generally does not follows the rules of standard English for conventions Does not follow the rules of standard English for conventions Conventions
