Multi-State Collaborative for Advancing Quality Student Learning
Slides showcasing results from the demonstration year (2015-2016) of the Multi-State Collaborative for Advancing Quality Student Learning involving 12 states. Study findings reflect the use of common rubrics to assess student work and highlight the challenges and benefits of the VALUE rubrics. Faculty found the rubrics valid, with actionable data generated for student achievement and learning outcomes. A web-based platform facilitated assessment processes and training programs.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Demonstration Year Results (2015-2016)Rev. 2/3/2017 9/6/2024
Participating States 12 States Connecticut, Hawaii, Indiana, Kentucky, Maine, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Texas, and Utah 9/6/2024
Study Findings and Summary These slides summarize results from the demonstration year of the Multi-State Collaborative for Advancing Quality Student Learning involving twelve states using common rubrics to assess more than 7,000 pieces of student work. The sample of student work represented the near-graduation students across the participating institutions only; therefore, the results are not generalizable for all students in each participating state or nationwide. 9/6/2024
Inherent Challenge for VALUE Navigating Methodological Complexity
Purpose = Discuss validity & reliability in relation to inherent complexity of VALUE Scores (rubrics) Scorers Assignments
A careful balancing act Philosophical/ Methodological Pedagogical
Faculty & staff saw the VALUE rubrics as valid. Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use Useful for evaluating student work 89% Scoring levels provided sufficient range 86% Descriptors were understandable 83% Descriptors were relevant 80% Encompassed meaning of outcome 75% These results are not generalizable across participating states or the nation in any way. Please use appropriately.
Lessons Learned Actionable data about student achievement and improvement of key learning outcomes on specific key dimensions of these important learning outcomes can be generated via a common rubric-based assessment approach. Faculty can effectively use common rubrics to evaluate student work products even those produced for courses outside their area of expertise. Following training, faculty members can produce reliable results using a rubric-based assessment approach. Faculty report that the VALUE Rubrics used in the study do encompass key elements of each learning outcome studied, and were very useful for assessing student work and for improving assignments. A web-based platform can create an easily usable framework for uploading student work products and facilitating their assessment.
Representativeness: Gender Graduates in Participating States Relative to MSC Project Sample Demographics IPEDS MSC Male 41% Male 42% Female 58% Female 59% Population data used are 2015 IPEDS graduates (associates, bachelors) in participating states. 9/6/2024
Representativeness: Pell Eligibility Enrollees in Participating States Relative to MSC Project Sample Demographics Percent of Students with Pell-Eligibility 2 Year 4 Year 75% 51% 50% 41% 41% 34% 25% 0% IPEDS MSC Population data used are 2013 IPEDS enrollees (2-year public, 4-year public) in participating states. 9/6/2024
Representativeness: Age Graduates in Participating States Relative to MSC Project Sample Demographics 2 Year 4 Year 100% 12% 13% 6% 16% 29% 22% 75% 38% 38% 50% 66% 66% 25% 49% 45% 0% IPEDS MSC IPEDS MSC 18-24 25-39 40+ 18-24 25-39 40+ Population data used are 2015 IPEDS graduates (associates, bachelors) in participating states. 9/6/2024
Representativeness: Race/Ethnicity Graduates in Participating States Relative to MSC Project Sample Demographics 2 Year IPEDS 60% White MSC 67% White 20% 20% 18% 16% 14% American Indian / Alaska Native 14% Asian 12% Black / African American 10% 9% Hispanic / Latino 8% Native Hawaiian / Pacific Islander 7% Two or more races 6% 4% 4% Non-resident alien 3% 4% 3% 2% 2% 1% 1% 1% 0% 0% 0% IPEDS MSC Population data used are 2015 IPEDS graduates (associates, bachelors) in participating states. 9/6/2024
Representativeness: Race/Ethnicity Graduates in Participating States Relative to MSC Project Sample Demographics 4 Year IPEDS 64% White MSC 70% White 20% 18% 16% 14% American Indian / Alaska Native 14% Asian 12% Black / African American 10% Hispanic / Latino 7% 8% 7% Native Hawaiian / Pacific Islander 6% 6% 5% Two or more races 6% 4% 4% Non-resident alien 4% 3% 2% 2% 0% 0% 0% 0% 0% IPEDS MSC Population data used are 2015 IPEDS graduates (associates, bachelors) in participating states. 9/6/2024
Assignments and Artifacts Number of Assignments: 1,156 The number of assignments approximates the number of faculty participants Critical Thinking Quantitative Reasoning Written Communication Totals 2 Year 180 87 212 479 4 Year 286 109 282 677 Total 466 196 494 1,156 Number of Artifacts (Pieces of student work): 7,114 The number of artifacts approximates the number of student participants Critical Thinking Quantitative Reasoning Written Communication Totals 2 Year 840 576 919 2,335 4 Year 2,056 787 1,936 4,779 Total 2,896 1,363 2,855 7,114 9/6/2024
Massachusetts vs. Project Level Outcome Scores 9/6/2024
Massachusetts Participating Institutions: 2016 This academic year included three community colleges, two state universities, and two UMass. campuses
Critical Thinking Rubric Dimensions Capstone Milestones Benchmark 4 3 2 1 Issue/problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding. Issue/problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions. Issue/problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown. Information is taken from source(s) with some interpretation/evaluation, but not enough to develop a coherent analysis or synthesis. Viewpoints of experts are taken as mostly fact, with little questioning. Issue/problem to be considered critically is stated without clarification or description. Explanation of issues Information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis. Viewpoints of experts are questioned thoroughly. Information is taken from source(s) with enough interpretation/evaluation to develop a coherent analysis or synthesis. Viewpoints of experts are subject to questioning. Information is taken from source(s) without any interpretation/evaluation. Viewpoints of experts are taken as fact, without question. Evidence Selecting and using information to investigate a point of view or conclusion Thoroughly (systematically and methodically) analyzes own and others' assumptions and carefully evaluates the relevance of contexts when presenting a position. Specific position (perspective, thesis/hypothesis) is imaginative, taking into account the complexities of an issue. Limits of position (perspective, thesis/hypothesis) are acknowledged. Others' points of view are synthesized within position (perspective, thesis/hypothesis). Conclusions and related outcomes (consequences and implications) are logical and reflect student s informed evaluation and ability to place evidence and perspectives discussed in priority order. Identifies own and others' assumptions and several relevant contexts when presenting a position. Questions some assumptions. Identifies several relevant contexts when presenting a position. May be more aware of others' assumptions than one's own (or vice versa). Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions). Begins to identify some contexts when presenting a position. Specific position (perspective, thesis/hypothesis) is stated, but is simplistic and obvious. Influence of context and assumptions Specific position (perspective, thesis/hypothesis) takes into account the complexities of an issue. Others' points of view are acknowledged within position (perspective, thesis/hypothesis). Specific position (perspective, thesis/hypothesis) acknowledges different sides of an issue. Student's position (perspective, thesis/hypothesis) Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified. Conclusions and related outcomes (implications and consequences) 9/6/2024 For full text of AAC&U VALUE Rubric for Critical Thinking, see: https://www.aacu.org/value/rubrics/critical-thinking.
Critical Thinking at 2-year institutions Explanation of Issues Evidence Context/Assumptions Project (n=840) Massachusetts (n=108) Student's Position Conclusions/Outcomes 0 1 2 3 4
Critical Thinking at 4-year institutions Explanation of Issues Evidence Context/Assumptions Project (n=2,056) Massachusetts (n=378) Student's Position Conclusions/Outcomes 0 1 2 3 4
Analysis: Critical Thinking 2 Year: Mass. exceeds Project level data 4 Year: Mass. is comparable to Project level data However, insignificant change from 2 to 4 Year suggesting more effort is required to achieve higher results everywhere. Employers rate critical thinking (82%) as highly valued competency after problem solving (91%), ethical issues (87%), community problem solving (86%), and civic knowledge, skills, and judgment (82%).* *It Takes More than a Major: Employer Priorities for College Learning and Student Success (Washington, DC: Association of American Colleges and Universities, 2013).
Written Communication Rubric Dimensions Capstone Milestones Benchmark 4 3 2 1 Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the assigned task(s) and focuses all elements of the work. Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assigned task(s) (e.g., the task aligns with audience, purpose, and context). Demonstrates awareness of context, audience, purpose, and to the assigned tasks(s) (e.g., begins to show awareness of audience's perceptions and assumptions). Demonstrates minimal attention to context, audience, purpose, and to the assigned tasks(s) (e.g., expectation of instructor or self as audience). Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s). Uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer's understanding, and shaping the whole work. Uses appropriate, relevant, and compelling content to explore ideas within the context of the discipline and shape the whole work. Uses appropriate and relevant content to develop and explore ideas through most of the work. Uses appropriate and relevant content to develop simple ideas in some parts of the work. Content Development Demonstrates detailed attention to and successful execution of a wide range of conventions particular to a specific discipline and/or writing task (s) including organization, content, presentation, formatting, and stylistic choices Demonstrates consistent use of important conventions particular to a specific discipline and/or writing task(s), including organization, content, presentation, and stylistic choices Follows expectations appropriate to a specific discipline and/or writing task(s) for basic organization, content, and presentation Attempts to use a consistent system for basic organization and presentation. Genre and Disciplinary Conventions Formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields (please see glossary). Demonstrates skillful use of high-quality, credible, relevant sources to develop ideas that are appropriate for the discipline and genre of the writing Demonstrates consistent use of credible, relevant sources to support ideas that are situated within the discipline and genre of the writing. Demonstrates an attempt to use credible and/or relevant sources to support ideas that are appropriate for the discipline and genre of the writing. Demonstrates an attempt to use sources to support ideas in the writing. Sources and Evidence Uses graceful language that skillfully communicates meaning to readers with clarity and fluency, and is virtually error- free. Uses straightforward language that generally conveys meaning to readers. The language in the portfolio has few errors. Uses language that generally conveys meaning to readers with clarity, although writing may include some errors. Uses language that sometimes impedes meaning because of errors in usage. Control of Syntax and Mechanics 9/6/2024 For full text of AAC&U VALUE Rubric for Written Communication, see: https://www.aacu.org/value/rubrics/written-communication.
Written Communication at 2-year institutions Context/Purpose Content Development Project (n=919) Genre/Conventions Massachusetts (n=123) Sources/Evidence Syntax/Mechanics 0 1 2 3 4
Written Communication at 4-year institutions Context/Purpose Content Development Project (n=1,936) Genre/Conventions Massachusetts (n=259) Sources/Evidence Syntax/Mechanics 0 1 2 3 4
Analysis: Written Communications 2 Year: Other than Sources/Evidence, Mass. lags Project level data 4 Year: Performance improves except for Syntax/Mechanics, which is about the same as 2 Year students.
Quantitative Literacy Rubric Dimensions Capstone Milestones Benchmark 4 3 2 1 Provides accurate explanations of information presented in mathematical forms. Makes appropriate inferences based on that information. For example, accurately explains the trend data shown in a graph and makes reasonable predictions regarding what the data suggest about future events. Provides accurate explanations of information presented in mathematical forms. For instance, accurately explains the trend data shown in a graph. Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line. Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means. For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends. Interpretation Ability to explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words) Skillfully converts relevant information into an insightful mathematical portrayal in a way that contributes to a further or deeper understanding. Competently converts relevant information into an appropriate and desired mathematical portrayal. Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate. Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate. Representation Ability to convert relevant information into various mathematical forms (e.g., equations, graphs, diagrams, tables, words) Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations are also presented elegantly (clearly, concisely, etc.) Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations attempted are either unsuccessful or represent only a portion of the calculations required to comprehensively solve the problem. Calculations are attempted but are both unsuccessful and are not comprehensive. Calculation Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work. Uses the quantitative analysis of data as the basis for competent judgments, drawing reasonable and appropriately qualified conclusions from this work. Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work. Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work. Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Explicitly describes assumptions and provides compelling rationale for why each assumption is appropriate. Shows awareness that confidence in final conclusions is limited by the accuracy of the assumptions. Explicitly describes assumptions and provides compelling rationale for why assumptions are appropriate. Explicitly describes assumptions. Attempts to describe assumptions. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Uses quantitative information in connection with the argument or purpose of the work, presents it in an effective format, and explicates it with consistently high quality. Uses quantitative information in connection with the argument or purpose of the work, though data may be presented in a less than completely effective format or some parts of the explication may be uneven. Uses quantitative information, but does not effectively connect it to the argument or purpose of the work. Presents an argument for which quantitative evidence is pertinent, but does not provide adequate explicit numerical support. (May use quasi- quantitative words such as "many," "few," "increasing," "small," and the like in place of actual quantities.) Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized) 9/6/2024 For full text of AAC&U VALUE Rubric for Quantitative Literacy, see: https://www.aacu.org/value/rubrics/quantitative-literacy
Quantitative Literacy at 2-year institutions Interpretation Representation Calculation Project (n=576) Application/Analysis Massachusetts (n=24) Assumptions Communication 0 1 2 3 4
Quantitative Literacy at 4-year institutions Interpretation Representation Calculation Project (n=787) Massachusetts (n=43) Application/Analysis Assumptions Communication 0 1 2 3 4
Analysis: Quantitative Literacy The size of Mass. n is really problematic to draw conclusions 2 Year: Other than Applications/Analysis and Assumptions, Mass. is behind Project level data. How much of this is a function of the assignments and the size of the n? 4 Year: Mass. is comparable to Project level data except in Assumptions, which reflects a general improvement in all categories. However, no change in Assumptions within Mass. from 2 to 4 Year whereas project level data improved.
Suggested Actions Increased efforts in upper level courses to teach and assess for critical thinking across the curriculum in order to increase student learning. State-wide assignment design discipline-based workshops Significantly increased efforts required in writing across the curriculum and the restructuring of Developmental Writing. Smaller writing classes Use of peer editing (high impact practice) Requiring a minimum number of pages written per course and multiple drafts State-wide workshops on undergraduate writing Increased emphasis on syntax/mechanics across the curriculum at all levels. Significantly increased efforts in quantitative literacy across the curriculum in lower level courses in Interpretation, Representation, and Calculation. State-wide assignment design discipline-based workshops
Suggested Actions Increased emphasis on Quantitative Literacy: Assumptions in all courses across the curriculum and in assignments. Compare MSC-Mass. data to NESE data for possible correlations Attempt to achieve the stratified participation sample for Mass. in order to increase n and generalize from the results Gain participation of small institutions with low minority representation
Next Steps MSC Refinement Year (year three) 13 states, five with representative samples for the state 20,000 artifacts collected and uploaded Establishment of inter-state SWAT teams Increased focus on evaluation panel of data scientists Increased focus on equity Explore feasibility of sub-study following students into the workforce 31
Conclusion Open Discussion Next Steps Thank You!
