Performance Indicators in Higher Education: Trends and Critiques
Growing emphasis on performance indicators in higher education reflects a shift towards accountability and responsiveness to labor market needs. Rationales for using PIs include appeal to quality control and neutrality, but critiques highlight issues with citation counts and impact factors. Common forms of PIs include citation counts, impact factors, Hirsch index, completion rates, grant dollars, and league rankings.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
We Are Here: Phase 5: Performance Indicators (1990s to present): Growing faith in numbers-driven pse policy Mantra of accountability flexibility the responsive institution Calls for closer ties to the labour market All concerns about ideology of market-driven orientation of pse research and teaching deflected by reference to neutral and objective indicators of productivity. Attractive in environment of managerialism, emphasis on outputs rather than on public investment in pse as social good
Rationales for the use of PIs: Appeal to quality control: transferability, standardization Threatens academic freedom, new disciplines, risk-taking, diversity, disengaged from social objectives, diversity Appeal to accountability: justification of public spending Pse already accountable, as are academic staff. Confuses reporting on activities with setting direction
Rationales for the use of PIs: Appeal to neutrality and objectivity: removes the subjective judgements of peer review Peer review can be contested, grieved. PIs no less open to bias, but baked in, easy for non-academic managers to pass judgment on staff, programs Appeal to flexibility/responsiveness Cloak for market-driven incentives, short-term changes in academic direction. Surrenders notion of institutional autonomy from govt/industry.
Common forms of PIs: Citation counts: number of times a publication has been cited by others Impact factors: average # of times a journal s articles have been cited over past 2 years Hirsch or h-Index: author-level metric that tries to quantify both productivity and impact Completion rates/times: measures number of students who complete degree and/or how long they took Grant dollars: assesses academic staff according to amount of external $$ brought in League rankings: rates pse institutions, often relying on some of the above PIs
Critiques of PIs: Problems with citation counts Can count double entries, or miss entries because of errors, misspellings Self-citations Everyone cites the outrageous article (and not because it s good) New scholars need time for work to circulate Citations vary both within and across disciplines.
Critiques of PIs: Problems with impact factors: Gives average # of citations of a journal when only a fraction of articles actually cited skewed Meaningless as predictor of whether an article will be cited Thompson Reuters has ownership over opaque formula used to calculate the JIF
Critiques of PIs: Problems with impact factors confirmed by a massive study undertaken by a consortium of 11 science journals who paid for access to TR s data and manually counted citations to see if they matched the impact factors given to journals. "We hope that this analysis helps to expose the exaggerated value attributed to the JIF and strengthens the contention that it is an inappropriate indicator for the evaluation of research or researchers," the team wrote in a preprint paper posted to the open-access site bioRxiv. "JIF isn t the problem. It s the way we think about scholarly progress that needs the work.
Critiques of PIs: Problems with the Hirsch-index: H-index disadvantages newer scholars; does not capture research activity over time; may not register research activity that appears in non-print or community venues It does not include citations to the same work that have small mistakes in their referencing (of which for some publications there are many);
Critiques of PIs: It only includes citation to journal articles (not to books, book chapters, working papers, reports, conference papers, etc.); It only includes citations in journals that are listed in the ISI [Institute for Scientific Information] Thomson database, which especially for the Social Sciences and Humanities only includes a small proportion of academic journals in the field. Harzing (2016)
Critiques of PIs: Jorge E. Hirsch on the H-Index: Obviously a single number can never give more than a rough approximation to an individual s multifaceted profile, and many other factors should be considered in combination in evaluating an individual. This and the fact that there can always be exceptions to rules should be kept in mind especially in life-changing decision such as the granting or denying of tenure. (2005)
Critiques of PIs: Problems with completion rates/times: No allowance for diversity among disciplines (History vs Engineering vs longitudinal studies in Psychology) No recognition of learning differences among students No allowance for differential access to funding for students Contingencies of life for both students and staff One rate (2 years MA or 4 years PhD) does not fit all
Critiques of PIs: Problems with using grant dollars as metric for research quality: Massive difference in costs of research across disciplines (good library vs cutting edge spectrometer). Cheap research not lesser quality Size of grants not equal across disciplines (NSERC, CIHR, SSHRC) Can t control success rates. Fundamental Science Review Panel Report: getting SSHRC a lottery at 20% success rate. CIHR less than 10% Rewards giant teams with large numbers of co-applicants to boost application scores. Disadvantages areas not associated with huge teams/labs
Critiques of PIs: Problems with institutional rankings and league tables: Comparing institutions founded to serve different areas, interests, populations Disadvantages poorer colleges and universities; the Matthew Effect Discourages diversity, favours sciences over humanities Favours (contrived) internationalization over local research Can be gamed: experience of RAE in UK; researchers recruited to boost pub scores All of the problems associated with local and individual PIs compounded when rolled into suspect rankings of wildly diverse institutions world wide Improving rankings (from Maclean s to OECD) becomes the goal of management rather than education
PIs: What are they good for? Attack on peer review; transfer expertise from academics to formulae Provide cover for cuts in overall public spending (often with lip-service to marginalized groups) Provide reason to divert funds from front line teaching, research, service to assessment tasks undertaken by new levels of administration Fuel anti-intellectual distrust of academic staff Increased control/management of academics through system of internal reward/punishment; compliance culture
Assist in transformation of pse into job training Rationalize internal re-allocation of resources, PPP exercises Facilitate closer ties to business/industry at expense of liberal education, corporatization of academia Erode academic freedom, tenure, independence of institutions Instrument of discipline and reformation of the pse landscape and through it, of education into private good
Critical voices on the prevalence of PIs: It seems to me that many people are pushing universities to make them do what they themselves want the universities to do. That, to me, is not accountability, that is an attempt to intervene in decision making. Dr. David Strangway, former President, UBC It is not possible even in principle to approach the degree of commitment to promotion of free inquiry and expression statistically or to measure success in instilling a sense of ethical conscience. Drewes Committee, Trent University, 1994
Using figures to replace judgment is just stupid. --Yves Gingras, UQAM
Things to Remember The function of the university is not simply to teach bread-winning, or to furnish teachers for the public schools or to be a center of polite society; it is above all, to be the organ of that fine adjustment between real life and the growing knowledge of life, an adjustment which forms the secret of civilization. --W.E.B. Du Bois, The Souls of Black Folk, 1903 The college was not founded to give society what it wants. Quite the contrary. May Sarton, The Small Room, 1961
