In this article the results of the surveys of the experiences of the interviewees during the audit visits to three institutions in 2005 are reported. But the audit visits should not be seen in isolation from the broader, multi-year, multi-phased audit process as a whole. “Audit” should therefore not be equated with “audit visit”. Research (locally and internationally) has indicated that the visit and the interviews, in themselves, are perhaps the least valuable aspects of the audit process as a whole — especially the development of quality (and not only the monitoring of quality) is an important consideration. More valuable aspects of the audit process for quality development are the self-study process (over many months, even years) which precedes the audit visit and the implementation of the (multi-year) quality improvement plan which follows after the visit.
University of Cape Town
After the audit visit a questionnaire was distributed by the institutional audit steering committee to task team members and staff who participated in the interviews with the external panel. Forty-five responses were received from the 258 questionnaries sent out.
Generally, respondents rated the value of the audit visit process lower than that of the internal audit preparation process and 38% of respondents felt that the interviews by higher education quality committee (HEQC) panelists added value to UCT’s own self-evaluation exercise, 29% were not sure and 27% disagreed.
Also, 57% of respondents felt that the internal process of preparing for the audit was useful because it promoted reflective practices within the institution and a common understanding of the strengths and weaknesses of UCT’s current quality management systems.
The findings of the survey of the audit process contained mixed views about the value of the external panel. Most of the negative perceptions of the audit visit related to the shortness of the interviews.
The over-reliance on peer review as a method for conducting quality assurance has been challenged by some writers on the grounds of the subjectivity of peers in a competitive market-based and hierarchical system. Similar concerns surfaced during the UCT audit.
Other respondents felt that meshing internal processes with external peer reviews added more credibility to internal reflective processes and helped to add weight to the importance of actually addressing acknowledged weaknesses. Significantly, the interviews were experienced positively where they were perceived as adding value to internal members’ understanding of particular issues and challenges facing UCT through the asking of pertinent and incisive questions.
Respondents were less convinced about the value of the HEQC site visit, although the majority supported in principle the idea of external validation and the requirement for public accountability.
Rhodes University
A questionnaire was sent out to interviewees six weeks after the audit visit. A 41% response rate was obtained. In addition to 15 closed questions, respondents were asked to answer one open-ended question:
“In retrospect, how did you experience the HEQC audit process (self-evaluation, development of the audit portfolio, briefing, panel interview and post-audit feedback)? Did the process meet your expectations?”
Of those who chose to answer this question, 88% indicated that their experience of the HEQC audit process was generally positive.
Regarding the 15 focused questions, the consistent overall opinion was that the HEQC audit achieved what it set out to achieve:
• 90% of the respondents indicated that the questions asked during the interviews were related to the audit criteria.
• 84% of the respondents agreed or strongly agreed that the panel listened with an open mind to the responses to their questions.
• Only 4% felt the tone of the questions was negative towards Rhodes University.
• Only 2% indicated that the audit panel was ill-prepared for the interviews with 90% believing the panel was well prepared.
• 46% supported the perception of the audit visit being dominated by transformation issues, while 29% disagreed and 25% were neutral.
• 76% of respondents thought that the interviews were focused on quality assurance issues.
• 24% got the impression that the audit was focused on accountability, 43% were neutral and 30% felt it was more focused on improvement.
While a majority of respondents, 75%, were positive that the audit process would help the university improve its quality, only 50% agreed that the interviews provided insight into how the university can improve its quality.
These findings support international conclusions that internal self-evaluation is of more benefit to the institution than the actual audit visit.
University of Stellenbosch
Interviewees were requested to complete a paper-based questionnaire during a debriefing session after each interview and to participate in a facilitated reflection on each interview. Almost 500 responses were received. Respondents were invited to indicate to what extent they agreed or disagreed with statements on the panel interview and other issues pertaining to the HEQC audit as a whole.
In their responses to the statement, the panel’s questions were related to the HEQC’s audit criteria. The interviewees were satisfied that the panel did not act outside the agreed upon framework, taking into account that the questioning was focused on issues of transformation. Consider the following comments from interviewees:
• “The panel focused on transformation and the success of black students.”
• “There was a general concern for student well-being and transformation.”
In their response to the question about the panel listening with an open mind to the responses to their questions, 92% of the interviewees agreed, 6% were neutral and 2% disagreed. This is an indication that interviewees were mostly satisfied that the HEQC audit panels were not prejudiced.
The percentage of interviewees who agreed with the statement “the general tone of questions was not negative towards the university” was slightly lower than the response to the previous statement: 86%, with 10% indicating that they were not sure whether they agreed or disagreed while 4% experienced the tone of the questioning as negative.
In response to the statement “the panel took the university’s own goals seriously”, 80% agreed, 16% were neutral and 4% disagreed. These responses confirm that the “fitness for purpose” definition of quality did play a role in the questions of the HEQC panel.
In response to the statement “the interviews provided insight into how the university can improve its quality”, only 58% agreed, with 29% neutral. It is significant that the percentage that agreed with this question was much lower than the responses to the other statements in the survey.
The interviews conducted during the site visit clearly have a limited value for quality improvement. On the other hand, in the response to the statement “this HEQC audit will help the university to improve its quality”, 76% indicated that they agreed, with 21% neutral.
Although the context and approach to the audit was unique to each institution, the participants’ experience of the audit process in all three institutions was similar in a number of respects. In all three institutions, the audit preparation process was experienced to be a more useful learning and developmental experience than the audit visit.
Compiled by Jan Botha (University of Stellenbosch), Judy Favish (University of Cape Town), and Sandra Stephenson (Rhodes University)
The paper from which this is an extract has been accepted for publication in the international journal Quality in Higher Education