Regulatory Compliance and Quality Review Programme 2019

Audit quality review programme results

19. Our assessment is calculated from the combined results of AQRT inspections and the results of firms’ internal QMRs. Audit quality review assessments are made on each element of the auditor’s work at a local authority; financial statements, VFM arrangements conclusion and housing benefit certification. We also report the results of the independent AQRT assessments separately.

AQRT inspections

20. We commissioned inspections of all five firms by the FRC’s AQRT for this year’s QRP. The AQRT inspected ten financial statements opinion files and five VFM arrangements conclusion files across each firm’s 2017/18 work under PSAA audit contracts. The AQRT also provided an updated commentary on the applicability of firm-wide procedures to our audits. The financial statements reviews were targeted at firms with a larger share of the market, and towards more complex audits where it was more likely that learning for the regime could be obtained. Having considered the review points raised by the AQRT, we assessed the results using our four-point scoring system (Table 2). A score of 1 is our benchmark for acceptable performance. The full AQRT review scores are noted below in Tables 4 and 6.
21. The following scoring scale is used in assessing the quality of an audit.

Table 2: PSAA assessment scale

Overall audit quality results

22. The combined scores for the AQRT inspections for PSAA together with firms’ internal QMR scores generate our audit quality review assessments. We calculate the scores for overall audit quality on a weighted assessment using the weightings detailed in Appendix 1. We concluded that all firms’ QMRs were sufficiently detailed and rigorous for us to place reliance on all the reviews provided by the firm. This was supported by the results of AQRT reviews of firms’ QMR processes and reviews.
23. Figure 1 shows the assessment of firms’ overall audit quality performance taken from the combined results of the AQRT reviews and their own QMRs.

Figure 1: 2017/18 Audit quality performance

24. Our QRP methodology is designed to highlight any weaknesses at individual file level, and specifically where our benchmark score of ‘1’ is not met, which may have ordinarily been masked behind a high average score across the various elements (financial statements, VFM arrangements and HB certification) of the QRP.
25. We use the principles detailed in Appendix 1 to calculate a comparative red, amber, green (RAG) indicator for each element of the QRP and for the overall audit quality score from the numerical results of the quality reviews.
26. For 2017/18, all bar EY’s ratings were amber because their weighted score for overall audit quality was below ‘2’ the score required for a green rating. We consider each of the individual elements making up this rating below.

Table 3: 2017/18 Comparative performance for audit quality

Financial Statements

27. Auditors are required to give an opinion on whether the financial statements of an audited body give a true and fair view of its financial position and of its income and expenditure for the period then ended. They have other reporting responsibilities with respect to the preparation of the financial statements, the remuneration report and other information published with the financial statements.
28. The results of the AQRT reviews of a sample of 10 PSAA financial statement engagements for 2017/18 show that whilst in general firms are meeting the quality requirements overall, there are areas where the quality of their Local Audit needs to improve (Table 3). Although the AQRT reviews covered a small non-statistical sample the main areas of concern highlighted are all ones that have been raised repeatedly for several years around property plant and equipment and pension fund valuations. There is a need for significant improvement in firms’ procedures relating to management’s valuation experts, including corroboration of the experts’ samples; meeting with and challenging the experts on key valuation matters and assessing the completeness and accuracy of information used by the experts. It is disappointing that consistent and sustained improvement on these topics has not been achieved.

Table 4: Financial statements – AQRT review scores

29. The number of audits the AQRT identified as requiring significant improvement has decreased from 3 to 2 compared to last year. The overall results show four financial statement reviews in the lowest two assessment categories compared with seven in the previous year.
30. The principal issues identified this year across all the firms in the AQRT reviews for PSAA were:
    • the involvement of the Responsible Individual and Engagement Quality Control reviewers in the areas of key audit significance was not timely and to the level expected, and was a causal factor for many of the issues identified at the audits for which they were responsible;
    • the need for better audit responses to significant risks identified, with a bespoke response to those specific risks and circumstances of each audited body. This should improve both the quality of audit evidence and the effectiveness of procedures in arriving at an appropriate conclusion;
    • insufficient challenge and independent corroboration of management experts’ assumptions and valuations of PPE and other fixed assets, including approach to beacon properties;
    • insufficient consideration of how PPE valuations at the start of the accounting year remained materially valid and appropriate at the year-end;
    • insufficient consideration of the completeness and accuracy of source data provided to, and used by management’s experts (both valuers and actuaries);
    • insufficient testing of the valuation assumptions set and used by third party actuaries in establishing the value of pension fund investments;
    • insufficient audit procedures to assess the completeness of local authority’s bank accounts and over which audit procedures might be needed; and
    • poorly evidenced strategies for testing the completeness and cut-off of expenditure and liabilities.
31. Firms use their own methodology for assessing financial statement audit work (converting their financial statements results to our scoring system as necessary).
32. The firms provided the results of 25 internal QMRs for financial statement audit files. PSAA reviewed these and concurred with all the firm’s assessments. The AQRT reviewed a sample of these assessments, and commented that based on the number of issues raised on three files they would have been minded to have assessed them as ‘requiring limited improvement’ rather than ‘good’. This would not have changed any of the outcomes in this report.
33. In addition to the matters reported by the AQRT (noted above) the principal improvement area reported from firms’ individual QMRs was the need for clearer documentation on file of strategy for and execution of journals testing. There were no firm systemic concerns about the procedures in place to deliver audits of appropriate quality.
34. Figure 2 shows the comparative performance for financial statement audits based on the results of the QMRs and the AQRT review. All firms achieved acceptable performance.

Figure 2: 2018 financial statements performance

35. Our overall QRP methodology is designed also to highlight any weaknesses at individual file level, and specifically where our benchmark score of ‘1’ is not met, which may have ordinarily been masked behind a high average score across the audits reviewed. We use the principles detailed in Appendix 1 to calculate a comparative red, amber, green (RAG) indicator for each element of the QRP. As overall two of Mazars’ financial statements audits was judged as ‘0 ‘(significant improvements required) the firm received a comparative red rating for this element. Three firms, BDO, GT and KPMG were rated as amber as their average score was less than ‘2’. EY was rated as green, having a score of greater than 2.

Table 5:  2017/18 Comparative performance for financial statements audit work

Value for money arrangements

36. Auditors are required to give a value for money (VFM) arrangements conclusion as to whether the audited body has made proper arrangements for securing economy, efficiency and effectiveness in its use of resources.
37. The results of the AQRT reviews of a sample of five PSAA VFM arrangements conclusion files for 2017/18 found that all audits were rated as requiring only limited improvement or better (Table 6).

Table 6: VFM arrangements conclusion – AQRT review scores

38. The AQRT did not have any general improvement areas to report on the VFM arrangements work, which reflects the level of the quality scores.
39. Firms follow PSAA’s methodology and reporting format for their internal QMRs for VFM arrangements conclusion audit work and they provided the results of 24 audit reviews. PSAA reviewed these and agreed with all the firms’ assessments. We were satisfied from the results that the audit evidence was sufficient to support auditors’ VFM arrangements conclusions. The principal improvement area reported from firms’ individual QMRs was the need for clearer documentation on file of the timeliness of Engagement Lead reviews and involvement in risk planning throughout the engagement.
40. Figure 3 shows the comparative performance for VFM arrangements work based on the results of the QMRs and the AQRT reviews. All audits were rated as either good or with only limited improvements required.

Figure 3: 2017/18 VFM conclusion performance

41. For 2017/18 VFM arrangements conclusion work, the firms’ RAG ratings are shown in Table 7. All firms were rated green.

Table 7: 2017/18 Comparative performance for VFM conclusion audit work

Housing benefit certification work

42. Certification work is not an audit, but a type of assurance engagement. For 2017/18, PSAA auditors certified local authorities’ claims to provide assurance to the DWP that housing benefit subsidy claims complied with its terms and conditions. This is the last year for which these transitional arrangements apply and PSAA has no further responsibilities with respect to HB certification work, bar finalisation of a small number of 2017/18 certificates.
43. Firms followed PSAA’s methodology and reporting format for their internal QMRs for housing benefit certification work.
44. For each firm we reviewed a sample of their internal quality monitoring reviews of certification work to assess whether the auditor had followed the prescribed tests and guidance. We found that all suppliers complied with the requirements, but in a small number of cases there was scope for improving compliance with some aspects of the certification instructions.
45. The firms provided the results of 18 QMRs. We reviewed the results of these and we agreed with the firms’ assessments in all bar one case where we determined that a score of ‘2’ (acceptable with limited improvements required) was more appropriate than a score of ‘1’ (acceptable overall with improvements required).
46. Figure 4 shows the comparative performance of each firm based on the QMRs. All firms achieved an acceptable performance.

Figure 4: 2017/18 Housing Benefit certification   

47. The improvement areas from the firms’ individual QMRs included:
    • ensuring documentation of investigation thresholds, ‘nil’ denominator follow-up, assessment and follow up of prior year issues and overall conclusions on variance and ratio analysis is complete;
    • ensuring changes in audit strategy are documented clearly;
    • ensuring testing confirms the accuracy of determining an error as isolated;
    • ensuring a consistent standard of documentation where multiple assessors have completed HB Count workbooks.
    • ensuring evidence retained on file; and
    • completing documentation reviews and checklists on a timely basis.
48. Firms have noted these improvements for their own monitoring purposes.
49. The firms’ RAG ratings for 2017/18 certification work are shown in Table 8. As KPMG’s average score was less than ‘2’ they have an amber rating.

Table 8: 2018 Comparative performance for work on HB Certification

Terms of Appointment compliance – Contract Monitoring

50. Our Terms of Appointment specify for firms contractual expectations and indicators. Our monitoring of auditors’ performance focuses on 11 key indicators. These include the target dates for issuing audit opinions on the financial statements and VFM arrangements conclusions, reports on the whole of government accounts returns, producing annual audit letters, and sending us specified information and returns.
51. PSAA monitors and reports on compliance with these indicators on a quarterly basis. We have included a summary at Appendix 2 of the results of the 2018/19 Terms of Appointment compliance monitoring RAG ratings comparing the firms’ performance.
52. Table 9 details the firms’ overall Terms of Appointment compliance RAG rating. Full details are given in Appendix 2.

Table 9: 2018 Comparative performance for Terms of Appointment compliance

53. We are pleased to note that the majority of indicators were scored as green, where the requirement was either fully met, or met within a specified tolerance, resulting in an overall green rating. The information on individual indicators is provided in Appendix 2. We comment on three of these below.
54. Four firms were rated red and one amber for the target of issuing audit opinions by the publishing date of 31 July. The target was met at 87% of audited bodies. In almost all cases the delays were for reasons outside the auditor’s control, such as delays at audited bodies in producing financial statements or the requirement to resolve a complex objection, or technical issue. As this was the first year of the earlier close arrangements with the statutory publishing date being brought forward by two months it was unsurprising that there were delays at a number of authorities. Detailed analysis of the results showing those authorities which failed to publish their accounts by 30 September 2018 is provided in our report ‘Reporting the results of auditors work 2017/18’.
55. Complaints were upheld against two firms (EY and KPMG) resulting in amber ratings for this specific indicator.
56. Four firms were rated red and one amber in respect of the timeliness of the resolution of elector objections. As at 31 March 2019 there were 47 objections that had not been dealt with within our target time limit of nine months. The ability of electors to raise objections at audit under the Local Audit and Accountability Act 2014 is an important part of the checks and balances on local government and the time taken to resolve them is a concern. However there is no statutory deadline for completion, and many of the objections cover complex financial or contractual areas which properly take time to deal with.

Systems for compliance with our Terms of Appointment

57. In 2018/19, all firms confirmed to PSAA that their systems and procedures for informing us of their Terms of Appointment compliance were the same as those in the previous year. Nothing came to PSAA’s attention in the year to suggest this is incorrect, and we concluded that we could continue to rely on the firms’ systems.

Systems for compliance with our information assurance requirements

58. In 2017 all firms provided updated information on the systems underpinning their information assurance arrangements for meeting information governance legislation. This year we asked all firms to provide updated information and have concluded that we can continue to rely on them.

Client satisfaction surveys

All firms agreed to undertake client satisfaction surveys for 2017/18 audits, and to report the results to PSAA. We specified questions to be included in the survey and asked firms to provide us with an analysis of the results from a sample of local authorities on completion of their 2017/18 audit. These results show that audited bodies continued to be satisfied with the level of service they received, with all ratings for client satisfaction being green. This is despite the pressures of the earlier deadlines.

Table 10: 2017/18 Comparative performance for client satisfaction

59. Table 11 details the questions and the average score per question for each firm. Where necessary we have converted firms’ own scoring to a 10 point scale. Appendix 1 shows how this is converted into a RAG rating.

Table 11: Satisfaction survey results

60. The firms have undertaken an analysis of any improvement points raised in the survey and have committed to taking action on any individual improvement points identified.

Transparency reports

61. Auditors of listed companies have a statutory duty to produce an annual Transparency Report, which provides information about the firm’s governance and its arrangements for ensuring the quality of its work. The content is specified by the Professional Oversight Board of the FRC. All the firms in our audit regime are covered by this requirement and our review of the Transparency Reports did not highlight any significant issues in respect of local audit.
62. Additional information has to be provided by firms with Local Audit responsibilities as specified by the FRC’s the Local Auditors (Transparency) Instrument 2015.
63. In respect of Local Audit for 2018 our contracted firms reported that:
    • they had suitable quality control systems in place; and
    • their audit staff and engagement leads had received suitable local audit training and were competent to undertake local audit work.
64. However the Local Audit information was not always clearly delineated within some firm’s reports and had to be inferred from the whole firm information provided.
65. The FRC have reviewed these disclosures as part of their firm oversight work referred to below and are also conducting an overall thematic review of Transparency Reports. We expect to be able to take further assurance from these reports in future years as the FRC develops its public reporting of its quality reviews of local audit.

Financial Reporting Council

66. On the 10 July 2019 the FRC published its annual reports on the audit firms subject to full scope statutory AQRT inspections for private sector work. This covers all of our suppliers.
67. The FRC reported on the private sector audits that poor quality audit work remained unacceptably common. All firms reviewed had performed root cause analysis and identified a number of themes relating to why the audits inspected did not always meet the required standard and why certain findings recur over a number of years. These themes included insufficient scepticism and weaknesses in project management or resourcing. In addition, the analysis also highlighted inconsistent execution of firms’ audit methodologies and quality control procedures.
68. The FRC do note because of the small non statistical sample sizes that changes in review results from one year to the next cannot, on their own, be relied upon to provide a complete picture of a firm’s performance and are not necessarily indicative of any overall changes in audit quality at the firm. They conclude that any inspection cycle with audits requiring more than limited improvements is a cause for concern and indicates the need for a firm to take action to achieve the necessary improvements.
69. We raised these issues with firms in relation to the action they will be taking to address them. We will continue to monitor their progress in these areas.

Responses to QRP findings

70. Appendices 3 – 7 provide details of the actions the firms have, or intend to take to address the improvement areas identified. In addition to addressing the actions required to improve audit quality response to the recurring significant issues, particularly the audit of property valuations and defined benefit pension arrangements we have discussed firms’ :
    • proposals to complete Root Cause Analysis on the findings; and
    • arrangements to monitor implementation by audit teams of agreed actions.

71. All the firms have made arrangements to report the QRP findings to a suitable management group. We report our findings to the respective Audit Quality Partner of each firm.
72. PSAA will work with other stakeholders to promote:
    • Audit Committee involvement in material complex transactions and matters of key estimation uncertainty and judgement; and
    • the elements specified by the International Auditing and Assurance Standards Board that create an environment for audit quality.

Back to top