Direct Examinations: Measuring and Evaluating Complex Subject Matters in the Government

Direct Examinations: Measuring and Evaluating Complex Subject Matters in the Government

By Christian Montull and Jack Doyle

In September 2020, the Auditing Standards Board (ASB) of the American Institute of Certified Public Accountants (AICPA) issued a new attestation standard (1) that creates a new form of attestation engagement for independent practitioners to perform direct examinations of an underlying subject matter in addition to assertion-based examinations. The new guidance is effective for reports dated on or after June 15, 2022, and allows independent practitioners to perform examinations and provide an opinion, even if a responsible party that engages the practitioner to perform the examination has not measured or evaluated the subject matter against the criteria or provides a written assertion.

What is distinctive about Direct Examination Engagements?

An examination is an engagement performed under attestation standards where an independent practitioner is engaged to obtain reasonable assurance and provide an opinion on a subject matter or assertion that is the responsibility of another party. Unlike financial audits, the subject matter in examinations may or may not include financial statements. Instead, they may consist of subject matters such as entity policy, cost representations, or System and Organization Controls (SOC).

Until recently, examinations were predicated on the concept that a responsible party would evaluate or measure the underlying subject matter against criteria and most often prepare a written assertion about the outcome of their measurement or evaluation before engaging an independent practitioner to express an opinion. Under the new standard, the independent practitioner performing a direct examination will express an opinion about the results of the independent practitioner’s direct measurement or evaluation rather than an opinion on the responsible party’s written assertion or previously evaluated subject matter.

Exhibit 1: Assertion-Based Examination Engagements and Direct Examination Engagements

How can the Government benefit from Direct Examinations?

Assurance, Independence, and Objectivity

The new standard allows Government entities to obtain examination reports on new and emerging financial and nonfinancial subject matters when the entity does not have the expertise or is not able to measure or evaluate a complex subject matter beforehand. Additionally, a direct examination makes it possible to obtain a high level of assurance and an independent opinion on areas that are increasingly important for the Government entity’s mission but tend to be difficult to assess due to limited resources or systems.

The entity responsible for the subject matter will not be required to expend resources to measure or evaluate the subject matter against criteria, which represents potentially large savings in terms of time, efforts, and expenditures. Instead, they can rely on the highly valued skills and expertise of an independent practitioner to measure or evaluate the complex subject matter and express an opinion that conveys the results of the measurement or evaluation performed. Mature entities can also benefit from independent and objective assessments by independent practitioners with experience performing examinations at other entities, including the application of best practices the responsible parties had not previously considered.

Impact on Government Procurement Process

Direct examinations can benefit the Government’s procurement process by simplifying solicitation requirements. These types of engagements will allow Government procurement officials to limit requirement specifications to the underlying subject matter, criteria, and organizational scope. Currently, the development of a request for proposal (RFP) or request for quotation (RFQ) for an assertion-based examination engagement requires a detailed understanding of the scope and complexity of the subject matter and criteria, such that proposals will ultimately align to the objective of the examination. By providing independent practitioners the flexibility to propose a scope and approach based on their expertise, procurement officials can reduce the initial involvement of technical personnel in the proposal development process, as well as reducing the cumbersome process of gathering the data necessary for an assertion, while also retaining the flexibility for practitioners and technical personnel to agree on the best approach and measurements for the subject matter.

Notable Use Cases

The following five cases are examples of where direct examinations may provide significant value to the Government.

Service Providers Performance Metrics

Government managers have and will continue to use service providers to achieve vast departmental and agency objectives. Service-Level Agreements (SLA) are a commonly used method to manage service provider performance over quantifiable objectives and associate them with variable costs for the service provider, through penalties and incentives. Service providers’ SLA metric reporting processes can involve complex calculations, particularly in the case of cloud service providers (CSP) and other SLA metric reporting with a high level of reliance on automated measurement systems. Government procurement and program management officials can coordinate to reduce material SLA reporting risks during the solicitation development process by requiring service providers to obtain assurance over the accuracy of SLA metrics by using direct examination engagements. The Government can also use direct examinations to inspect the nature of their supply chain risk management practices, helping establish and obtain assurance over related monitoring metrics.

Enterprise Risk Management (ERM)

Government operating and management models are unique in that their primary objectives and measures do not include typical commercial entity earnings goals. The Office of Management and Budget’s (OMB) update to Circular A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control, has elevated the importance of ERM metrics. Many ERM stakeholders are still working to establish a trusted baseline definition for key metrics; an assertion-based attestation will not be valuable if a defined, tested method for measuring metrics has not been established. The Government can use direct examination engagements to quickly confer with federal subject-matter experts and establish trusted metrics, then learn from direct examination processes or findings to mature internal metric management capabilities. In cases where key ERM metrics have been established, direct examination still has the advantage of validating the metrics remain appropriate for the subject matter and criteria.

Internal Controls and Related Assessments

Management is currently responsible for evaluating internal controls in order to provide reasonable assurance that the entity’s internal control over operations, reporting, and compliance is operating effectively. Agencies achieve this objective by conducting an assessment in accordance with OMB Circular A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control, and/or by engaging an independent practitioner to conduct an audit or examination engagement (e.g., audit readiness examinations) to provide an independent and objective opinion under applicable professional standards. The extent of documentation varies significantly depending on the size and complexity of the agency. Direct examination engagements would not require the assertion-level process formality and would allow stakeholders to quickly assess metrics without having to doubly invest in further process definition and documentation, while obtaining an increased level of assurance that comes with direct examination independence and objectivity.

Emerging Issues

In order to obtain reliable risk information on an emerging issue for which no self-evaluation or assertion has been performed, agencies currently need to engage practitioners to perform another type of service, such as agreed-upon procedures (2) (AUP), which may not provide the necessary assurance due to the structure and formality required by the related standards. Direct examination engagements would allow agencies to engage a practitioner to measure, evaluate, and report on the new and emerging financial or nonfinancial subject matter in accordance with (or based on) the specified criteria, without the agency having to measured or evaluated the subject matter beforehand, and with a higher level of attestation assurance.

Contractual or Regulatory Requirements

The new standard may be especially helpful for agencies that engage independent practitioners to verify that agencies and/or contractors comply with contractual and/or regulatory requirements. These agencies often engage independent practitioners to perform AUPs, performance audits, and/or assertion-based examination engagements for these services. Some of these services are limited to quantitative criteria, as qualitative assessments would require judgement not permitted by the standards (e.g., AUPs). Furthermore, the independent practitioner’s report for most of these engagements will be limited to presenting findings, conclusions, and recommendations based on specified criteria, rather than a concise opinion about whether the subject matter is in accordance with the contract or regulatory requirement. The new direct examination engagement will allow the independent practitioner to issue a report that would be more useful to the Government because it would report on both qualitative and quantitative criteria and deliver an easy-to-interpret opinion, without the need to measure or evaluate the subject matter beforehand or provide a written assertion.

Connect with Us

This publication is for informational purposes only and does not constitute professional advice or services. Readers should first consult with a professional before acting regard to the subjects mentioned herein.

Kearney & Company, P.C. (Kearney) is a CPA firm that is focused on providing accounting and consulting services to the Government.

¹ AICPA Statement on Standards for Attestation Engagements (SSAE) No. 21. The new standard adds AT-C section 206, Direct Examination Engagements, to the attestation standards, and changes the title of AT-C section 205 from Examination Engagements to Assertion-Based Examination Engagements.

² AICPA SSAE No. 19, Agreed-Upon Procedures Engagements