Evaluating Quality in Design Professions

As architects, designers, and engineers, we believe that Quality Assurance and Quality Control improve the work we do, but how do we really know that’s true? More specifically, how does a firm know if the QMS they have in place is effective, provides value, and improves project outcomes?

The answer to that question comes in three parts. The first is establishing a system of document review that is meaningful, repeatable, and consistent. The second is determining effective metrics to assess deliverable quality in the document review. And the final part is comparing the adoption rates of QA and QC processes with the document review results to determine whether the Quality Program and its components are yielding better work.

The first step in determining the Quality Program’s effectiveness is to establish a document review system that is consistent, repeatable on a regular cycle, and measurable. The system’s components are:

In the current AEC industry, the most predictable and consistent deliverables are made for the end of the Construction Documents phase, making the contract documents a good choice. Given the sheer volume of information contained in the specifications, reviewing only the drawings is more achievable and better suited to yield broadly actionable results. This also applies to projects that are fast tracked or have multiple bid packages; any drawing set delivered for bidding would be suitable.

Reviewing a representative cross-section of the firm’s work helps ensure the process is meaningful and broadly applicable. Being a multi-office firm, the author’s firm decided on a sliding scale between three and six drawing sets, based on office head count. The assumption being that larger offices produce more work so more sets are needed to get a representative sample.

Depending on the firm’s breadth of work, it may be necessary to establish parameters that help select the most representative work. That can mean identifying eligible project types or setting upper and lower cost or size limits. To ensure that current practices are being assessed, it is also important to establish a timeframe in which the documents were issued. Because the review is conducted annually, the timeframe is limited to the 12 months prior to the document review.

It can be helpful to collect information on project type, project size, cost, delivery method, and project-specific delivery conditions to add context to review results. For instance, if very large projects consistently score poorly, the firm might be ineffectively staffing or managing these types of projects.

Review process results are useful in any given cycle, but they are more likely to reveal trends and patterns if they’re gathered regularly. Establish a review frequency to ensure that the process repeats on a regular and predictable basis.

Reviewers can influence an evaluation system’s success or failure as much as any other factor. Reviewers need the requisite experience to effectively and efficiently evaluate the drawings, and they should also be open to novel and innovative approaches that can surface during a review process. Another consideration is whether to maintain a consistent reviewer group across cycles, to consistently select new reviewers, or to implement a combination. A consistent reviewer group can help reinforce firm standards and bring consistency to the process but can also be resistant to innovation and potentially institutionalize suboptimal practices. Regular turnover can result in inconsistencies, but it brings a variety of perspectives and approaches and can help encourage new delivery methods. The decision is primarily a matter of firm culture and review process goals.

After establishing evaluation criteria, it is equally important to create a scoring system to apply them. A system with a large range, say 1-10, allows nuance and flexibility in scoring, but it can also lead to a lack of clarity in the results. Conversely, a small range like 1-3 provides great clarity in differentiation and can help make sense of the inherent subjectivity of the evaluation criteria, but it lacks nuance. Considering all factors, a simpler and smaller range is recommended to provide clarity.

A large-scale evaluation of a firm’s deliverables is a significant undertaking, so establish a timeframe to help manage the cost and effort. In setting guidelines for the time spent on each set, remember this is an evaluation and not a detailed QC review. An hour or two per drawing set is sufficient for an experienced reviewer to make a reasonable evaluation.

Because the evaluation criteria are subjective and the reviewers bring their own perspectives and experiences to the task, it is important to control for subjectivity. Have multiple reviewers evaluate each of the submitted drawing sets and average the reviewer scores to balance out individual biases.

After reviews are complete, analyze the data to correlate reviewer scores with project parameters and attributes in a way that makes sense for the firm. Rankings can be based on individual project scores, projects by office (for multi-office firms), projects by building type, or delivery method. The analytics will be based on the firm’s structure and goals for the evaluation. Disseminating the results to firm leadership, office leadership, and project teams is critical to improving future outcomes.

Measuring the effectiveness of a Quality Program has two primary challenges. First, every building project is shaped by its unique site, program, constraints, requirements, and parameters. As a result, every building project is, essentially, a “prototype” that requires bespoke approaches, documentation, and delivery. In manufacturing, prototypes are made and then analyzed, tested, and revised to create an ideal product that production versions can be compared to. The unique nature of every building project means the design process and documentation differ from project to project and cannot be measured against predetermined “ideals.” How, then, do you measure the effectiveness of a Quality Program if you don’t have something to compare the outcome to?

Second, design and construction take a long time. Most projects take a year or more, and the largest and most complex can last for a decade. By the time the drawings can be evaluated for their effectiveness in conveying information, design processes, tools, and staff are likely to have changed, significantly limiting the feedback’s value.

Before discussing a proposed evaluation system, it is worthwhile to look at the metrics frequently suggested in the Architecture/Engineering/Construction (AEC) industry: RFIs, COs, and Legal Claims. While each of these can provide information on the quality or completeness of a set of documents, none are reliable as primary measures of deliverable quality.

RFIs: Tallying the number of RFIs on a project is easy, and we can use that number as an indication of document quality. While RFIs will be issued as a direct result of document quality, RFIs are also issued for many reasons that have nothing to do with the design documents. RFIs can be submitted due to unforeseen site conditions, market forces, confirmation of a change, or even mistakenly because the contractor missed information contained in the deliverables.

COs: Similarly, some COs may result from document deficiencies, but they are equally likely to stem from owner changes, client (architect) changes, regulatory requirements, or unforeseen site conditions.

Additionally, the total number of RFIs and COs and their causes will not be known until construction is complete and, as noted previously, months or years after the completion of the QA and QC processes. Even if RFIs and COs were effective measures of document quality, that information would not be available until too late to positively impact future design work.

Legal Claims: As with RFIs and COs, legal claims on building projects have many causes. Some are legitimate, some are spurious, and most have a multitude of contributing factors. They also typically take even longer to surface than RFIs and COs, making them even further removed from the design and documentation process and less effective as a metric.

If these are not the correct metrics, what are?

One measure of efficacy is QA/QC process uptake, which is fairly straightforward to evaluate. However, a high adoption rate of QA and QC tasks does not, by itself, prove the system is working. Understanding a Quality Program’s effectiveness in improving the design process and its outcomes is significantly trickier.

A two-tiered approach can be used in the document review to assesses construction documents’ quality.

The first tier that the author’s firm uses consists of the baseline criteria of proficiency, efficiency, and legibility that can be applied to any set of documents.

Proficiency measures whether all needed content is present and technically correct, if every area of the project is documented, and if the firm’s standards are utilized.

Efficiency measures whether the documents are organized in a clear, concise manner and if the content and number of drawings match the project’s scope and complexity.

Legibility assesses whether the set is easy to navigate, if information is easy to find and read, and if the sheets are laid out logically.

Tier 2 is applied at the beginning of the assessment program’s second year and has just one criterion: continuous improvement.

Continuous Improvement—During each review period, reviewers should identify “Start” (elements that all drawing sets should employ) and “Stop” (elements that should not be used in the future) practices. These will depend entirely on the firm’s processes, standards, and priorities.

After the first review period has been completed and the Start and Stop elements are broadcast to all design staff, adherence can then become a review criterion going forward.

The final step in determining the efficacy of a Quality Program is to compare review results with adoption rates of individual QA and QC components. This can be done broadly or narrowly.

Broadly, the combined scores of all assessment criteria can be compared to the overall adoption rate of all QA and QC processes by a project team, practice area, or office. This can be useful in surfacing broad trends across the firm.

More narrowly, individual criterion scores can be correlated with individual QA and QC processes to better understand their effectiveness. For instance, Efficiency scores can be compared to the frequency of projects doing cartoon sets to see if the cartooning process is improving productivity.

A combination of broad and granular assessments is likely to provide the best and most complete evaluation of whether a Quality Program and its individual components is working well, and it can also surface areas for improvement. Engaging in this process on a regular cycle will help demonstrate the Quality Program’s value and allow the firm to adjust that program to be most effective.

The work of architects, engineers, and designers is variable, constantly changing, and often difficult to evaluate objectively. A rigorous and repeating system of evaluating a firm’s work and correlating the results with the QA and QC process is one way to bring some order to the process, help firm leadership understand strengths and weaknesses, and improve project outcomes. ■

About the Author

Mark Walsh is an architect with 30 years of experience in design and coordination for all phases of project design and delivery, from programming and pre-design through construction contract administration. As Perkins&Will’s Firmwide Director of Technical Design, Walsh focuses on developing a culture that delivers design and technical excellence while embracing innovative delivery and construction techniques and seeking to improve efficiency across all aspects of the firm’s work.