1.040/1.401/ESD.018Project Management

Lecture 17

Project Quality Reviews & AuditsSpring 2007Sam Labi and Fred Moavenzadeh Massachusetts Institute of Technology Sam Labi

Recall: The 5 Phases of Project Management DEVELOPMENTOPERATIONS DESIGN,PLANNING FEASIBILITYOrganizationEstimationPlanningFinanceEvaluationMonitoring & ControlProject AberrationsChanges & ClaimsQuality & Reviews, & AuditsCLOSEOUTThis Lecture

Quality Performance What are the issues?

Project Reviews

Project Audits What are they?

Contents of this Lecture

Who cares? Also, is QP influenced by construction method?Interrelationships between Quality and the other performance metrics (Time, Cost)Quality ControlQuality AssuranceQuality Management Total Quality Management (TQM)Note: In recent years increasing focus on Quality, particularly on federal contractsQuality Performance What are the Issues?

OwnerResident Project RepresentativeArchitects superintendentManufacturersPrefabrication factory inspectorUtility companiesSite inspectorsProject Developer/ContractorField managerQuality managerForemenOther StakeholdersThe general public (or its reps) e.g., Local building department (code compliance)Safety and health inspectors (U.S. OSHA)Insurance company inspectorsFinancial institution inspectorsProject Quality Who Cares?

How is Quality Related to Construction Method?Method 1 - Product may be produced elsewhere and then brought to site, 123Source: Matthew NormanPrefab factory

or Method 2 - product may be produced at the SiteSource: brian moriarty

Many ways of classifying construction methods. One way is whether Projects physical component are - created on site (concrete cast in-situ) OR- created elsewhere and brought to site (precast concrete)

Pre-fabricated components: generally higher qualityTighter tolerancesManufactured under tightly-controlled conditions More rigorous quality control mechanismsShortcoming: Longer delay if identify problems!

Site-created components: generally lower qualityVulnerable to Weather effects, relaxed adherence to standards, etc.So, Is Product Quality Related to Construction Method?

Inspection may occur at the source (factory) or after they are delivered at the site.Examples of external sitesPrecast concrete plantsSteel plate fabrication plantsConcrete production plantsWelded steel plantsLarge, specialized pieces of equipmentQuality checks at external fabrication site OR at project site can be carried out by Owner (or his rep), or the ContractorMay also monitor product quality DURING transport of the productQuality Inspections of Externally-produced Components

Quality Control vs. Quality Assurance: Whats the Difference?

Quality Control vs. Quality Assurance: Whats the Difference?Analogy: You Driving on a Freeway

Quality Control vs. Quality Assurance: Whats the Difference?Analogy: You Driving on a FreewayDriving Quality AssuranceBefore Driving: adjust seats, mirrors, temp, etc.During Driving: Occasional glances at: - speed gage to ensure speed is not excessive - side mirror before changing lane - road sides to read road signs, etc.No distractive activities (reading, eating, etc.)These are preventive measures

Quality Control vs. Quality Assurance: Whats the Difference?Analogy: You Driving on a FreewayBefore Driving: adjust seats, mirrors, temp, etc.During Driving: Occasional glances at: - speed gage to ensure speed is not excessive - side mirror before changing lane - road sides to read road signs, etc.No distractive activities (reading, eating, etc.)During and After Driving: Generate/report data to prove that necessary precautions were taken and that your driving did not violate any rulesSwerving to avoid deer crossing the highwaySteering to right if car is straying into left lane Braking to avoid hitting slowed car in frontDriving Quality AssuranceDriving Quality ControlThese are corrective measures

Quality Control vs. Quality Assurance: Whats the Difference?Quality Assurance- Do it right the first time". Preventive Quality checks.

Quality Control: Fix it when ever it goes or is going wrong But note! In recent years, QA is defined to include QCNewer Definition of QA: The acts of (i) establishing that product is being delivered according to all specifications, (2) providing evidence needed to establish confidence that all quality-related activities are being performed effectively- Includes that part of project monitoring that covers project quality. - Covers all activities from design, development, production, installation, servicing and documentation- Includes regulation of the quality of raw materials, assemblies, products and components; services

Quality Control vs. Quality Assurance: Whats the Difference?ImportantExact definitions of QA-QC may vary from industry to industry, agency to agency, country to country.

The Relationships:Quality Control, Quality Assurance, Project Monitoring Project MonitoringQuality AssuranceQuality ControlMonitoring of project qualityMonitoring of other project performance metrics (time, money, etc.)

More on Quality Assurance (QA)Involves regular but random testing of materials and workmanship (time-based or work-based intervals) Prevent, identify, and correct quality-related problemsDuring the production process, QA instructors mostly provide guidance and leadership to the production people rather than criticizing their work

More on Quality Control (QC)QC mostly carried out by Contractor, but the Owners rep. may draw Contractors attention to carry out QC at any time. Owner also can have his independent testing.When the QC Division and Production Division belong to same organization, these parties often have adversarial relationshipsThe production people often tend to cover and hide their mistakes by nature.During/after every project activity, Contractor provides Owner with documentation that the product complies with specifications

What is Quality Management?Initiated and orchestrated by senior managersInvolves all parts of the organization (main C, sub-C, suppliers, etc.)Through a systematic, comprehensive and well-documented QA processRationale: Improved QA serves to enhance all other performance metrics of the project (time, cost, etc.) in long runObjective: A zero-defects productTechnique: Preventive as well as corrective -- Ensure quality NOT ONLY by rejecting failures but ALSO by investigating possible risks and threats to quality at root sources

What is Total Quality Management (TQM)?Not just an operational strategy for PM A PM philosophyAimed at continuous improvement of the organization and personal growth of its individual membersEnsures that the environment surrounding the Product is viewed in the broadest sense including:- Quality of Life- Well-being and satisfaction of all people Involved- Long-lasting relationships with customers and suppliers etc.

Linkages b/n Quality and Key Performance Metrics TradeoffsTimeQuality$Acceleration $(Overtime, shift work, Rework, higher-end equipment, better crews etc.)Less $ Low progress Resource reduction Selection of poor quality workers Default of contractor/subsQuality problems may result from overtime, shift work, new hiresQuality level impacts speed of work, Level of reworkNeed for rework imposes high expensesHigh quality needs can lead to costly miscalculations on labor timeTrying to save $Can lead to substitution, lower quality workmanshipSlow progress $ Delayed occupation, Higher interest on const. loan Loss of tenants Opportunity cost

Going back to examine the product from the project. - Has product (or part thereof) been produced? - When was it done? punctual?- Was it done right? - With the needed resources? Within the given time? Reviews enable us to learn from mistakes as well as successes. Models used in PM should also be reviewed for their efficaciesExample CPM, WBS/OBS/CBS, fishbone, etc.Project reviews may be carried out by external parties (e.g. consultants)However, transience of project activities/teams can impair the accumulation of institutional knowledge and thus can impede Project LearningSource: F. Pena-Mora 2003Project Reviews

The Need for Project ReviewsWhy do we review projects? Bridging Gaps Validation of Work Done Quality Assurance Learning

More on the Need for Project ReviewsWhy construction projects in particular need formal reviews:

Design and Planning Phase Generating 75 % of the Problems Encountered in the Construction Site Need for Understanding and Coordinating the Contract Documents and Technical Specifications

Current era: construction very fast paced Errors More Likely Frequent critical reviews are needed more and more to keep up with the pace.

Rising requirements for high quality and corporate effectiveness

Other Benefits of Project Reviews Other effects:- Knowledge Transfer - Team Building

These are positive side-effects rather than pursued goals

Team Building is a much desired side effect particularly for projects that are highly segmented (such as construction projects)

Project Review How to do it?Specific Approaches (Tools) for reviewing projects Peer Reviews Project Meetings and Walkthroughs Inspection Any combo of the above

Project Meetings A Useful Tool for Project ReviewPurposes and Types of Project Meetings

Assessing what has been done, what has happened?Reviews DiscussionMay also include inspections Visual validation

Assessing why it has happened: AuditsHow have resources been used?

How to Conduct a Successful Project MeetingSchedule your meetings well ahead of time to increase likelihood of participation Meetings should have well-defined purpose and agendaKeep project meetings very impersonal!Goal is to focus on problems rather than peopleDiscuss process rather than ad-hoc decisionsKeep focus on needs of projectAvoid holding meetings that are perceived as witch hunting or challenging some project party or team memberRecriminating meetings do not make progress can lead to ongoing rifts among personnel can lead to defensiveness and emotions that can overwhelm rational analysis

Some Categories of Project ReviewsLevel of the Review: Technical vs. managerialWhat is the orientation? Work output-oriented vs. process-orientedStatus of the Reviewers: Internal review vs. peer reviewTool for the Review: Meetings, Inspections, etc.

Project Reviews at Technical LevelFocus on Technical problems and issuesInterdependencies b/n design and construction methodsTypical technical review program at project start-upSystem requirements reviewSystem design reviewPreliminary design reviewTypically, project reviews become more technical over time

Project Reviews at Management LevelFocus on overall practices and processCommunication ChannelsInformation CoordinationTeamwork EffectivenessClient Relationships

Supervision EfficiencyReliabilityContract ManagementLearning Programs

Overall and general (not specific) levels attained for the performance measuresCostQualitySafetyPerformance

Technical vs. Managerial Reviews Pea-Mora ,et. al., 2003DEVELOPMENTOPERATIONS DESIGN,PLANNING FEASIBILITYCLOSEOUT

Output-Oriented vs. Process-Oriented ReviewsWork output-oriented reviewsSeek to identify issues with completed workPrimarily focused on shorter-term issues

Process-oriented reviewsSeek to identify problems with processesTend to be focused on longer-termCan recourse to higher-order reviews

Reviews could be comprised of both

Project Reviews by PeersAn alternative to Project Review by Internal personnelInformal review conducted among specialists in the same field, focused on a particular project aspectFocus is on work rather than project teamBenefits:Early Discovery of Mistakes and Reduction of ReworkAvoidance of Similar Pitfalls in the FutureEnhancement of Team SpiritPromotion of LearningDocumentation Limited to Memos and Duration not Exceeding 30 to 60 Minutes

Project Review by InspectionInspection viewed as one form of formal technical reviewsA formal review: technical or managerial personnel analyses of the quality of an original piece of work product the process itselfInspection in construction: substantial completion, final completion, Importance of documentation and formalism to foster feedback and diffusion of learned lessons

Project Review What form does it take?Major Forms of Overall Project Reviews: Value Engineering Review Constructability Walkthroughs Substantial Completion Inspection Final completion inspectionEach of these forms may involve one of more review tools previously discussed

Forms of Major Project Reviews I Value EngineeringValue engineering is the process of: Identifying and quantifying the performance of various systems in the design of a facility; Evaluating the costs and benefits of alternative solutions that achieve similar or better performance for the same or lower costs

Reviews are necessary for two reasons:Need of extensive communication and collaborationUncertainty in the construction industry

Constructability Walkthroughs Forms of Overall Project Reviews II -

Forms of Overall Project Reviews III Substantial Completion InspectionIs a formal inspection that follows the contractors request and a subsequent number of walkthroughs towards the end of the works.

The project manager, the owner, the engineer and the architect inspect the (almost) completed project, and decide whether it is suitable for occupancy for its intended purpose.

The SCI is a formal, contractually prescribed review, with important legal significance.

Often serves as trigger for release of retainage

Sometimes followed by final completion inspection

Summary Approaches, Functions, and Forms for Project Review

Approaches for Project Reviews: In form, formality and functionality, these range from informal peer reviews to contractually prescribed inspections with legal significance.

Key functions of Project Review: (1) Where has project reached and how/when, (2) quality assurance, and (3) knowledge transfer and learning.

Forms of Project Review established by the construction industry: Value engineering reviews, Constructability walkthroughs and Substantial completion inspections.

Definition: A thorough examination of the management of a project (methodology and procedures, records, properties, budgets and expenditures, and degree of completion. (Meredith and Mantel, 2006)

Difference: Project Review: What problems are there? When did they occur? How severe?Project Audit: Detailed form of Project Review Mostly, Why did the problems occur? More formal and more documented.Audits are the most formal and comprehensive form of project review.

Goal here is to understand why project is in its current stateProject audits may be for the entire project or only for parts thereofProject audits present the best opportunity for formal learning.Project Audits What are they?

Contents of a Project AuditCurrent status of the overall project: Why work actually completed does or does not match the planned level of completion?

Future status: Likelihood of any changes in schedules, cash flows, etc. Why?

Status of critical activities: Why the current and predicted status of tasks could make or break the project

A critical examination of project management issues at all management levels

Contents of a Project Audit (contd)

Risk assessment and analysis: What is the potential for project failure or monetary loss?

Lessons learned from (or to) other similar projects

Assumptions, limitations and quality of data used in the audit

Comments and conclusions by the author of the audit

Who Needs the Audit?The project manager, who seeks unbiased and comprehensive information from groups or individuals within the project organization.

The organization, which seeks to identify the errors made, track their causes, and learn not to repeat them.

The client, who can relate the value of project development to their own actions and decisions.

Any external stakeholders or sponsors of the project; financial institutions, government agencies, consumer groups, environmental or religious organizations and social groups.

Financial, Management and Project Audits Meredith J.& Mantel S., 2000

Focus on Project Auditing Scopes Pea-Mora ,et. al., 2003

Causal Relationships examined in a Project Audit What is affecting What?

Causal Relationships examined in a Project Audit Causal Relationships and the Dynamic Interaction between Schedule Cost QualityStaff Experience & Skill Quality Productivity Project Requirements Organization CultureStakeholders Value Perception Value DeliverySchedule, Cost and Quality Strategic and Tactical DecisionsProduct-Specific Requirements Work Breakdown Structure, Organizational Breakdown Structure, Product Development ModelInformation Flow & Record-keeping Work Breakdown Structure, Organizational Breakdown StructureMonitoring Configuration Corrective Actions Time Delays Ultimate EffectsResource Planning Cost Schedule Performance Hiring Policy Cost Quality PerformanceFinancial Strategy Explicit and Implicit Constraints Imposed Strategic and Tactical Decisions

Project Audit EssentialsIt is essential that a project audit

Succeeds in serving as a Learning Tool that consists of Two Separate Processes:Generation of the LessonAdoption of the Lessons by the Organization or Project Team

Consists of some key ingredients such as:An effective Audit TeamUnfettered Access to Project RecordsCommunication With Project Personnel and OthersAbility of Audit Team to investigate Exogenous Causes of Project problemsThe Audit Should be Truthful and HonestAuditing Performance of Project PersonnelAudit Report Distribution

The Need for Communication With Personnel during an AuditFree flow of information from project personnel to audit team

Two reasons for communication problems:Unavailability of certain personnel Distrust towards the audit team

Personnel mental and psychological training to audits (pre-announced scheduling can help!)

Compromise between friendliness and professionalism of the audit team

Auditing the Performance of Project PersonnelEvaluation of Personnel Performance Productivity, creativity, commitment, quality , response to unplanned situations , leadership, teamwork, adoptability to the project and organizational culture and relationships with other personnel.

Evaluation Often Viewed As criticism and threat to the Profession

Solution:Focus on Situations, Not IndividualsPersonnel Performance Taking Into Account the Organizational Structure and the Culture of the Organization

Audit Report Distribution Selective report distributionImportance of defining a distribution list for the report early in the life-cycle of the auditComprehensibility of the report to all the addresseesImplementation of information technology in report distribution

Phases of a Typical Project Audit

InitiationPlanningExecutionReporting and Release

Phase 1 : InitiationSetting of the Pursued GoalsFormal Determination of Audit Scope: length, formality, the recipient list and the parameters of the project Tasks Involved:Determination of the goals of the auditExpert Team Formation.Establishing the Purpose and the Scope of the Audit.Notification of the Project Team

Phase 2: PlanningThree Major Work Elements:Determine the cost, time and technical constraints that do or will govern the auditing effortDetermine the means and methods to be used in the auditAgreeing on a performance baseline

This Step Includes:building the questionnaires , the checklists and the data gathering formsplanning an interview schedule

Phase 3: Audit ExecutionInvestigation Starts With Data CollectionInvestigation Follows With Data Analysis and Investigation, Focusing On:Assessment of the Project Organization, Management, Methods and ControlsStatement of both Current and Former StatusPreliminary Statement of Forecasted Project StatusWorking Quality AssessmentDelivered Quality AssessmentLessons Learned Action Plan

Audit Execution Followed by Self-Scrutiny Review

Phase 4 : Report Preparation and ReleaseAudit Report should conform to the audit requirements and needsReport should be released according to the distribution planImportance of planning the logistics of report preparation and releaseThe policies for conducting and distributing the report: - balancing political correctness - ease of assimilation of the lessons- truthfulness - brevity.

Phase 5: Finishing up the Project AuditThree Essential Steps:Audit Database and Document Filing by audit service management division Post-Audit Consulting: the last chance for the project team to discuss with and learn from the auditorsAudit Program Evaluation. The effectiveness, quality, sophistication and depth of the audit are examined.- the problems the audit team encountered during their job are also listed

Project Audit ReportA formal document presenting all the work done and the conclusions reached

Table of Contents for a Typical Audit Report:Introduction: Project and Audit Scope, Objectives and CircumstancesCurrent Project Status: entire project ecosystemFuture ProjectionRecommendations about changes in technical approach, budget and schedule for the remaining tasksRisk Assessment and Analysis:Limitations and Assumptions of the Audit : listing of Time, depth, lost records, negative attitudes, focus in specific directions, assumptions and poor inputs

Project Audits - SummaryUse Project Audits to reveal the interactions between elements in the project ecosystem and their effects on the project. Also to investigate the project (or a part of it). In doing so, they must be honest and objective. Only then will they be useful.Plan audits carefully: Establish a baseline for comparison, but be open to in-depth investigation if necessary. If the project is long or complex, plan the Audit as a continuous learning processes. Make audits worthwhile: Introduce the auditing procedure as an integral part of the product development process. Teach people to appreciate it and learn from it. It works for them, not against them.

Questions?

in contrast, major phases: which means more frequentlyThe parties are placed in adversarial positions by the management: by NATUREThe production people tend to cover and hide their mistakes by nature: think tradeoffmore pressure on quality by control team, more hidden errors, lower quality.QM is done in a more COMPREHENSIVE manner.not only customer but also all project functions in supply chain including suppliers.is it good for quality of product? yes. WHY? good relationships with suppliers can lead to good qualityraw material and labor source. for example, some big construction companies are now taking care of their subcontractors, providing technologies and financial supports..Most product development industries, especially those developing less tangible products like software, use reviews as the primary practical means of checking and assuring quality and progress, overcoming technical difficulties and coordinating processes. The design and construction industry, dealing with very tangible products, was slow to adopt any form of formal reviewing. Rather, coordination in the building industry is carried out consciously but quite informally. The need for the transition to formal review processes in the design and construction industry is manifested in the findings of relevant research. According to Mendelsohn, about 75% of the problems encountered in the construction site are generated in the design phase with the preparation of unclear, inconcise and often conflicting drawings. Initial planning and design is also responsible for uncoordinated contract documents and technical specifications. The need for reviews emerges also from the current pace of building construction, which has significantly accelerated in the recent years and has become prone to errors due to inadequate understanding of the design concept or to last-minute changes in the construction sequence. Finally, formal reviews have gained popularity among design and construction firms through Total Quality Management programs, which were triggered by the rising requirements for high quality and corporate effectiveness. In a relevant study, Josephson et al summarize findings of other researchers reporting non-conformance and quality-related rework costs in construction rising up to 10-20% of the total project cost. Josephson also verifies that most construction problems root in the design phases. These findings provide incentives to employ formal review programs in design and construction projects. Most product development industries, especially those developing less tangible products like software, use reviews as the primary practical means of checking and assuring quality and progress, overcoming technical difficulties and coordinating processes. The design and construction industry, dealing with very tangible products, was slow to adopt any form of formal reviewing. Rather, coordination in the building industry is carried out consciously but quite informally. The need for the transition to formal review processes in the design and construction industry is manifested in the findings of relevant research. According to Mendelsohn, about 75% of the problems encountered in the construction site are generated in the design phase with the preparation of unclear, inconcise and often conflicting drawings. Initial planning and design is also responsible for uncoordinated contract documents and technical specifications. The need for reviews emerges also from the current pace of building construction, which has significantly accelerated in the recent years and has become prone to errors due to inadequate understanding of the design concept or to last-minute changes in the construction sequence. Finally, formal reviews have gained popularity among design and construction firms through Total Quality Management programs, which were triggered by the rising requirements for high quality and corporate effectiveness. In a relevant study, Josephson et al summarize findings of other researchers reporting non-conformance and quality-related rework costs in construction rising up to 10-20% of the total project cost. Josephson also verifies that most construction problems root in the design phases. These findings provide incentives to employ formal review programs in design and construction projects. Knowledge transfer and teambuilding are more of positive side-effects of reviewing rather than explicitly pursued goals: Teambuilding alone cannot be the reason for holding a review, or it will probably be a dull waste of time and resources. In a segmented industry like design and construction, teamwork and learning are vital for high-quality employee, peer and client relationships, high design and construction quality, and conformance to cost and schedule. In their technical version, peer reviews are most suitable for promoting individual and organizational learning within the limits of a workgroup. If external experts are invited, peer reviews can also facilitate the import of knowledge in the organization (although this can be hindered by competitive relationships between the organization and the external reviewers). Broadening the definition of peer reviews to include stakeholders from the entire project ecosystem brings us to managerial peer reviews, a powerful facilitator of learning and teambuilding in the construction industry. The segmentation in the design and construction industry provides plenty opportunities for misunderstandings, adversarial relationships between stakeholders, poor transfer of knowledge and learning. The documents alone that define the functioning of the parties involved, rarely succeed in communicating true intents. Design drawings and specifications only provide directions to the contractor they do not convey the design rationale. Contracts and specifications are lengthy, sometimes excessively detailed, written in tiresome legal or technical language, and aimed at covering the parties interests in as many possible situations as possible. Reviews, informal peer reviews in particular, can effectively communicate ideas and intents relevant to every aspect of a project its mission, its goals, its objectives and its strategy to optimize project management .In a nutshell, Reviews are instruments for solving technical and managerial issues in design and construction. They are used for bridging gaps in project development, validating work done, quality assurance and learning. Different review configurations (peer reviews, walkthroughs and inspections) are used for different purposes. Some are very informal, while some are contractually prescribed and have legal significance. In construction, established review processes include the value engineering review, the constructability review and the substantial completion inspection. Technical reviews are frequent during the initial planning, preliminary design and detailing phase of a construction project. Their focus is on technical issues but their scope extends to more than solving problems or verifying technical solutions given. Technical reviews focus also on the life-cycle economics of the operational structure and the interdependencies between the design and the construction methods.A technical review program starts very early in the life cycle of a project. In what can be generally described as a System Requirements Review (SRR), the designers collaborate with the customer to establish the requirements for the structure or system. The nature and logistics of this initial technical review depends on the sophistication of the customer, the delivery system and the nature of the project. Sophisticated customers can be very explicit about their requirements, especially for simple projects. On the other hand, complex structures whose economic sustainability depends largely on technical issues require much more extensive effort up-front. A System Design Review (SDR) follows the SRR for the purpose of presenting the conceptual solution that satisfies the requirements set forth in the SRR. The preliminary design is approved by the customer and/or other technical experts in a Preliminary Design Review (PDR). Further downstream, as the design goes into detail, reviews become more technical and peers of the design team and construction experts often replace the customer in meetings .Project management reviews are not focused on the technical aspects of a project, but on the techniques and practices and processes followed for its implementation. These include measurable quantities such as cost, quality, safety and performance as well as less tangible features like the communication channels, information coordination, teamwork effectiveness, client relationships, supervision efficiency and reliability, contract management and learning programs. Formal review programs incorporate series of both technical and managerial reviews, two sequences which evolve in parallel. In practice, technical review programs focus heavily on the initial planning and design phases of a project, whereas project management reviews are more frequent during construction . However, both technical and managerial reviews yield direct and indirect benefits for the success of a project and the efficiency of the organization. A peer review is conducted among specialists in the same field, usually engineering. The objectives of a peer review range from verifying that a work unit conforms to its requirements, to detecting faults, violations of development standards and other problems. Usually, the term refers to an informal yet focused investigation into a particular aspect of a project. Some contracts however, especially for complex engineering design, require that formal peer reviews verify the competency of the engineering solutions given. These reviews are formal in that they produce a report and make the reviewers (who are usually third parties, not associated with the development organization) responsible for their statements and conclusions.While peer reviews originate from engineering environments, they can also be focused on small-scale management problems, like a situation in logistics or accounting. If the review takes the form of an interview, it is an informal discussion on the work of the interviewee. Peer reviews are common practice in any project environment, and cannot (and perhaps should not) be formalized. It is the smallest feedback loop found in an engineering or management environment, and its informal character is essential to the benefits associated with it.The benefits of an informal peer review are: Early discovery of mistakes: An early discovery of mistakes reduces the amount of rework associated with design errors, production management-related causes and technical breakdown of machinery. Avoidance of similar pitfalls in the future: Spotting and correcting problems draws everyones attention to them. People learn and do not repeat the same or similar errors. Enhancement of team spirit: Peer reviews enhance team collaboration and bonding, because the focus is on the work, not the project team. The absence of superior personnel and the informal atmosphere draw the emphasis away from personal appraisal and criticism. Promotion of Learning: Peer reviews promote learning in two ways: First they facilitate extending individual learning from mistakes to team learning. Second, they can be used for hands-on training of inexperienced personnel. New members of the team are exposed to other peoples thinking and work, and thus knowledge is passed on and disseminated within the organization. It is generally agreed that peer reviews should be conducted according to need rather than follow a schedule. Managers can try to enhance team spirit among project personnel hoping that team members will become more eager to hold peer reviews on their own initiative. On the other hand, managers (or other higher-rank personnel) should not participate in peer reviews: participants should be of equal rank. A peer review meeting is most efficient when participation is kept below 3-4 people, and there is no need for a secretary or a moderator. Documentation can be limited to memos, and the duration of the meeting should not exceed 30 to 60 minutes: learning and focusing capabilities are lost after this long, together with half the benefits of holding a meeting in the first place. Peer meetings are most productive when the benefits derive from the personal interaction between the individuals and the vibrant exchange of experiences. In sum, peer reviews yield the highest benefits when they are called for team building and problem solving.An inspection is one form of formal technical reviews.A Formal Review (FR) is a sequence of structured meetings in which a group of technical or managerial personnel analyses or improves the quality of an original piece of work product as well as the quality of the process itself. The process is very well defined hence the name formal. It is split in distinct phases, each of which involves its own procedures. Qualified personnel are assigned specific roles, and all work together towards well-defined goals. In cases where the review subject allows it, the process uses and subsequently produces quantifiable statistical measurements. For this reason, standard forms and consistent data collection are elements closely associated with formal reviews.A formal review may be focused on managerial (FMR, e.g., for adherence to schedule, criticality of constraints, quality of supervision), logistical FLR, e.g., for materials management) or technical issues (FTR). A Formal Technical Review (FTR) is a version of the process oriented and adopted to address technical issues. FTRs are conducted according to the schedule of completion of major milestones. As the name implies, they are focused on the degree to which technical standards are met during project development, the need for changes and the effects of changes already made. An inspection is one form of formal technical reviews. It is conducted after the completion of major milestones in a certain aspect of the project. The process was originally introduced by Fagan (IBM) in 1976 for the purpose of code reviewing, but has also been implemented in projects ranging from civil and mechanical design to manufacturing and software development. The inspections original scope was to find and remove defects in any formal document (specifications, source code, contracts, test plans, test cases, shop drawings). The most often-encountered inspections in construction are the substantial completion inspections, in which the structure is declared to be occupied for its intended purpose, although work needs still to be done. The substantial completion inspection is a multi-stage, formal process with legal and contractual significance .During an inspection, the work completed is checked against a documented list of concerns. These concerns derive from statistically repeated errors, issues that have come up in previous walkthroughs, or statements in the specifications and requirements associated with high risks and high importance to quality. In a way of feedback, the information revealed in an inspection should be documented in such a way as to provide material for improvement of both the development processes and future inspections. In short, it should be exploitable in a statistical way. Without the formalism of the inspection process, feedback accumulates in the experience of the reviewers and does not diffuse in the organization as lessons learned. In effect, the review process itself does not improve. The formalism of inspections, apart from its legal significance, is aimed to improve both the quality of project development and the review process itself. In a nutshell, Reviews are instruments for solving technical and managerial issues in design and construction. They are used for bridging gaps in project development, validating work done, quality assurance and learning. Different review configurations (peer reviews, walkthroughs and inspections) are used for different purposes. Some are very informal, while some are contractually prescribed and have legal significance. In construction, established review processes include the value engineering review, the constructability review and the substantial completion inspection. The most important reviews in construction are the value engineering review, the constructability review, and the substantial completion inspection.Value engineering is the term used for the process of identifying and quantifying the performance of various systems in the design of a facility, and evaluating the costs and benefits of alternative solutions that achieve similar or better performance for the same or lower costs. The corresponding analysis, value analysis, is performed for a time horizon comparable to the life of the project. Value engineering consulting is performed either by experts, called value engineers (or value analysts), and can be led by an experienced individual in the designers or contractors team. The value engineering team needs to be interdisciplinary, because the value analysis needs to comprehensively touch every major system in the facility.Reviews are necessary in value engineering for two reasons: First, because the value analysis team needs to be interdisciplinary, where extensive communication and collaboration is required. This need is usually fulfilled with peer reviews between team members.Second, some of the requirements for civil engineering systems are subject to a lot of uncertainty and their cost and benefits are hard to determine. Other aspects of design are simply impossible to express in cost terms. For example, the siting decision for a new highway depends on the current and projected traffic load, the current and projected condition of the landscape, the desired flexibility for upgrading, long-term maintenance concerns, urban development projections and constitutional restrictions, among others. The cost and benefit of these factors, although easily quantified in theory, is very difficult to estimate. On the other hand, the owner of a new commercial building will need to make decisions on how much beauty, prestige or good community relations should cost her. These decisions rarely fit into analytical optimization or decision models, and extensive collaboration between the owner and the value engineering team is necessary. This collaboration is achieved with peer reviews and walkthroughs of alternative designs.Of all the reviews and walkthroughs the engineers, architects and the owner perform on-site, perhaps the most significant is the substantial completion inspection (SCI). It is a formal inspection that follows the contractors request and a subsequent number of walkthroughs towards the end of the works. During the SCI, the project manager, the owner, the engineer and the architect inspect the (almost) completed project, and decide whether it is suitable for occupancy for its intended purpose. A list of remaining work items is given to the contractor, which must be completed within a specified time frame. The SCI is a formal, contractually prescribed review, with important legal significance. Reviews can be useful for bringing together technical and managerial personnel to examine prior work and projections into the future.Technical reviews must be separated from managerial, mainly for logistical reasons. Some technical input is required in managerial reviews though, and vice-versa.In form, formality and functionality, reviews range from informal peer reviews to contractually prescribed inspections with legal significance.The most important functions of reviewing are (1) passing progress gates, (2) quality assurance, and (3) knowledge transfer and learning.The construction industry has established value engineering reviews, constructability walkthroughs and substantial completion inspections as regular reviewing instruments. In a nutshell, project audits are formal and comprehensive reviews. The major purpose of audits is to provide insight into some or all interacting elements of the project. To achieve this, they have to be planned and organized as standalone projects with an articulated goal, a life-cycle, a baseline and specific deliverables. Audits represent the most formal learning opportunity for the organization. Project audits should include at least the following elements:-The current status of the project in terms of cost, schedule, progress and quality measurements, as well as environmental, social and political impacts, with special attention given to critical (at the time the report is created) tasks.-Anticipated future project developments.-Critical management issues that need attention at all three (strategic, tactical and operational) levels.-Risk analysis.-Assumptions, limitations and quality of data used in the audit.-Comments and conclusions by the author of the report.The conclusions of the final audit report are meant to be used in the remaining phases of the project and for other projects. The project audit can forecast the next parts of a project and can offer solutions for problematic issues and areas of possible improvement. With regard to the current and projected performance, management issues and risk analysis, an audit should focus on the following components:-A study of the initial proposal (plans, specifications, requirements, as well as implicit goals of the project for the organization and the stakeholders).-A comparison of the achieved performance with what was planned.-Evaluation and interpretation of the results for the stakeholders (including the organization)-Suggested actions to correct variances.-Highlight the nuggets of the project for future reference.The content and timing of an audit depends largely on who the stakeholders are, who requests it, and its intended purpose. Therefore, it is very important to identify the parties that usually shape the auditing procedure and content :-The project manager, who seeks unbiased and comprehensive information from groups or individuals within the project organization.-The organization, which seeks to identify the errors made, track their causes, and learn not to repeat them.-The clients, who can relate the value of project development to their own actions and decisions. -Any external stakeholders or sponsors of the project; financial institutions, government agencies, consumer groups, environmental or religious organizations and social groups.Each of the groups above can benefit from an audit. For organizations and customers, the project audit creates a structured feedback mechanism regarding the project, suggesting self-correcting actions. A successful audit extracts subtle information about the project, which cannot be easily linked to performance otherwise. For example in the construction industry, an audit can link overall performance to the delivery method used by investigating questions that are not usually asked in other forms of review. Such mid-progress audits can promote learning and future project performance. Finally, customers can discover the mechanisms by which their own involvement in the project affects value creation.For the other stakeholders, an audit provides more than dry numerical cost or schedule information about the project. The information included in an audit can potentially prove useful to external stakeholders whose interests are not shared by the development organization. For this reason, the audit report is sometimes characterized as confidential, or deliberately omits sensitive information. Finally, a full fledge, comprehensive audit , is time-consuming, expensive and difficult. The audit has to balance the trade-offs of being thorough and focused on a particular problem or mode of project behaviour. Thus, vast experience in the industry is required and the scope and goals of an audit should be defined in advance.A financial, a management and a project audit are three separate things.Management audits examine the organizations management while project audits are focused on the impact of the management organization on a particular project and vice-versa. Moreover, both financial and project audits may share processes and investigation procedures. However, they have a different focus. Project audits focus, among other things, on the financial performance, the expenditures and the time spent on the project by the time of the audit, as they compare to the plan. Financial audits focus on the use and preservation of the organization assets. The scope of the project audit is broader compared to the limited scope of the financial audit. The table shows the difference between the three types of audits. A full-extent audit is a very time-consuming and costly procedure, and it is not usually conducted unless there is a reason to do so. It may be sufficient to audit one integrated and complete part of the project the one that seemingly caused problems or the one that has more lessons to offer. This way, organizations can achieve the best return on the process of auditing; i.e., minimize its cost while maximizing the benefit. Cost need not only be expressed in terms of money. The parameters that can be identified as costs associated with a project audit are the following:Time: More often than not, time is seen as a constraint rather than the cost parameter. Indeed, audits are time-consuming, and projects are short of time. Therefore, it is the time constraint that partly dictates the depth of the audit. The relationship between time allocated and depth of the audits should be identified in the beginning of the audit life cycle if the audits are pre-planned, or mid-way through the project, whenever the audit is ordered. If the audit is a final evaluation of the project, conducted after it is over, then time may not represent a constrain, but rather a direct cost. In that case, the organization has to decide what can be the best benefit-cost ratio for an audit. Note that the time needed for an audit depends on the nature of the project, its size, its complexity, whether it is localized or geographically dispersed, and the degree of completion. As the value of an audit changes according to the stage on the life cycle of a project that it is performed, the degree of completion must be taken seriously into account when managing the trade-off between time and depth of the audit process .Money: The budget for an audit, and therefore its depth and extend, are most often determined by its requester. In any case, the expected cost must be justified by the expected benefits. These may vary from claims settlement to pure learning. However, in general, pre-planned audits tend to cost less than exceptionally requested ones: The average cost of an audit decreases, as the procedure is standardized and follows a predetermined baseline. Impact on Performance: In the case of project audits conducted in the course of a project, the price paid for the audit is effect on personnel performance. Apart from the time the personnel has to spend to provide the audit team with all the necessary information, and the distraction this may cause, their morale is adversely affected. These effects can be alleviated with the use of information technology and pull information systems. Nevertheless, the more comprehensive, deep and time-consuming the audit process is, the greater its effect on personnel performance.The time and money required and the impact on personnel performance are reasons for wanting to keep the auditing process as small and focused as possible. However, this contradicts the character of an audit as a holistic evaluation; if it is to fulfil the objectives it is designed for, the audit must address at least one relatively independent aspect of the project comprehensively. Depending on the goals of the audit and the constraints mentioned above, organizations choose to conduct the following types of audits:The General Audit (or Overview) is usually carried out after the completion of milestones specified by predefined contractual agreements. The resulting report superficially covers different issues such as current project status, future project status, status of critical tasks, risk assessment and limitations and assumptions of the process. An overview is conducted when the available time and money pose significant constraints. The Detailed (or Administrative) Audit goes a step further than the overview. It examines some of the issues covered in the general audit in greater depth. The focus of the administrative audit may be on factors that offer themselves as lessons, cause problems or exhibit risky behavior. Care must be taken so that the detailed audit does not place factors out of context. Often, it will not be possible to isolate one aspect of the project and examine it independently; doing so may cause the auditor to miss cause-and-effect relationships and subsequently lead to wrong evaluations.The Technical (or Technical Quality) Audit may be independent of the general overview of the administrative audit, as long as the technical aspects audited in loosely woven with the managerial ones. Some construction projects of low (or platform) technology, such as steel structures, may be audited independently from a technical perspective. The same applies to projects where the management methods completely (or almost completely) follow the technical issues, such as in some R&D projects. Technical audits are usually conducted early in the lifecycle of a project, because such decisions are taken early.The second primary goal we will consider is to clarify the causal relationships within the project ecosystem.The snapshot of the project ecosystem is valuable but insufficient information: it is static and lacks information about the dynamics of the ecosystem or the evolution of its elements. In addition to the time history of the relevant parameters up to the point of the audit, their causal relationships and dynamic interaction, even if not quantified, should be investigated and explained. While identifying the causal structure of the project ecosystem requires skill, experience and is extremely project-specific, some of the relationships that should draw the auditors attention are:-Schedule Cost Quality-Staff Experience & Skill Quality Productivity -Project Requirements Organization Culture-Stakeholders Value Perception Value Delivery-Feasibility of Current Schedule Project Status-Schedule, Cost and Quality Natural & Sociopolitical Environment-Schedule, Cost and Quality Changing Customer Requirements-Schedule, Cost and Quality Strategic and Tactical Decisions-Product-Specific Requirements Work Breakdown Structure, Organizational Breakdown Structure, Product Development Model-Information Flow & Record-keeping Work Breakdown Structure, Organizational Breakdown Structure-Monitoring Configuration Corrective Actions Time Delays Ultimate Effects-Contractual Arrangements Work Breakdown Structure, Organizational Breakdown Structure, Cost, Product Development Model-Resource Planning Cost Schedule Performance -Hiring Policy Cost Quality Performance-Financial Strategy Explicit and Implicit Constraints Imposed Strategic and Tactical DecisionsA project audit should be seen as a learning tool. The lesson may benefit the development of the project or the organization itself for handling future projects more effectively. As a learning process, it is comprised of two separate processes: generation of the lesson and adoption of the lessons by the organization or the project team. Both processes are difficult to achieve; the lesson will not be generated if the project team does not give away data, regardless of how good the auditor is. Likewise, the lesson will not be learned if it is not presented the right way by the auditor. However, there are some recipes for success in both processes:-The Audit Team-Access to Project Records-Communication With Project Personnel and Others-Exogenous Causes Should Be Sought Inside the Organization-The Audit Should be Truthful and Honest-Auditing Personnel Performance-Audit Report Distribution

The relationship between the audit team and the personnel is of paramount importance, particularly in unplanned audits, when something has gone wrong, or when the auditing team is not from the organization itself. We identify two reasons why communication problems between the audit team and the personnel may arise: (1) Unavailability of certain personnel (as in the case of geographically dispersed project teams), and (2) distrust towards the audit team. Trust and respect towards the auditors work builds up slowly. Experience has taught that it is most important that everyone in the project team is notified about an audit in advance. Personnel must be trained both mentally and psychologically to the purpose the audit is designed to achieve, the methods that will be followed and the background of the auditing team. It is even better if the auditing team is formally introduced during an initial meeting.During the process of the audit, the audit team must work their way into obtaining information from the project personnel . Some intimacy and friendliness between the auditors and personnel is essential for the latter to open up and share information. On the other hand, the audit team must retain their professionalism and persuade that proper information flow and authority is maintained during the audit. In the spirit of maintaining the proper information channels, the audit team must obtain management clearance before contacting the customer, contractors or other parties other than the project team. This includes parties within the organization who are not directly involved in the project.One goal of the audit is to evaluate personnel performance, in terms of productivity, creativity, commitment, quality (errors made), response to unplanned situations (for higher ranks), potential for leadership, teamwork capabilities, adoptability to the project and organizational culture and relationships with other personnel. Such evaluations are never welcome because the personnel perceive them as a criticism and a profession threat. Consequently, they will hide or distort information to protect their own standing, thus compromising the auditors work. What is more important, personnel will probably not be receptive to learning after being severely criticised. Evaluation should be formulated so that it focuses on situations, not people. It needs to take into account both the organizational structure and the culture of the organization. That is, personnel performance should be compared to the requirements posed by the official line of responsibility, the established organizational breakdown structure and the culture of the organization. Personnel should not be regarded responsible for actions outside their scope of work. Auditing personnel performance is a delicate matter. It is easily turned into personal praise or criticism. The auditor should avoid presenting the evaluation as a personal issue. The tone of the entire procedure for personnel evaluation should be oriented to explore space for improvement in project management practices, not individuals.The audit report should not be available to everyone in the organization. Who reads the audit report directly depends on who requested the audit in the first place, what the goals and outcomes of the audit are, the status of the project, the relationships with other parties (customers or contractors), knowledge propriety issues, and the culture of the organization. It is advisable to define the distribution list for the report early in the life cycle of the audit. Effectively, the process is configured to address the parties that will ultimately benefit from it. Care must be taken to make the audit comprehensible to everyone in the list, i.e., balance the detail in technical, financial and managerial information to convey the lessons learned, without overwhelming the readers. Finally, the information infrastructure that will be used for the distribution of the audit report must be established. It can also affect the format and distribution list. The decision must be made as to whether the information will be push (i.e. forwarded to the recipients) or pull (i.e. available to the recipients on demand). The use of information technology has taken the idea of a recipient list further. Some organizations segment the audit report so that all personnel have access to a part of it, while other parts can only be retrieved using a restricted access password. This technology can even be used to personalize some parts of the report, to address specific individuals or teams while they are not read by others.The initiation of an auditing program sets the goals pursued. These goals do not have to be in a formal statement, but they have to be commonly agreed on by all the levels of management in the organization and all the major stakeholders. Once the aims of the audit are identified, its scope must be determined, including the length, the formality, the recipient list of the report and the parameters of the project that it will include. This has to be formal. A clear definition of the audit baseline can prevent many problems associated with personnel and stakeholder reactions. Initiating an audit can become a fairly standardized process, requiring specific tasks:Determination of the goals of the audit: Initiation is signified by the request for the audit, which in turn must be accompanied with a preliminary list of goals. They can be prescribed in the contractual agreements or they can be exceptionally formulated. Expert Team Formation. According to the goals, the requirement in expertise, availability of personnel and cultural factors of the organization, the leader of the audit team is assigned. With her help, the formation of the project audit team takes place. It is important that it is credible and respected by all the parties in the recipient list of the audit report, and that it is qualified to produce a realistic and educated evaluation of all the aspects defined in the scope. Establishing the Purpose and the Scope of the Audit. The responsibility for the audit has passed to the hands of the audit team. The project auditor should meet with the audit requester to determine the details of the purpose, scope and constraints in the process.Notification of the Project Team. Even by now, the project team may not have been notified of the audit plan. It is best if the audit program is communicated to the entire project team. This can be done in a meeting where an outline of the audit requirements and expectations is made public to project personnel. Planning an audit program is comprised of three major work elements:-Determine the cost, time and technical constraints that do or will govern the auditing effort. Just like a project manager, the auditor must then optimise the use of resources to get the results of maximum value to the project and the organization.-Determine the means and methods to be used in the audit. For audit programs, standardizing the methods can really save the organization money and time, and it can provide a standard basis for data and comparison.-Agreeing on a performance baseline. The baseline has to be established from the start of the audit program. It should be constructed in such a way as to be useful to the organization and the other interested parties, and also provide information about the evolution of selected parameters between audits. The baseline can be defined with respect to the contracts (which have to be examined and analysed in terms of quality, expenses, milestones and natural and socio-political effects). The contracts can also provide a perception of the risk policies adopted in the project and the potential dangers. The baseline can also include an examination of the value proposition of the project in relation to the stakeholders expectations, because these expectations often drive the evolutions of large projects.Developing the detailed audit program constitutes the principal objective of the second phase. The milestones that will be followed by audits, their specific content and intent as well as the details of the methods towards the audits goals are specified. This step includes building the questionnaires, the checklists and the data gathering forms, and planning an interview schedule. The baseline definition is to an audit what the detailed planning and design is to a project.The audit program is executed every time an audit is held. Depending on the important issues at the time an audit is conducted, the audit team will need to deviate from the baseline design and improvise, in order to make the investigation more effective and realistic. The outline of the audit investigation starts with the collection of data. The investigation proceeds with their analysis and interpretation, focusing on a selection of the following issues:-Assessment of the Project Organization, Management, Methods and Controls. The assessment is driven by three criteria: How these parameters affect the development process and value delivery;(2) whether they are effective in doing so(3) how they have changed since the last audit (or initiation) and why.-Statement of both Current and Former Status. The project metrics, as established in the baseline definition, should be used to assess the progress between milestones. Again, progress should be compared to plan, and reasons for deviations should be sought.-Preliminary Statement of Forecasted Project Status. As part of the consulting role inherent to an audit, attempts to forecast future trends of significant parameters should be made. -Working Quality Assessment. Quality, as a fraction of tasks executed correctly over tasks that need rework, is assessed with regard to all project management and technical processes. Reasons for low quality are sought.-Delivered Quality Assessment. The required quality of delivered work fluctuates sometimes, in light of technical difficulties or inputs from the sociopolitical and natural environment. Low delivered quality can be an important cause of problems in project development and must be monitored.-Lessons Learned Action Plan. Most managers will like to see an explicit action plan following the recorded information in an audit. The completion of the audit execution is followed by a self-scrutiny review. The audit process needs to be adaptive. The appropriateness of the established baseline must be reviewed in the light of the new data. If needed, the baseline may change to fit the actual progress of the project better.The audit report should be prepared according to the audit requirements and needs. It is released according to the distribution plan. Planning the logistics of report preparation and release, the audit team should make sure that they do not surprise the project manager and the stakeholders. Surprising the stakeholders with aggressive audit reports is sure to compromise the primary function of the audit: Learning. The audit team can call last-minute interviews with each of the interested parties, during which they can present their conclusions unofficially, verify their sources of data and confirm the stakeholders consent with the content of the report. A preliminary draft could also be discussed during these meetings. On the other hand, the audit team must not let the (higher ranked) stakeholders intimidate them. The policies for conducting and distributing the report must balance political correctness, ease of assimilation of the lessons, truthfulness and brevity. The audit program requires a closeout process to ensure that its entire value is delivered, and that it can be referenced after the audit team has broken up. The essential steps are three:Audit Database and Document Filing. The copies of the documents used in the audit should be filed and stored appropriately for future reference. Compiled information, conclusions and the audit reports should be filed separately, since they should be referenced more often. The audit service management is the division of the organization that is responsible for the audit administrative functions. This division can be a virtual entity, manned by staff with other primary occupations. They are responsible for the documents management following the audits, accounts settling and reimbursement and statistical information regarding financial, distribution, feedback and control issues.Post-Audit Consulting. Although this may not be the rule, in some cases post-audit consulting may be needed. In most cases, it cannot extend too long after the last audit because the audit team is inevitably broken up. Post-audit consulting is the last chance for the project team to discuss with and learn from the auditors. Audit Program Evaluation. The effectiveness, quality, sophistication and depth of the audit are examined. Also, the problems the audit team encountered during their job are also listed, as most of them will be directly due to the organizations culture and practices. Part of the lesson learned for the organization is how to be more receptive to audits. The audit program itself must be evaluated so that future audits can be more effective.The audit report is a formal document that presents all the work done and the conclusions reached. Often, the format of the report is dictated by the contracts, but even if it not, some formalism must be followed so that reports of different audits or projects can be compared. The report should at least include the following parts:Introduction: In the introduction, a general description of the project is given, along with the circumstances under which the audit is conducted (regular or unplanned and who the audit requester is). The project scope and objectives and the audit scope and objectives should be presented in the introduction.Current Project Status: The audit report should address the current status of the project in terms of cost, schedule, true progress, quality, scope, and the natural and socio-political environment, i.e., the entire project ecosystem. The account should be from a critical perspective, and comparisons to the established baseline should be included wherever possible. Future Projection: This part of the project audit report should contain predictions regarding the future progress that can be drawn from historical information and the understanding of the project ecosystem. Predictions regard the entire project ecosystem, for which an adequate level of understanding has been obtained, but should not include speculations. Predictions should be justified using accepted tools and methods. Recommendations: Recommendations can be made about changes in technical approach, budget and schedule for the remaining tasks. The quality and scope of recommendations must be consistent with the extent, quality and type of audit, and the level of understanding obtained. The audit team should not attempt to provide recommendations for policies that were not extensively examined during the audit: This would compromise the credibility of both the team and their work.Risk Assessment and Analysis: Although double-guessing the initial risk assessment is not a direct goal of the audit or the report, its effectiveness should be evaluated by comparison of the risk factors identified, and the realized evolution of the project ecosystem.Limitations and Assumptions of the Audit: The last part should include all the limitations and the constraints that the project auditor faced. This section is necessary for further evaluation and improvement of the auditing process. Furthermore, it provides a basis for measuring the credibility of the audit. Time, depth, lost records, negative attitudes, focus in specific directions, assumptions and poor inputs should be listed, with an account of how these factors affected the data and the outputs. Finally, the audit report should include a complete list of references and sources of information.