How to Reduce Fear of Discretion in Public Procurement

Lessons Learnt from Procurement and Contracts Management of Large and Complex Contracts Financed by Multilateral Development Banks (MDBs): A Procurement Practitioner’s Perspective
Devesh C. Mishra and Belita Manka 1

  1. Introduction: This paper seeks to analyze the impact of fear of discretion in procurement and contracts management of large and complex contracts in MDB financed projects. “Fear of discretion” is a term originally penned by Steven Kelman in early 1990s2, and as this paper shows, is still a pertinent topic nowadays. The “fear of discretion” displays itself in different forms and contexts, including in procurement contracts financed by governments or international development organizations, and in different geographical regions where MDBs finance investment projects. The paper identifies various aspects in all phases of project implementation and analyses how lack of exercising good judgment and healthy discretion impact the overall project quality and delivery.  First and foremost, these aspects relate to situations of lack of professional judgement where compliance with the applicable rules, or non-economic influences and considerations, prevail over the development outcomes. The paper further explores how within the boundaries of rules, there is room for the procurement officials (both within the borrowing government and by reviewers in MDBs, where applicable) and how to work with flexibility to obtain quality goods, works and services for a better outcome and effective use of public money and, finally few suggestions are given as to how to reduce fear in exercising discretion.

  2. Background:  Procurement is an integral part of any country’s budget execution process. As reported by numerous sources, procurement represents from 15-30 % of GDP3.  MDBs, for their part, finance billions of dollars each year in developing countries in the form of loans, credits or grants. The World Bank finances on average between US$ 14 to 20 billion worth of procurement contracts each year4. The financing is spent through procurement of activities ranging from textbooks, vaccines, feasibility studies, project management to construction of highways, rural roads, and power plants. Procurement as part of project implementation and service delivery starts from the time the need for goods, works or services is identified till the need is satisfied meeting the quality, economy and efficiency criteria. Throughout these stages, there is undoubtedly a need and scope for professional judgment and discretionary decision-making on behalf of the government, at all levels.

    Public procurement systems, directives and model procurement rules, guidelines and regulations by international institutions like the European Commission (EC), UNCITRAL and MDBs have matured over the last two decades, in many respects by constant interaction among these institutions and consequent harmonization. Either implicitly or explicitly these instruments have increasingly provided for the procurement officials to exercise proper discretion for a better procurement outcome. In particular, the World Bank procurement rules have evolved over the years to reflect changes to the World Bank’s approach to lending and to accommodate the needs and nature of procurement activities financed by its projects.

    The public officials responsible for procurement need to be adequately trained and empowered to exercise discretion and professional judgement as necessary to ensure the successful outcome of the project, while ensuring that all such decisions are transparent, fully documented and justified.  Development of skills to exercise professional judgment (or discretion) without fear becomes more relevant for public officials (as also by MDB staff responsible for review) when procurement rules and regulations are guided by procurement principles of: value for money, economy, integrity, fit for purpose, efficiency, transparency and fairness.

    There are several publications5 on this subject of “Fear of Discretion in Government Procurement” citing examples from advanced economies and instances of “procurement failures”.  However, the scope of this paper is limited to examination of issues in the client countries of MDBs. It is presented from the perspective of a project implementer and procurement practitioner on the side of the government as also a development partner and reviewer on MDB side, termed in this paper as “Procurement Practitioner”.

  3. Critical Procurement and Contract Management Issuesin MDB-financed Projects:  The Procurement Practitioners’ fear of exercising discretion in MDB-financed projects can often result in a number of critical issues that warrant attention:

    • Why fear of exercising discretion or professional judgment does not lead to selection of best contractors/consultants and quality of performance of selected firms;
    • Why those responsible for procurement and contract implementation “tend to be safe rather than sorry”;
    • What are the consequences of not exercising sound judgment in particular in procurement and contract implementation of large and complex projects;
    • Why so little time is spent in choosing the procurement method to regret later on the adopted choice leading to waste of time and money;
    • Why Abnormally Low Tender (ALT) is accepted without due examination;
    • Why standards for determination on responsiveness of bids (like responsiveness of bid security, major or minor deviations in technical bids) differ between MDBs and that of the members of evaluation committee of the government agency leading to waste of time in decision making;
    • Why no decision is taken on valid contract claims by the Employer leading to arbitration and court cases;
    • Why contract closing procedures do not formally exist and contracts passively “expire” rather than actively “closed” in terms of contract provisions;
    • Can it be assumed that procurement officials somewhat tend to take risk and exercise discretion in MDB-financed contracts compared to contracts financed by the government? If yes, why? What could be done by MDBs to encourage and support reasonably grounded exercise of discretion by borrowing countries?
  4.  
  5. Discretion in World Bank/MDBs funded projects: The Black’s Law Dictionary defines “discretion” as “[as applied to public officers] connotes action taken in light of reason as applied to all facts and with view to rights of all parties to action while having regard for what is right and equitable under all circumstances and law.” For some, discretion has a bad connotation, and the increased use of “discretion” in public procurement is perceived as increased risk to the governance and transparency of public procurement. But for us, the procurement professionals, the term means nothing more than “using good professional judgment” within the bounds of rules. As shown below, through concrete examples from MDB financed contracts, there is an overall reluctance to exercise the professional judgement.

    In government funded procurement, such discretion is expected to be enabled both through the legal rules, as well as the culture and the overall authorizing environment in the public administration. However, in developing or underdeveloped economies often there are situations of imperfect legal and legislative frameworks, weak institutional capacity and inadequate capacity of public administration, lack of interest from foreign bidders, or governance and integrity issues. In certain countries it could be just lack of interest and incentive on the part of government officials in timely decision-making in the face of actual or perceived “fear” of adverse action from control bodies within the government. Senior officials and staff at different levels are reluctant to take decisions and avoid taking risk.

    In MDB-financed projects, procurement is carried out based on the MDB’s procurement rules. For example, the World Bank’s rules are made applicable through the financing agreement signed between a borrowing country and the Bank. The Bank’s role is to supervise the implementation of the projects, with the goal of ensuring that funds are used for the purposes intended. As part of this role, the Bank carries out a prior or ex-post review of the procurement actions undertaken by the borrower. Despite the applicable rules, however, the implementation and institutional arrangements under these projects, but also the culture and professional norms, are similar to those existing in the relevant implementing entity which are often characterized by failure to take timely action6 or avoid taking decisions, etc. These behaviors lead to long and unnecessary delays in project implementation and this, in turn, limits the borrowing government’s ability to achieve its objectives of effectiveness in service delivery. Unfortunately, such behavior is also influenced because many staff working on MDB projects are hired as consultants to the Government, i.e. their employment often depends on government staff willingness to continue hiring them and they do not belong to any professional body or the formal civil service.

    Most decision-makers recognize that a decision which is technically correct but “politically incorrect” is considered “wrong”. Kelman7 notes that “[t]his stems among others from an environment that heavily penalizes ‘wrong’ decision, offers little reward for timely decisions, and no penalties for avoiding decisions”.

    The experience of implementing MDB financed projects reveals that the guiding principles of economy, efficiency, transparency, equity and integrity often turn into an exercise of enforcing procurement rules rather than using good judgement for a better outcome.

    The examples from MDB financed projects show that in several government organizations there is a culture of “anticipatory compliance” (skill of knowing beforehand how the seniors in higher levels in government are likely to react to a proposal), not saying what needs to be said at the right time based on professional judgement and thereby putting the procurement process and project at risk.

    However, in such decision-making, Kahneman8 says, “the voice of reason may be much fainter than the loud and clear voice of an erroneous intuition, and questioning your intuitions is unpleasant when you face the stress of a big decision. More doubt is the last thing you want when you are in trouble. The upshot is that it is much easier to identify a minefield when you observe others wandering into it than when you are about to do so…”. In summary, in our procurement decision making we tend to follow rules as it takes the least effort and reduces the possibility of the decision maker getting into trouble, irrespective of the outcome.  Therefore, what is the incentive in using judgement and inviting trouble in the future?!

    So the issue is: How to strike a balance among competing procurement principles, work with flexibility and achieve “fit-for-purpose” and “value-for-money” irrespective of funding source (government or MDB) and how to empower project officials to exercise due discretion that leads to a successful project outcome?

  6. Geography on Fear of Discretion: As noted earlier, based on the authors’ working experience in different countries, small or large economy, this fear of discretion is a global “phobia”, with few exceptions.  The examples given below are taken from the various procurement assessments and reviews carried out by the World Bank/MDBs of several national procurement systems with an example from developed economy as well.

    • Example# 1. A country in Central Asia: - extracts from Integrated Country Fiduciary Review: Oversight of Public Financial Management
      “The system of internal control reviews (also called financial inspections, akin to internal auditing), which exists at all levels of government and covers financial, compliance (including procurement), and efficiency reviews, takes a punitive approach to correct errant practices and officials, rather than a supportive approach to assist the organization in delivering efficient and effective public services.”

      Example#2. A country in South Asia- extracts from Country/Sector Procurement Assessment Report
        “Procurement Complaint review mechanism is weak due to high fees for review of complaint and multiple jurisdiction for filing procurement complaint including to Commission on Investigation of Abuse of Authority. Fearing the hassles of interrogation of the CIAA, the project officers often postpone decisions on contract award, seriously affecting the progress of project implementation.”

      Example #3 A country in the European Union - extracts from a Fiduciary Review Mission:
      “Decision makers, PMU staff, evaluation committee members could become subject to personal liabilities for public loss by the decisions of the Country’s Court of Accounts which creates a culture of fear and avoidance. As a result, the Disbursement Ratio in this country for MDB portfolio is about 3% in a row over the last three years. Establishing motivated proactive PMUs is not easy. Reference was made to the personal liability for civil servants which may result in large financial sanctions even in cases when a civil servant was acting in good faith and in the public interest. In order to motivate PMU staff with relatively low salary levels, organized training in or outside of Country may be a good incentive. “

      Example#4. Experience of a developed economy in North America
      “There were procurement failures in purchase of complex equipment and computer system after fully complying with “principles of equity-to provide fair access to bidders in competing for government business; integrity- to reduce the chances of corruption in procurement process; and economy and efficiency - to procure at the lowest possible price for goods or services of quality desired”9

      By following all these principles and not working with flexibility, the government did not get what they needed.

      In all the above examples, the situation of fear of discretion or lack of any incentive for procurement practitioners to exercise due discretion and sound judgement appears to be created by the government rules which leads to an undesirable or absurd outcome and not in the best interest of the government.

  7. Procurement Rules of World Bank: Historical Background on Procurement and Consultancy Guidelines (collectively “The Guidelines”): The history of the World Bank’s procurement rules could be traced back to 1951, when Raymond A. Wheeler, the Bank’s newly appointed engineer advisor, introduced International Competitive Bidding (ICB) for Bank-financed projects and open competition as the best way to ensure its primary objective of “allow[ing] borrowing countries to buy high-quality goods and services as economically as possible.”10 A written set of procurement guidelines was compiled in the form of guidance to staff in 1961, and the first set of codified procurement rules (called the “Procurement Guidelines”) was approved by the Board of Executive Directors in  For four decades the procurement rules went through significant changes and revisions. Some of those main changes include:

    • International competition and advertising, pre-qualification, public bid opening, currency provisions, language, bid and performance bonds, general conditions of contract and specific contract conditions, all were dealt with under 1964Guidelines;

    • Protection for domestic suppliers was introduced in 1966, and currency for bid comparison more clearly regulated in 1971;

    • Three main principles guiding procurement i.e., economy and efficiency, equal opportunity and further participation of domestic industry were introduced, detailed conditions for preference for domestic suppliers that had appeared for the first time (1974);

    • Maintenance of currency value, introduction of other types of procurement methods/techniques like Limited Competitive Bidding, Shopping, Force Account and Direct Contracting (1977);

    • Introduction of Bank’s “three official languages”, two detailed annexes one on Bank’s prior review and one on domestic preference and procurement by UN agencies (1984);

    • Introduction of the use of community participation in procurement, procurement of Build Operate Transfer( BOT) contracts; expansion of content of bidding documents by detailing the provisions to be incorporated therein, mandatory use of the Bank’s standard bidding documents for ICB; introduction of eligibility for participation of State –owned enterprises (SoEs)--a policy adopted by the Bank in 1989-- mirrored conflict of interest provisions applying to consultancy services and introduced a new UN sanctions-related ineligibility; incorporation of fraud and corruption provisions (1995/1996);

    • ICB was no longer the default procurement method- other methods to be used “where ICB was not the most appropriate procurement method”, the definition of fraud and corruptionexpanded to include practices engaged in indirectly or through agents, coercive practices were added, collusion and fraudulent practices were split into distinct practices, audit rights extended to bids submitted by bidders, advertising of bidding opportunities, issuance of bidding documents and bid submission could be done by electronic means of communication ( 2004); and one notable change brought by the 2004 Guidelines was abolishing of eligibility restrictions relating to the country of origin, signaling the Bank’s support for “untied” procurement. This restriction was put in place in 1956 (even though initially was not there), restricting the eligibility of providers of goods, works and services from member countries and Switzerland. In 2004 such restriction was abolished.

    • Introducing new provisions pertaining to conflict of interest, expanding fraud and corruption provisions to encompass other potential participants in the procurement process, explicitly providing for cross-debarment with other MDBs, and temporary suspension as grounds for ineligibility of bidders, replacing the earlier National Competitive Bidding provisions with “Borrowers shall have an effective and independent protest mechanism in place allowing bidders to protest and have their protests handled in a timely manner”, use of country systems, expanding misprocurement to situations involving fraud and corruption and failure to inform the Bank, and made explicitly applicable in certain circumstances such as: dilatory conduct, inconsistency of contract provisions with agreed procedures (2011).

    Throughout the history of its Procurement and Consultant Guidelines, the Bank has taken numerous steps, which in the Bank’s standpoint, expand flexibility. For example, both Procurement and Consultant Guidelines of 2004 abolished country eligibility restrictions (after 53 years), introduced a diverse menu of procurement and selection methods, allowed flexibility in terms of “substantially responsive bids” or quantification of non-material deviations, use of weighted criteria, allowed use of electronic procurement and made changes to procedural requirements for greater transparency. The World Bank’s Procurement Guidelines continued to be principle-based with enough room for flexibility till 1980s and even in 1990s. There were countries who sought World Bank/MDBs financing to ward-off political interference in procurement process, exercise good judgement for a better outcome and be more flexible compared to their own government procurement rules.  In general, it was seen that with MDB’s prior review or oversight, the government departments were able to exercise more flexibility for a better outcome, without the fear or doubt in their decision-taking.

    However, the revisions over the years, aiming to improve the existing rules, also introduced inconsistencies and unintended limitations.  It appears that because of these minor tweaks the Guidelines have moved towards take caring of rule-based concerns to the extent that a 23 page (with 3 footnotes) Procurement Guidelines of March 1977 (Revised July 1980) and Consultant Guidelines of 1981 in 38 pages (10 footnotes) morphed into a 2011 Procurement Guidelines of 77 pages and 88 footnotes and 2011 Consultant Guidelines of 76 pages and 68 footnotes in similarly sized documents.  More guidance led to loss of flexibility. More footnotes dragged the feet of procurement practitioners in MDB-financed projects in moving forward and exercising professional judgment. It was felt that over the years, the professional judgement was slowly replaced by a strict reading of the Procurement Guidelines, both by borrowers and within the Bank. A study11 by the World Bank revealed that “the compliance with Bank rules is becoming an objective in itself with less regards to needs.”

    A paper published by Gutman12 examines this issue from a broad policy angle with a need to modernize International Financial Institutions financed policies for better project outcome.

    The World Bank’s new Procurement Framework, which became effective on July 1, 2016, has adopted fit-for-purpose and value-for-money as the overarching considerations, and it provides a number of tools for exercising flexibility in carrying out procurement in Bank-financed projects. Some of those features include: an expanded menu of procurement methods/approaches, an increased use of national procurement arrangements, reliance on procurement rules and arrangements of other MDBs or bilateral agencies in case of co-financing, permission to use negotiations, permission to use best and final offer (BAFO) approach, competitive dialogue, flexibility in procurement procedures for PPPs, etc.

    However, the new procurement framework is yet to be tested. As is widely recognized, exercise of flexibility does not rely only on the legal framework. It requires in-depth knowledge and experience of procurement by the procurement officials, technical staff, decision-makers, an enabling environment that encourages use of discretion for better outcome and where all the stakeholders, even if not in agreement due to competing interests or objectives, at least understand that decisions are taken for the best project outcome and most effective use of public money.

  8. Fear of Discretion at different stages of project design, preparation, procurement process, and contract implementation for large and complex projects. Below are some of the main aspects and examples under World Bank/MDB funded projects where failure to exercise professional judgment has led to undesirable outcome. They are based on the authors’ personal experiences in MDB-financed procurements.

    • 7.1 Project Design and Preparation Stage

      Aspect # 1 Impact of “Optimism Bias” in Project Design and Preparation
      Based on experience of project design and preparation for large and complex contracts it is seen that an “optimism bias” at this stage and fear of not expressing their views on technical impossibility in the face of “order” from above could lead to delays, significant claims or failure of contract performance.

      There is an optimism bias in our project design and we tend to just ignore for convenience hard data based on past track-record.  And we keep on repeating our mistakes rather than learning from it. Therefore, when stakes are high we need to take a more deliberate approach, develop our capacity to say “No” backed with data and not let the decision be taken based on “optimism bias” under non-economic considerations, only to regret later.

      A case in hand relates to a large road contract (MDB-FIDIC Based Bidding Document), shows consequences of lack of judgement and discretion, in the face of a strong request from senior levels of the government to fast track the procurement process by following post qualification and not prequalification.  In retrospect it was seen packaging of contract by dividing it in three lots (Roads, Bridge and Tunnel) was not appropriate due to the facts that these contracts were interdependent due to terrain, tunnel design was not ready at the time of bidding that led to rebidding for this part, contracts for Road (Lot 1) and Bridge (Lot 2) were awarded to a contractor (same contractor for both) at abnormally low bid price, land acquisition was not completed though confirmed that it would be available soon after contract award. Later, it turned out that the contractor for Road and Bridge was in financial difficulties and this contractor kept on submitting unjustified claims without showing any progress. This led to contract termination for Road and Bridge Contract and re-contracting.  A contract which was supposed to be fast tracked and completed in 24 months was finally completed in 48 months. This case illustrates the consequences of “optimism bias” and moving ahead without adequate preparation at the design stage.

      Aspect #2 Selection of Consultants- Consequences of Not Exercising Discretion or Judgement
      For projects financed by MDBs, in several situations, the Borrower lacks capacity to design, prepare and implement a complex infrastructure project or the project may be based on new technology and could be a one-off project, not to be replicated in near future by the Borrower.  In this scenario, use of international consulting firms with in-depth knowledge and experience is required for an effective design and implementation of the project. Most often this requires consultants to carry out feasibility studies, do preliminary and detail design and provide further project management and supervision services on behalf of the Client. For example, in a project of USD 100 million, the cost of consultancy for such services may be in the range of USD 5 to 10 million, depending on the nature and complexity of the project. As brought out earlier, with the advent of QCBS in 1997 Consultant Guidelines, Clients have increasingly relied on this method of selection rather than quality being the sole criteria to start with. In fact, to avoid facing complaints, borrowers push every consultant firm that submitted proposals to pass the minimum technical score needed for the consultant’s financial proposal to be opened. This approach creates the least risk for the borrower to be challenged by consultants, but all this takes place at the cost of quality, so much needed and predicated in the case of intellectual services.

      Based on the experience of projects financed by MDBs it is seen that the government implementing agency is not willing to seek such external expertise. In situations where the implementing agency agrees to hire a consultant, these government agencies lack skills to prepare a well-defined Terms of Reference and prepare a robust short list in a timely manner. In award of contract in practice cost plays a major role and projects get designed and implemented by those who may not be the best consultants. Normally, the entire process of selection of a consultant has focused on following rules at all stages of selection. If rules are set for a better outcome that is fine, but these rules allow the Client and MDBs to move in a mechanical manner, with the least effort in application of discretion and judgement, all of which leads to selected consultant hiring experts who just cross the threshold of expertise, but are not the best.  Several international firms just lend their name to get shortlisted and technical proposals may contain stellar CVs, which leads to selection of an experienced Joint Venture firm. After award replacement of key experts with apparently equal or better qualification (we all believe so!) begins to the extent that the entire team is replaced mid-way during the contract period. Several leading international firms do not show any effective presence in the execution of the contract and leave it in the hands of a less experienced Joint Venture partner or local sub-consultants. Poor contract supervision by the Client does not seem to focus on these aspects, so the Client follows the rules but does not get the high-quality services it sought to obtain in the first place through the selection process.

      7.2 Evaluation and Award Process- Consequences of Fear of Discretion and Lack of Judgement

      Aspect # 3 Award of contract to a Bidder with Lowest Evaluated Substantially Responsive Bid
      Often the World Bank Procurement Guidelines have been criticized for promoting award to bidders offering the lowest cost. That criticism, however, is not entirely grounded. The Guidelines refer to the award to the bidder who meets the standards of capability and resources, and whose bid has been determined to be (i) substantially responsive to the bidding documents, and offers the lowest evaluated cost. In evaluating the cost, the Guidelines permit taking into consideration other factors such as: payment schedules, efficiency and compatibility of equipment, safety and environmental benefits. This is similar to the concept of most advantageous offer adopted by most of the international procurement instruments and the recently adopted World Bank’s Procurement Regulations for Borrowers (July 2016).

      Interestingly, since 1977, all Procurement Guidelines provided that “the bid with the lowest evaluated cost, but not necessarily the lowest submitted price, be judged the most advantageous offer.” In relation to the use of criteria other than price, the 1977 Guidelines required that all non-price factors be quantified in monetary terms, to the extent practicable, or given a relative weight in the evaluation provisions of the bidding documents. It was in the 2011 Guidelines that reference to weighted criteria was removed making application of weighted criteria exceptional to the procurement of textbooks and IT systems.  For unknown reason(s), however, even though the rules permitted the use of merit point system or weighted criteria, its application did not happen in practice with few exceptions and only when vetted by the Bank.

      Furthermore, in case of evaluation of goods and works as per Procurement Guidelines, there was always a distinction made between evaluating the qualification of a “bidder” and determining the responsiveness of a “bid”.

      With regard to bidder in case of prequalification, it shall be based entirely upon the capability and resources of prospective eligible bidders to perform the particular contract taking into account objective and measurable factors, including: (i) relevant general and specific experience, and satisfactory past performance and successful completion of contracts over a given period; (b) financial position; and where relevant , (c) capability of construction and/or manufacturing facilities (Paragraph 2.9 of Procurement Guidelines). A similar determination is to be made in a situation of post-qualification when recommending award of contract to a “bidder” who has submitted a lowest evaluated responsive “bid”.

      World Bank’s Procurement Guidelines and those of other MDBs (and respective SBDs) stipulate that: “if a bid, including with regard to the bid security, is not substantially responsive, that is it contains material deviations from or reservations to the terms, conditions, and specification in the bidding documents, it shall not be considered further. The bidder shall neither be permitted nor invited by the Borrower to correct or withdraw material deviations or reservations once bids have been opened.” (Paragraph 2.48 of Procurement Guidelines 2011). And further the purpose of bid evaluation is to determine the lowest evaluated cost (but not necessarily the lowest submitted price) by adjusting for arithmetical errors, quantifiable non-material deviations or reservations. Such system of evaluation in addition to payment schedule, delivery time and other factors like operating cost, efficiency, and compatibility of equipment, availability of service and spare parts, related training safety and environmental benefits to be taken into account as a factor in evaluation (Paragraph 2.52 of Procurement Guidelines 2011).

      The definition of substantial responsiveness and material deviation in the bidding document has remained the same in MDBs over several decades. “A substantially responsive bid is one that meets the requirements of the Bidding Documents without material deviation, reservation, or omission.  A material deviation, reservation, or omission is one that:  (a) if accepted, would: (i) affect in any substantial way the scope, quality, or performance of the Works specified in the Contract; or (ii) limit in any substantial way, inconsistent with the Bidding Documents, the Employer’s rights or the Bidder’s obligations under the proposed Contract; or (b) if rectified, would unfairly affect the competitive position of other Bidders presenting substantially responsive bids.” Hence a substantially responsive bid is one devoid of material deviations.

      In the process of examination, any clarification is allowed for missing historical information and non-material non-conformities. Otherwise, responsiveness has to be determined on the content of the bid as submitted without recourse to clarification. A non-responsive bid cannot be made responsive through a process of clarification. In summary it should be possible to “live with” whatever is provided as part of technical proposal and commercial terms and conditions as a test of responsiveness and minor details finalized at contract award stage.

      Finally, contract is to be awarded to the “bidder” who meets the appropriate standards of capability and resources and whose “bid” has been determined (i) substantially responsive to the bidding documents and (ii) offers the lowest evaluated cost (Para 2.59 Procurement Guidelines 2011).

      However, the application of the above provisions has not been as smooth as one would think. In numerous cases, both high and low value, the World Bank is presented with recommendation to award contract to the higher priced bids as the borrower rejects bids with lower prices for minor deviations which can be quantified in monetary terms, or due to missing documentation of historical nature that can be submitted after bid opening. There is a general reluctance to seek clarifications from bidders, particularly from fear of being accused of unfairly treating other bidders who have submitted all the required information. In some cases, the “overzealous inspection” of bids is used to reject bids from bidders that are not “preferred” by the evaluation committee.

      On comparison of the above provisions of 2011 Procurement Guidelines with similar provisions of March 1977 (revised July 1980) Procurement Guidelines on determination of qualification of “bidder” and substantial responsiveness of “bid”, it is seen that there is hardly any change over the 35 years in terms of principles of evaluation. All the above principles allow for a combination of measurable and quantifiable factors with backed with use of discretion both for determination of qualification of “bidder” and substantial responsiveness of “bid”.

      Procurement practitioners could sometimes be put to difficult test in determining the responsiveness of bids when: “Bidders often attempt to optimize their chances of securing an award without adhering to the conditions of contract that they are unwilling to accept. To that end they will formulate reservations as mere suggestions, or qualify the compliance of their bid to a varying degrees, sometimes to the point that the contents of the bid become self-contradictory, and bidder’s intent impossible to ascertain. Other bidders prefer to take a greater risk that their bid may be rejected, by submitting explicitly binding deviations.”13  Such ambiguities could also be due to inconsistencies in rates and prices, which after arithmetical correction as required by bidding document could change the inter-bid position compared to the read out bid prices.  These situations require careful examination and need for exercising judgement and discretion, be flexible within the overall bounds of evaluation criteria with a focus on better outcome for the project.

      Another challenge that procurement practitioners face in exercising judgement is the influence from senior governmental officials (both on the side of government organization and MDBs) or other unethical behavior. The evaluation process gets delayed and dirtied due to improper interference in certain country situations. In such situations, the issue is not lack of discretion or judgement, but prevention of such improper interference, which also results into back-and-forth between the Borrower and MDB reviewers. Similarly, in recent years, the fear of control bodies in government organization, sometimes undue interference by higher authorities in the government and the load of precedence in the face of complaints from losing bidders have led to an application of rule-based system rather than application of sound judgement. The consequences would include inter alia further delays, not the most economical procurement, or declaration of misprocurement on the side of the Bank.

      Aspect #4:   Award of Large Civil Works Contract to Abnormally Low Bids without due examination
      The subject of Abnormally Low Tenders (ALT) is a matter of concern for delivery of projects within time, cost and with quality across all the client countries of World Bank/MDBs. This issue was one of the important topics for discussions for over more than a decade in international institutions like UNCITRAL, the European Commission (EC), and the MDBs through several expert groups and seminars. Some of these institutions have developed policies and procedures to deal with ALT but not much evidence is available for its application possibly due to fear of exercising discretion by procuring entities particularly in developing countries.

      There are several reported cases in public domain on award of contract for large civil works contracts to ALT without due examination, where fear of rejecting an ALT had some role, which led to undesirable outcome including termination of contract.

      Based on country and sector assessment carried out for a country with small economy in South Asia, and on data collected for 27 contracts of civil works procured through National Competitive Bidding (NCB),  it was seen that contract prices were below 40 to 60% of estimates. It appears contracts were awarded even though it was known that the contractors may not be able to execute the work. Government officials would not like to take personal risk of ignoring or rejecting a low priced-bid. Low bid prices contributed to delays, non-performance and use of unauthorized method of construction (machinery instead of labor) with possible impact on quality of work.  The situation was handled by termination of few contracts due to non-performance and close monitoring of these low value contracts. Not to mention, that a contract awarded to an ALT bid would presumably need increased efforts (in terms of time and resources) from the Government to supervise the implementation of contract. Absence of such increased supervision would lead to a poor performance or failure of the contract altogether.

      It is necessary that the method of seeking clarification and details to be provided by the bidder is made a part of the bidding document and bidder is required to provide clarification in situation of ALT. For example, for Unit Rate Contracts, the price justification form could be asked for a construction statement, work schedule resource input and breakdown of unit rate as direct labor cost, materials, equipment, site overheads, headquarter overhead and other financial cost like insurance and profit.

      The current Standard Bidding Document for Large Works (MDB harmonized version) following ITB 16.1 stipulates that: “The Bidder shall furnish a Technical Proposal including a statement of work methods, equipment, personnel, schedule, and any other information as stipulated in Section 4 (Bidding Forms), in sufficient detail to demonstrate the adequacy of the Bidders’ proposal to meet the work requirements and the completion time”. Similar clauses exist in all SBDs.

      The due diligence of ALT should be included under these provisions including the required forms that at the stage of Price Proposal in case bidder’s technical proposal is inadequate and bidder is not able to provide justification to satisfactorily perform the contract based on ALT, the bid should be treated as non-responsive and rejected. Stronger emphasis should be placed on the enforceability of the provisions on ALT. Procurement practitioners should be expected to take a more substantive review of the bidder’s technical proposal, so as to better evaluate whether a bid is, in fact, abnormally low.

      This is one area where application of ALT provisions in UNCITRAL, EC and World Bank Procurement Regulations have converged as brought out at (i) UNCITRAL Model Law on Public Procurement (July 2011) Article 20; (ii) EC Directive 2014/24/EU on Public Procurement Article 69, and (iii) World Bank’s Procurement Regulations for Borrowers (July 2016) paragraphs 5.65-5.67 (including the Guidance Note).

      It is hoped that the above manner on treatment of ALT could pave a way for following similar provision in Public Procurements Laws and Rules and provide a tool for procurement practitioners for a documented discretion.

      7.3 Contract Implementation Stage

      Aspect #5 Ineffective Contract administration
      MDB/World Bank Procurement Guidelines do stipulate that conditions of contract shall include provisions dealing with the applicable law and the forum for settlement of disputes. In case of large and complex contracts, the dispute settlement provisions include mechanisms such as Dispute Review Boards (DRB) or adjudicators, which are designed to permit a speedier dispute settlement.

      The first aspect is delay in handling contractor’s claims. This issue is the result of fear to take action that requires judgement and discretion.  Such lack of action on the part of the Employer combined with absence of a Dispute Review Board at the beginning of the contract could result into contract failure. It is seen that for large and complex contracts Employers lack capacity to respond to contractor’s claim in a timely manner. Contractor’s claim could be frivolous or justified. In either scenario it needs to be dealt with in a timely and effective manner. Similarly, when there are delays for the fault of contractor, levy of delay damage is ignored or contract execution continues without settling claims or application of delay (or liquidated) damages. Similarly, despite reported performance issues, in few cases the borrower calls off the performance security. More often such claims do not receive any attention as handling these requires effort and decision-making. All these unresolved issues and claims at some point leads to contract implementation coming to a stop.

      Furthermore, it is seen that in majority of large and complex contracts, DRB does not get constituted at the beginning of the contract as it is not mandated and enforced and Employers consider it as an unnecessary expenditure without any benefit.  In case of a dispute arises, however, DRB gets constituted. That is too late and of course is not the purpose of DRB, which had to be put in place early in the contract so to be able to get acquainted with the contract and prevent claims from becoming disputes.  DRB payment rules require sharing of costs of DRB equally as provided in conditions of contract.  It is worth considering, if DRB constitution is mandated, monitored by an independent institution and cost of institutional monitoring and fees paid to DRB members by the Employer be made an eligible expenditure for Bank financing.

      Aspect #6 Fear of Closing Contract
      For large and complex contracts, it is quite common that signing of contract is a matter of celebration for both the contracting parties often publicized in media. As the contract progresses such enthusiasm wanes. Due to unresolved claims and disputes the contracting parties rarely celebrate a contract that is completed within time and cost with quality and need for claims and disputes. Contracts are generally let to passively expire rather than actively closed due to fear of releasing the contractor from all its obligations.

      Based on the provisions of contract and standard procedures, a completion certificate is to be given by the employer to the contractor on completion of the work. This also requires strict enforcement of contract conditions. It is essential that the contractor fulfils all its/his obligations in terms of the contract, all due payments of the contractor is released, all records are available for future reference, all claims and disputes are resolved and contract is closed in a methodical manner and no claim certificate obtained from the contractor. In practice it is seen that usually it is the contractor running after the borrower to close the contract rather than the opposite. A failure from borrower to take over the works or plant leads to situations when the road or plant or goods are damaged or malfunction during a period when the defects liability or warranty periods have not started yet leading to further disputes between the parties which could continue for years after contract completion.  A well- documented check-list on contract closing certificates and obtaining a no-claim certificate on the lines that the contractor certifies that with the final payment, they shall have no claim, whatsoever of any description, on any account, against the government, against the contract agreement executed by them and they declare unequivocally, that with this payment, they have received all the amounts payable to us, and have no dispute of any description whatsoever.

  9. Working with Flexibility:  It is always a challenge for a procurement practitioner to work with flexibility within the bounds of rules, be seen as impartial and fair in the eyes of losing bidders and survive the onslaught of control bodies in government who focus on a particular transaction and if rules were followed rather than project outcome. It is seen that when faced with an emergency situation or unforeseen circumstances, both the government organization and MDB tend to be more flexible and ready to work for a better project outcome. These situations are moments of learning and an opportunity to experiment and use these learning to bring better flexibility in “normal” circumstances. A few examples are given below in brief where working with flexibility and use of documented discretion led to good project outcome. The following examples are taken from the authors’ experience and relate to contracts/projects implemented over the last three decades in Eastern Europe, Central Asia and South Asia. These examples illustrate the use of flexibility in situations when following rules and procedures is not the best interest of the government organization.

    Example 1:  Procurement of Generator Transformer for a 500 MW Coal-based Power Plant: Converting a Fixed Price into a Price Adjustment Contract (ICB- MDB –financed):
    This relates to supply and installation of Generator Transformer which is a critical link in power evacuation of any conventional coal based power plant. Contract was awarded by the Employer in India in late 1980s to a supplier in Yugoslavia for a project financed by MDB. As the contract completion time was less than 18 months, according to the procurement rules that would be a fixed price contract.  While the supply of the contract was mid-way, the situation in the country deteriorated and with only partial supply of the Generator Transformer equipment, the commissioning of a 500 MW power station was at stake. The contract for the Generator Transformer was USD 15 Million as compared to the cost of overall power station of about USD 500 Million, not to mention that the delay in commissioning would have led to further losses in terms of cost escalation for the power station and loss of revenue for power generated.  All other equipment for the project were supplied. A Generator Transformer is a long delivery item and therefore there was an impossibility of getting an alternate supplier. The supplier from Yugoslavia expressed its difficulties in completing the contract as inflation in the country was very high and it was not possible to get required input at the given contract price. The supplier asked for a 50% price increase. For the power company which was a government entity the option of strictly enforcing the contract (i.e. not revising its price) was not in their best interest. Therefore, in consultation with MDB, based on the country and market situation the contract was renegotiated and with exercise of due discretion and judgement a price increase of 30% agreed. The contract was completed just before the disintegration of Yugoslavia and the power station was commissioned on time.

    Lesson learnt: In the subsequent revisions of the MDB’s Procurement Guidelines, it is permitted to use price adjustment when future local or foreign inflation is expected to be high in contracts of duration less than 18 months, including for those contracts where for major inputs of the product (like copper in transformer or aluminum in electrical conductor for transmission lines) are subject to price volatility.

    Example 2:   Selection of Consulting Firms (three separate assignments) as Procurement Agent, Audit and Financial Management: Use of Modified Selection Based on Consultant’s Qualification (CQS) in a Post-Conflict Country
    This relates to selection of three consulting firms for a post conflict country in 2002 where MDB got reengaged with the country after two decades. At the time of this reengagement, the government itself was in the process of getting reconstituted and there were no counterpart officials who could initiate the process of selection of three consulting assignments each valued at around USD 3 million. The selection of consultants was to be completed in 2 months and the usual Quality-and Cost-Based Selection if adopted would have taken 5-6 months. In this scenario the Borrower approached MDB to help them with the process of selection following a flexible approach. MDB Guidelines allow for a flexible method, which is Selection Based on Consultants’ Qualification – CQS, but this method was generally to be used for assignments not exceeding US$100,000. In this selection Borrower is to prepare a TOR, request expressions of interest on consultants’ experience and competence relevant to the assignment, establish a short list and select firm with the most appropriate qualifications and references.  The selected firm is then asked to submit a combined technical-financial proposal and then invited to negotiate the contract.  MDB took the responsibility of opening a dedicated website to seek Expression of Interest based on Modified CQS where a brief Terms of Reference with evaluation criteria (for experience of the firm, expected key personnel and response to the TOR) was indicated in the published EOI. MDB received all the applications including the physical submissions which were sent to the consultant of the Borrower. In the period when EOI was under evaluation, the consultant of the Borrower also prepared three Request for Proposal Documents for the three assignments. After evaluation of EOI, respective RFPs were issued to the firm(s) selected for each assignment and these firms were requested to submit a combined-technical proposal for examination and negotiations, if found acceptable. The entire process of selection of these three consultancies at contract price in the range of USD 2.5 to USD 3 million each were completed not in 2 months as targeted, but three months.

    Lesson learnt: The above experience led MDB to replicate this expedited procedure in other post-conflict countries and in emergency situation, as also to increase the limit for CQS to US$ 300,000 in 2011 Consultancy Guidelines.

    Example 3:  Renegotiation of a Time Based Contract for Construction Supervision (MDB- Financed)
    A time-based construction supervision contract for USD 10 million was signed with a firm following method of selection as QCBS for Water Supply and Wastewater Treatment Project. At the time of project appraisal, it was envisaged that the Owner Water Company who was the Project Implementer would carry out all feasibility study, detailed design and preparation of prequalification and bidding document so that bids for plant and construction contracts are invited and finalized. The owner company had agreed to do advance contracting as part of project readiness. The signing of contract for construction supervision consultant was made by MDB as an effectiveness condition for disbursement of project loan. In 2010, it was found by MDB team that the Construction Supervision Contract was already signed and due to an ineffective Project Implementing Unit of the Owner Water Company, the feasibility reports were not prepared and no action was taken on detailed engineering and preparation of the bidding document. The total investment for the project was around USD 250 million.  Based on the request from the government, the situation was salvaged by first changing the PIU to a capable counterpart, with the help of this new PIU the Construction Supervision Consultancy contract was renegotiated to include in its scope a Lump Sum part of approx. USD 5 million for preparation of feasibility report (including topographic surveys, geologic and hydrologic investigations, land acquisition design for each sub-project and preparation of PQ and bidding document).  The entire negotiation process was well documented and justified both by the new PIU and MDB, and led to salvaging a USD 250 million investment by bringing the project on track.

    Lesson learnt: Agility in adapting the implementation arrangements to the needs helped both the borrower and the Bank to bring the project implementation on track.

    Example 4: Non-Responsiveness of Bids due to a shorter Bid Security Validity Period
    An MDB Procurement Guidelines and Bidding conditions stipulate that a bid should be accompanied by a substantially responsive bid security. In general, the validity of bid security should be 28 days beyond bid validity period. If bid validity period is 90 days the bid security should be valid till 118 days after bid submission deadline. In practice, it often happens that the bidders ask for extension in deadline for bid submission and the Employer extend this deadline say for 10 days and inform such extension to prospective bidders say 5 days prior to deadline for bid submission. The Bid Security Form has got a provision of twenty –eight days of validity after the end of Bid Validity period. However, bankers while issuing the bid security stipulate a specific date of expiry of bid security. This results into submission of the bid where bid is valid for 90 days, but bid security is valid for say 108 days instead of 118 days as the bidder did not ask the banker to amend the date of expiry before the bids were submitted along with the bid security. This situation calls for an exercise of judgement or discretion. However, it is for the Employer to exercise the discretion in a consistent manner that leads to a better outcome. For example if in a contract with estimated value as USD100 million the lowest evaluated responsive bid is say USD 102 million (with the only flaw that bid security that it is short by 10 days) and the next bidder’s evaluated price is say USD 110 million ( with no such flaw), in the spirit of a substantially responsive bid and bid security, it would be an extreme application of rule ( and lack of judgement and discretion) if the Borrower decides to ignore the lowest evaluated bid and recommend award of contract to bidder at the higher price of USD 110 million. But MDBs do receive such recommendations from the Borrower very frequently, and it becomes difficult when Borrowers insist upon extreme application of rules which are not beneficial to the government itself.

    Lesson learnt: No matter how much flexibility and clarity the applicable procurement rules provide, there will always be situations when rules do not provide for a solution. It is therefore, upon the professional judgment of the procurement and project officials to assess whether the benefits outweigh the risks associated with a certain decision.


  10. Final thoughts and recommendations: It is hoped that the World Bank’s new Procurement Framework will increase the willingness to exercise professional judgement at all levels of procurement practitioners, both in the government and in the World Bank itself.  However, as noted earlier, flexible rules do not guarantee application of good judgement with transparency and integrity. Procurement practitioners working in the World Bank projects would need to enhance their skills and knowledge to gain proficiency and competence in the new rules. For government officials implementing those projects, the rules and the knowledge may not be enough. An enabling environment that does not penalize “healthy use of discretion” is equally important.

    There are no silver bullets or pills/potions to reduce fear of discretion. As procurement practitioners we should be ready to face complex procurement issues and recognize the difficulty of exercising discretion due to diverse country situations and different enabling environment. The following steps could help improve the situation on reducing the fear of discretion for a better outcome.

    Action by MDBs

    • MDBs need to identify the source of fear and deal with it upfront: As part of project preparation it is essential to identify the source of fear and mechanism to deal with it through a dialogue with such bodies in the government whether it is a control body like Court of Accounts or Commission for Abuse of Authority. It is seen that control bodies are not able to distinguish between MDB procurement procedure and that of the government, which leads to audit queries and situation of fear for procurement practitioners.  A dialogue among MDBs, government departments and control bodies is required at part of project preparation and procurement strategy so that systems and procedures are clarified and understood on the same basis by all parties.  MDBs could play a vital role in dealing with source of fear by engaging with such control bodies rather than blaming them unfairly. This is especially important since under the World Bank’s new procurement framework, national audit institutions may be assigned an enhanced role in carrying out post review of the procurement activities, especially for those activities under national procurement arrangements.

    • Guidance Notes, Improvements in Standard Bidding Documents and Borrower Training for exercising discretion: A principle based procurement would require a series of guidance notes and constant improvement in bidding documents to facilitate documented discretion. Guidance note on ALT is a good example.  Often the procurement practitioners working in MDB -funded projects lack a good understanding of procurement principles of MDBs, but they are put in a situation to decide procurement cases. Their only training is to follow rules. These officials (including those from control bodies) need to be sensitized through case examples on how to use discretion and flexibility in procurement for a better project outcome.


    Action by the borrowing governments

    • Building a Cadre of Procurement Professionals and Develop a Culture on Use of Discretion: In order to motivate procurement staff organized training, stability of job/tasks that provides a continuity in learning skills and continuous application may be a good incentive. What is important is handling challenging task and being recognized for the contribution with a career ahead. It is also important that procurement practitioners are remunerated well. There is a need to develop an organization culture that encourages use of discretion for better outcome.

    • Abolish Provisions of “Personal Liability” in Public Procurement:  Many borrowing countries have adopted through their public procurement laws or other laws, extensive provisions on financial penalties (in some countries even criminal liability) and other punishment forms if such law(s) is violated. Procurement practitioners are performing their duties on behalf of the government, making their best professional judgement and taking responsibility for their decision, but only in their official capacity. They are of course subject to audit requirements and performance review. If these officials have acted in good faith and have not indulged in any fraud and corruption or other unethical behavior, why should they face “personal liability” for decisions taken?  In many developing countries this fear of facing personal liability in the course of carrying public duty has led to enormous risk aversion and hesitation to take on a procurement job or even do anything that is not explicitly provided for in the procurement law. May be there is difference in approach between common and civil law countries, but this issue may require further examination by procurement legal professionals.


    Action by the Practitioners of Procurement

    • Develop your capacity to say what needs to be said based on your conviction:  Therefore when stakes are high we need to take a more deliberate approach, develop our capacity to say “No” backed with data and not let decision taken based on “optimism bias” under non-economic considerations.   We need to act based on our conviction and good judgement and not convenience, only to regret later.

    • Do not blame or hide behind the system-do your best in the given circumstances: Based on observations and experience of working in government organizations, MDBs, it is seen that all system and work environment provide for enough flexibility and empowerment to act in most of the situation, provided you as a decision maker have clarity of thought to act in the best interest of the organization and not any personal gain. Fear emanates from doubt on your own lack of ability or expertise or when acting under the direction of higher-ups against your own good judgement which leads to situation of “guilty mind” and inconsistent actions.

    • And finally, build expertise and reputation as a problem solver: Expertise is built through acquisition of knowledge and skills and its constant application in our chosen area.  Let our curiosity not die with more experience. We need to open channels of communication, professional partnership and community of practices among procurement practitioners so that when in doubt we could seek an impartial feedback on our thoughts and actions. We need to be ready to solve complex procurement situations with proper analysis and a well- documented exercise of discretion.  All these actions in the interest of the government builds upon the reputation of the procurement practitioner and his/her ability to exercise due discretion without fear.


List of References:

  1. The World Bank: Guidelines on Procurement of Goods, Works, and Non-Consulting Services (January 2011 -Revised July 2014)
  2. The World Bank: Guidelines on Selection and Employment of Consultants (January 2011- Revised July 2014)
  3. Revision of the World Bank’s Procurement Guidelines (Board Papers December 7, 2010)
  4. The World Bank: Guidelines for Procurement (March 1977- July 1980 Edition)
  5. The World Bank: Guidelines for Use of Consultants (August 1981)
  6. The World Bank: Procurement Regulations for IPF Borrowers (July 2016)
  7. The World Bank: Procurement Documents at worldbank.org/procure
  8. Website of EU – Public Procurement
  9. Website of UNCITRAL
  10. EFCA Roundtable 18/11/2008, Brussels
  11. Public Procurement Contract for Supply of Goods- Compliance with contract terms at the time of bidding- Francoise Bentchikou, IBA Conference Papers, Prague (September 2005)
  12. Brookings: Is there Room for Discretion? Reforming Public Procurement in a Compliance-oriented World”:  Gutman (Working Paper 74, May 2014)
  13. Brookings: Aid Procurement and the Development of Local Industry- A question for Africa., Zhang and Gutman (Working Paper 88, June 2015)
  14. Procurement and Public Management- The Fear of Discretion and the Quality of Government Performance- Steven Kelman (1990)
  15. Thinking Fast and Slow – Daniel Kahneman (2011)
  16. Various World Bank internal documents, case studies, project documents, procurement assessments.



  1. About authors: Devesh C. Mishra is a retired Regional Procurement Manager (Europe and Central Asia Region) of the World Bank and currently engaged as a procurement advisor/consultant for MDBs (ADB/WB) located in Washington DC area. Belita Manka is currently working as a Senior Counsel (Procurement) in the Legal- Operations Unit within Legal Department of the World Bank, Washington DC, USA. The views expressed in this paper are those of the authors and do not represent the official position or views of the organizations they work for.
  2. “Procurement and Public Management- The Fear of Discretion and the Quality of Government Performance” - Steven Kelman (1990)
  3. United Nations, “UN Procurement from Developing Countries and Economies in Transition,” in 2009 Statistical Report on Procurement Supplement (2010). http://www.ungm.org/Areas/Public/Downloads/ASR_2009_supplement.pdf.
  4. World Bank’s internal data. http://pubdocs.worldbank.org/en/433521492462144143/Procurement-App-Guidance.pdf.
  5. “Procurement and Public Management- The Fear of Discretion and the Quality of Government Performance” - Steven Kelman (1990); Brookings: Is there Room for Discretion? Reforming Public Procurement in a Compliance-oriented World”: Gutman (Working Paper 74, May 2014); Brookings: Aid Procurement and the Development of Local Industry- A question for Africa., Zhang and Gutman (Working Paper 88, June 2015)
  6. For example, the contract award recommendation for Supply and Installation of an External Assistance Management Information System (EAMIS) in country X was cleared by the World Bank on February 11, 2016 and until October 25, 2016 the contract was not signed. No reasonable justifications were given by the Borrower for this delay.
  7. “Procurement and Public Management- The Fear of Discretion and the Quality of Government Performance”, Steven Kelman (1990)
  8. “Thinking Fast and Slow” Daniel Kahneman, 2011
  9. “Procurement and Public Management- The Fear of Discretion and the Quality of Government Performance” Steven Kelman, 1990
  10. Aid Procurement and Development of Local Industry- A Question for Africa (Christine Zhang & Jeffrey Gutman, Global Economy and Development at Brookings ( Working paper 88, June 2015)
  11. EFCA Roundtable 18/11/2008, Brussels
  12. Is there Room for Discretion? Reforming Public Procurement in a Compliance-oriented World”, Gutman (Working Paper 74, May 2014)
  13. Francoise Bentchikou, former Chief Counsel of the Legal Procurement Unit in the World Bank, International Bar Association Paper, Prague, September 2005