Why SNC-Lavalin's poor technical score for Stage 2 LRT hardly mattered

This week's unprecedented document dump of previously confidential information on the controversial Trillium Line Stage 2 procurement has raised umpteen questions about how the contract was awarded.

But the release of new information on the financial scoring makes one thing crystal clear: when it comes to awarding a big infrastructure project, price reigns supreme.

All the concerns the city's technical evaluators had about SNC-Lavalin's detailed submission for how they planned to build Stage 2 — the absence of a signalling and train control system, no snow-removal plan or strategy for using existing trains, wacky designs for stations with no elevators or stairs — hardly made any difference in the final analysis.

Because of the way many procurements for public-private partnerships (P3s) are designed, including the one for LRT Stage 2, the scoring is weighted so heavily toward price that technical score almost doesn't matter.

It's not a new observation.

The cliché that you get what you pay for takes on a dangerous twist when it comes to the evaluation of price in tenders.​​​- Eddy Jin, procurement consultant

In 2014, Ontario's auditor general said the system of scoring for most P3s "places more weight on a low bid than on technical merits," giving the lowest bidder "a decided edge."

Eddy Jin, an international procurement consultant and former director of procurement for University of Toronto, is even more blunt.

"The cliché that you get what you pay for takes on a dangerous twist when it comes to the evaluation of price in tenders," wrote Jin in an article for a procurement trade journal

It's a scoring scheme, he writes, that "is flawed numerically and is unfair to bidders."

2nd-place bidder had highest technical score

For a year now, CBC has been reporting on the technical scores for the Stage 2 procurement and the fact SNC-Lavalin won the contract without meeting the minimum technical grade of 70 per cent.

But if you looked only at the final scores, you'd never know the giant engineering company scored so poorly in the technical round.

SNC-Lavalin scored 822 of a possible 1,000 points, whereas as the second-place finisher T-Link — a consortium of infrastructure specialist Acciona, investment firm Fengate, and a number of other firms, including local constructors Thomas Cavanagh — scored just 641. 

But in reality, the bids were much closer than those final scores might indicate.

Consider this: SNC-Lavalin's bid scored 67 per cent in technical, and came with a price tag $1.6 billion; T-Link clocked in with the highest technical score of 86 per cent and a price of $1.7 billion.

How the scoring worked

On the face of it, the scoring seems to give equivalent importance to the technical and financial submissions, allotting a maximum of 500 points to each category. The two are added together, and whichever bid scores the highest out of a total possible 1,000 points is the preferred choice.

But when you drill down deeper, that description is deceptive.

On the technical side, the scoring is fairly straightforward, with a team of technical experts reviewing and scoring the bid according to a predetermined score card. (For example, the design part of the submission was worth a total of 225 points.) The evaluators add up all the subtotals, and that sum is essentially the technical score.

SNC-Lavalin scored 336.6 out of 500, or 67 per cent. while T-Link scored 428.9, or 85.7 per cent.

But the financial scoring works much differently.

Financial scoring 'relative'

Of the 500 points available for the financial bid, 50 are set aside to assess the quality of the financing, which generally looks at whether long-term funding is committed. Bidders need to score a minimum of 35 points, or 70 per cent, in this category; SNC-Lavalin scored the minimum 35, while T-Link scored 42.6 points.

The remaining 450 points are allotted to price, and the bidder that comes in the lowest — in this case SNC-Lavalin — received the maximum number. So its final score, combining the 450 for price and the 35 for quality of financing, was a whopping 485.

Here's where it gets a bit tricky, however. The other bidders are scored based on how their prices compare to the lowest bid, something Jin describes as a "relative scoring approach" that's used in most P3 procurements.

The difference in the bid prices between SNC-Lavalin and T-Link was six per cent. The difference in their scores was 62 per cent.

In the Stage 2 process, the other bidders had 30 points deducted for every percentage point their prices exceeded that of SNC-Lavalin. That meant T-Link and its roughly $1.7-billion bid, $100 million more than SNC-Lavalin, scored just 169.82 points.

Combined with its bid quality score, T-Link ended up with just 212.32 points out of 500.

If you looked at the total scores, you might think the prices were much farther apart than they are in reality. In fact, the difference in the bid prices between SNC-Lavalin and T-Link was six per cent, but the difference in their scores was 62 per cent.

(Although T-Link's proposal would have cost $100 million more in the long run, it did call for about $200 million more in spending in the short term, money the city would have had to cover on its own.)

Question of value for money

City leaders, including Mayor Jim Watson and city manager Steve Kanellakos, have characterized SNC-Lavalin's bid as good value for taxpayers.

But value and cheapest are not necessarily synonymous.

"We should be looking to deliver projects that provide value for money," said Matti Siemiatycki, the interim director of the University of Toronto's School of Cities.

"It doesn't necessarily mean the lowest price at all costs, because price can be reduced, but that can also be at the expense of reducing quality … the key is to find a technical score [so] that the technical specifications are met and exceeded, and that technical score is being delivered at an affordable price."

Defence is 'high standard' of technical evaluation

In 2015, Siemiatycki co-authored a paper that looked at evaluation scores for 50 projects provided by Infrastructure Ontario.

The paper found that "although technical and financial scores are weighted evenly, their relative impact on the total score is not even."

The report explained that proponents of the scoring defend the system because, as Infrastructure Ontario told the auditor general back in 2014, "technical scores are subject to a minimum threshold, and proposals not meeting that threshold are removed from consideration." 

"By writing good specifications, by ensuring our technical evaluators have a good understanding of those specifications and by ensuring that teams have met a minimum threshold in each of the relevant technical categories, we are ensuring that the teams that pass through are meeting a very high standard of technical," according to the report, quoting an  Infrastructure Ontario participant.

Of course, we know that SNC-Lavalin did not meet the minimum threshold — although the folks overseeing the procurement have argued that because the technical evaluations are somewhat subjective, 67 per cent was close enough.