Impact of modelling assumptions on the IDM results

• Author: Giulio Caperna - Eleni Papadimitriou
• Pages: 13-17

13

4 Impact of modelling assumptions on the IDM results

A fundamental step of the statistical analysis of a composite indicator is to assess the effect of different

modelling assumptions among reasonable alternatives. Despite the efforts given in the building process, there

is an unavoidable subject ivity (or uncertainty ) in the resulting choices. This subje ctivity can be explor ed

comparing the results of different – alte rnative – choi ces.

The literature on the topic, and the usual JRC ap proach, suggest s assessing the r obustness basing on Monte

Carlo simulat ions and multi-modelling approach, assuming ‘error free’ data as eventual errors have been

corrected in the prelim inary stages of the construction of the composite indicator (M Saisana, T arantola, &

Saltelli, 2 005; Michael a Saisana, D’Ho mbres, & Sal telli, 2011) .

Nevertheless, the IDM presents a completely different setting with respect to the traditional indicators

considered in t he literature. The main two reasons of its uniqueness are: the micro level variables observed

directly on subjects and the presence of a vast majority of non-quantitative indicators in t he framework. These

two reasons motivated the JRC team to avoid the step of the Monte Carlo simulations, mainly be cause the

perturbation of weights on thousands of individual observation would not have the same meaning and

interpretation of the same procedure applied on stable entities (e.g. countries, regions).

The modelling issues considered in the assessment of the IDM were the aggregation formula the

inclusion/exclusion of the Time-use dime nsion (dim 14) and the dimensio n weights.

Aggregation formula. Regarding the aggr egation formula , the IDM team opted for th e arithmetic average of

the fift een dimensions whic h implies a strong compensability that allows outstanding performance in

some aspect s to balance for weaknesses in others and vice-versa. This approach means that subject s with high

and low scores at the indicator, theme or dimension level are considered similarly deprived as those with

average scores.

To assess the impact of this choice, a compari son with the geometric mean is included in the analysis. The

comparison of the two aggregation approaches should be able to highlight the individuals with

unbalanced profiles because the geometric mean tends to penalize the existence of a low value, even when the

other values are not so low.

Weights. The weighting system of the IDM is particularly strong, with the 5 most important dimensions

(dimensions 1-5) receiving a t otal weighting of 50% (10% e ach), the 5 next-most import ant dimensions

(dimensions 6-10) receiving a tot al weighting of 33% (6.6% each) and t he remaining 5 (dimensions 11-15)

receiving a total weighting of 17% (3.4% each). In order to evaluate the effect of this choice, the scores and

rank of subjects are compared with those that would be obtained using equal weighting.

Inclusion of dimensi on 14 (Time Use). The Time Use dimension has been proven to be poorly correlated with

the rest of the framework. For this reason, the effect of its exc lusion from the final aggregation is evaluat ed.

To compare the three alternative options on aggregation formula, structure and weights with the proposed one,

four models were considered and they can be seen in Table 8.

It is important to mention that the results of the analysi s are shown only for the values of the 45 areas of Fiji,

since the representation of all the individuals does not seem to be informative in this context.