2. Basic statistics for histogram data
The statistics of HistDAWass are based on the L2 Wasserstein distance between two distributions \(f\) and \(g\), i.e. the Euclidean distance between their quantile functions \(Q_f\) and \(Q_g\): \[d_W^2(f, g) = \int_0^1 \left(Q_f(p) - Q_g(p)\right)^2 dp.\] With this distance, the mean of a set of histograms is the histogram whose quantile function is the mean of the quantile functions, and variances, covariances and correlations are defined accordingly (Irpino and Verde, 2015).
We use the BLOOD data: 14 groups of patients described
by three histogram-valued variables.
BLOOD
#> a matrix of distributions
#> 3 variables 14 rows
#> each distibution in the cell is represented by the mean and the standard deviation
#> Cholesterol Hemoglobin Hematocrit
#> u1: F-20 [m= 150.1 ,s= 26.336 ] [m= 13.695 ,s= 0.55031 ] [m= 41.526 ,s= 2.1968 ]
#> u2: F-30 [m= 150.71 ,s= 25.284 ] [m= 12.158 ,s= 0.52834 ] [m= 36.497 ,s= 2.1225 ]
#> u3: F-40 [m= 164.96 ,s= 25.334 ] [m= 12.134 ,s= 0.50739 ] [m= 36.549 ,s= 2.2299 ]
#> u4: F-50 [m= 186.51 ,s= 26.655 ] [m= 12.133 ,s= 0.58514 ] [m= 36.48 ,s= 2.1985 ]
#> u5: F-60 [m= 194.03 ,s= 25.215 ] [m= 12.145 ,s= 0.52031 ] [m= 36.341 ,s= 2.0979 ]
#> u6: F-70 [m= 193.2 ,s= 26.561 ] [m= 12.205 ,s= 0.52258 ] [m= 36.703 ,s= 2.1818 ]
#> u7: F-80+ [m= 187.14 ,s= 24.592 ] [m= 12.141 ,s= 0.55247 ] [m= 36.503 ,s= 2.1911 ]
#> u8: M-20 [m= 159.62 ,s= 19.844 ] [m= 13.557 ,s= 0.29974 ] [m= 40.5 ,s= 1.6358 ]
#> u9: M-30 [m= 164.43 ,s= 26.486 ] [m= 12.088 ,s= 0.62237 ] [m= 35.914 ,s= 2.1141 ]
#> u10: M-40 [m= 170.06 ,s= 20.011 ] [m= 12.092 ,s= 0.52656 ] [m= 36.456 ,s= 2.2476 ]
#> u11: M-50 [m= 194.22 ,s= 30.165 ] [m= 12.214 ,s= 0.59708 ] [m= 36.72 ,s= 2.0024 ]
#> u12: M-60 [m= 203.36 ,s= 26.223 ] [m= 12.245 ,s= 0.50862 ] [m= 35.814 ,s= 2.0083 ]
#> u13: M-70 [m= 205.66 ,s= 22.499 ] [m= 12.15 ,s= 0.33425 ] [m= 35.75 ,s= 2.1651 ]
#> u14: M-80+ [m= 205.48 ,s= 23.537 ] [m= 12.12 ,s= 0.6163 ] [m= 38.45 ,s= 2.6158 ]Statistics of a single histogram
d <- BLOOD@M[[1, 1]] # cholesterol of the first group
d
#> X p
#> Bin_1 [ 80 ; 100 ) 0.025
#> Bin_2 [ 100 ; 120 ) 0.075
#> Bin_3 [ 120 ; 135 ) 0.175
#> Bin_4 [ 135 ; 150 ) 0.25
#> Bin_5 [ 150 ; 165 ) 0.2
#> Bin_6 [ 165 ; 180 ) 0.162
#> Bin_7 [ 180 ; 200 ) 0.088
#> Bin_8 [ 200 ; 240 ] 0.025
#>
#> mean = 150.1025 std = 26.3356126518827
#>
meanH(d) # mean
#> [1] 150.1025
stdH(d) # standard deviation
#> [1] 26.33561
skewH(d) # skewness
#> [1] 0.2293406
kurtH(d) # kurtosis (3 for a Gaussian)
#> [1] 3.366591
compQ(d, 0.5) # the median
#> [1] 148.5
compQ(d, 0.25); compQ(d, 0.75) # first and third quartiles
#> [1] 132.8571
#> [1] 167.3148
compP(d, 150) # the cdf at 150
#> [1] 0.525The centres, the half-widths and the probabilities of the bins:
Arithmetic and distance between two histograms
Sums and products by a number act on the quantile functions:
d1 <- BLOOD@M[[1, 1]]
d2 <- BLOOD@M[[2, 1]]
d12 <- 0.5 * d1 + 0.5 * d2 # the Wasserstein mean of the two histograms
c(meanH(d1), meanH(d2), meanH(d12))
#> [1] 150.1025 150.7100 150.4063The squared distance and its decomposition into a
position (difference of the means), a
size (difference of the standard deviations) and a
shape component; rQQ is the correlation
between the quantile functions:
Statistics of each histogram of a matrix
get.MatH.stats() returns a matrix with a statistic for
each cell:
get.MatH.stats(BLOOD, stat = "mean")$mat
#> Cholesterol Hemoglobin Hematocrit
#> u1: F-20 150.1025 13.69530 41.52550
#> u2: F-30 150.7100 12.15805 36.49700
#> u3: F-40 164.9600 12.13355 36.54900
#> u4: F-50 186.5075 12.13270 36.48025
#> u5: F-60 194.0300 12.14510 36.34075
#> u6: F-70 193.2000 12.20480 36.70300
#> u7: F-80+ 187.1400 12.14085 36.50350
#> u8: M-20 159.6225 13.55700 40.50000
#> u9: M-30 164.4300 12.08820 35.91400
#> u10: M-40 170.0550 12.09160 36.45650
#> u11: M-50 194.2200 12.21360 36.72000
#> u12: M-60 203.3600 12.24500 35.81450
#> u13: M-70 205.6650 12.15000 35.75000
#> u14: M-80+ 205.4750 12.11960 38.45000
get.MatH.stats(BLOOD, stat = "std")$mat[1:5, ]
#> Cholesterol Hemoglobin Hematocrit
#> u1: F-20 26.33561 0.5503071 2.196782
#> u2: F-30 25.28364 0.5283404 2.122457
#> u3: F-40 25.33374 0.5073865 2.229933
#> u4: F-50 26.65510 0.5851359 2.198471
#> u5: F-60 25.21492 0.5203070 2.097889
get.MatH.stats(BLOOD, stat = "quantile", prob = 0.9)$mat[1:5, ]
#> Cholesterol Hemoglobin Hematocrit
#> u1: F-20 182.9545 14.40000 44.74390
#> u2: F-30 181.1364 12.72166 39.27941
#> u3: F-40 197.0608 12.73448 39.32057
#> u4: F-50 220.2000 12.87619 39.33929
#> u5: F-60 227.9452 12.82787 39.00549Available statistics: "mean", "median",
"min", "max", "std",
"skewness", "kurtosis" and
"quantile".
Mean histogram, variance, covariance and correlation of the variables
The mean histogram (the Wasserstein barycentre) of a variable:
m_chol <- WH.vec.mean(BLOOD[, "Cholesterol"])
m_chol
#> Output shows the first five and the last five bins due to eccesive length
#> X p
#> Bin_1 [113.57 ; 121.57) 0.009
#> Bin_2 [121.57 ; 124) 0.003
#> Bin_3 [124 ; 124.63) 0.001
#> Bin_4 [124.63 ; 126.78) 0.004
#> Bin_5 [126.78 ; 127.25) 0.001
#> ... ... ...
#> Bin_80 [229.74 ; 231.18) 0.003
#> Bin_81 [231.18 ; 234.99) 0.007
#> Bin_82 [234.99 ; 235.66) 0.001
#> Bin_83 [235.66 ; 239.15) 0.004
#> Bin_84 [239.15 ; 246.79) 0.008
#>
#> mean = 180.676964285714 std = 24.7830931266904
#>
plot(m_chol)
Weights can be given, e.g. proportional to the size of the groups:
The variance-covariance and the correlation matrices of the histogram variables, and the matrix of the sums of squared deviations from the mean:
WH.var.covar(BLOOD)
#> Cholesterol Hemoglobin Hematocrit
#> Cholesterol 388.137633 -5.0005134 -14.9202378
#> Hemoglobin -5.000513 0.2802215 0.8266558
#> Hematocrit -14.920238 0.8266558 2.9779786
WH.correlation(BLOOD)
#> Cholesterol Hemoglobin Hematocrit
#> Cholesterol 1.0000000 -0.4794806 -0.4388560
#> Hemoglobin -0.4794806 1.0000000 0.9049264
#> Hematocrit -0.4388560 0.9049264 1.0000000
WH.SSQ(BLOOD)
#> Cholesterol Hemoglobin Hematocrit
#> Cholesterol 5433.92687 -70.007187 -208.88333
#> Hemoglobin -70.00719 3.923101 11.57318
#> Hematocrit -208.88333 11.573182 41.69170The versions with suffix 2 compute the same quantities between the variables of two matrices with the same rows:
Distances between individuals
The matrix of the L2 Wasserstein distances between the rows of a
MatH (summing the squared distances over the variables),
optionally after standardising the variables:
D <- WH_MAT_DIST(BLOOD, standardize = TRUE)
round(D[1:5, 1:5], 3)
#> u1: F-20 u2: F-30 u3: F-40 u4: F-50 u5: F-60
#> u1: F-20 0.000 4.123 4.204 4.553 4.761
#> u2: F-30 4.123 0.000 0.748 1.840 2.215
#> u3: F-40 4.204 0.748 0.000 1.122 1.496
#> u4: F-50 4.553 1.840 1.122 0.000 0.455
#> u5: F-60 4.761 2.215 1.496 0.455 0.000For large matrices of histograms with different bins,
simplify = TRUE approximates each histogram with
qua equally spaced quantiles: the computations are much
faster, at the cost of an approximation error that decreases with
qua. The relative errors of the distances:
u <- upper.tri(D)
for (q in c(20, 100)) {
Dq <- WH_MAT_DIST(BLOOD, standardize = TRUE, simplify = TRUE, qua = q)
err <- abs(Dq[u] - D[u]) / D[u]
cat("qua =", q, " median:", signif(median(err), 2),
" maximum:", signif(max(err), 2), "\n")
}
#> qua = 20 median: 0.006 maximum: 0.13
#> qua = 100 median: 0.00032 maximum: 0.0075The largest relative errors concern the smallest distances; values of
qua between 50 and 100 are usually a good compromise.
Reference
Irpino, A., Verde, R. (2015) Basic statistics for distributional symbolic variables: a new metric-based approach. Advances in Data Analysis and Classification, 9(2), 143-175. https://doi.org/10.1007/s11634-014-0176-4