FORECAST PRO EXAMPLE #2: AUTOSALE revisited.

First, as a benchmark, here is a battery of models fitted in the Forecasting procedure in Statgraphics. Models A and B are the simple and linear exponential smoothing models with multiplicative seasonal adjustment. A fixed-rate inflation adjustment of 0.5% per period has been added to the simple exponential smoothing model to put some long-term trend into its forecasts and reduce error heteroscedasticity. The Winter's seasonal smoothing model has been specified as model C. Two ARIMA models fitted to the logged data are included as models D and E.

Notice that all the models are quite similar in their all-around performance in the estimation period, and models A-D are very similar in the validation period, with model E lagging a bit behind. Interestingly, the ARIMA models did relatively better in terms of percentage error because, being fitted in log units, they were actually optimized in terms of percentage error rather than absolute error.

```Model Comparison
----------------
Data variable: AUTOSALE
Number of observations = 281
Start index = 1/70
Sampling interval = 1.0 month(s)
Length of seasonality = 12
Number of periods withheld for validation: 36

Models
------
(A) Simple exponential smoothing with alpha = 0.4021
Inflation adjustment: 0.5 percent at end of period
(B) Brown's linear exp. smoothing with alpha = 0.1375
(C) Winter's exp. smoothing with alpha = 0.3117, beta = 0.0001, gamma = 0.2495
(D) ARIMA(0,1,1)x(0,1,1)12
(E) ARIMA(1,0,1)x(0,1,1)12 with constant

Estimation Period
Model  MSE          MAE          MAPE         ME           MPE
------------------------------------------------------------------------
(A)    1.40991      0.770647     4.70927      0.0662132    0.276835
(B)    1.41955      0.791537     5.02856      0.0248242    0.282996
(C)    1.53718      0.842734     5.12773      0.0154446    -0.454623
(D)    1.58972      0.823505     4.77485      -0.0302983   -0.279636
(E)    1.56714      0.819359     4.71859      0.0307884    0.143125

Model  RMSE         RUNS  RUNM  AUTO  MEAN  VAR
-----------------------------------------------
(A)    1.1874        OK    OK    ***   OK   ***
(B)    1.19145       OK    ***   ***   OK   ***
(C)    1.23983       OK    *     OK    OK   ***
(D)    1.26084       OK    OK    *     OK   **
(E)    1.25185       OK    OK    *     *    **

Validation Period
Model  MSE          MAE          MAPE         ME           MPE
------------------------------------------------------------------------
(A)    1.09317      0.947184     2.94464      -0.12697     -0.532003
(B)    1.23223      0.951346     2.94155      0.234821     0.57216
(C)    1.35863      0.959531     2.97817      0.00353942   -0.144202
(D)    1.15307      0.912145     2.82051      -0.144336    -0.571749
(E)    2.20643      1.19816      3.76333      -1.06931     -3.37841
```

Here are the estimated coefficients of the ARIMA(0,1,1)x(0,1,1) model:

```                            ARIMA Model Summary
Parameter           Estimate        Stnd. Error     t               P-value
----------------------------------------------------------------------------
MA(1)               0.50284         0.0570327       8.8167          0.000000
SMA(1)              0.896011        0.0243729       36.7626         0.000000
----------------------------------------------------------------------------
Backforecasting: yes
Estimated white noise variance = 0.0040629 with 266 degrees of freedom
Estimated white noise standard deviation = 0.0637409
Number of iterations: 7

```

Now let's fit the same series using the built-in expert in Forecast Pro:

```Forecast Pro for Windows Standard Edition Version 2.00
Sat Oct 05 17:12:33 1996

Expert data exploration of dependent variable AUTOSALE
---------------------------------------------------------------------
Length 245  Minimum 4.485  Maximum 37.068
Mean 16.781 Standard deviation 8.770

Classical decomposition (multiplicative)
Trend-cycle: 96.53%  Seasonal: 2.55%  Irregular: 0.92%

Log transform recommended for Box-Jenkins.

There are no strongly significant regressors, so I will choose
a univariate method.

Exponential smoothing outperforms Box-Jenkins by 1.754 to 1.939
out-of-sample (MAD). I tried 78 forecasts up to a maximum horizon 12.
For Box-Jenkins, I used a log transform.

```

Out-of-sample forecast errors are used by Forecast Pro to compare models of different types during the automatic data exploration phase. The 78 forecasts are made on a rolling basis for a 12-month period, yielding 12 one-step ahead forecasts, 11 two-step-ahead forecasts, etc., for a total of 78 forecasts at various horizons within the same year. This is a generally good method to compare models, but keep in mind that it focuses only on a single year: some models may get luckier than others in that year!

```Series is trended and seasonal.

Recommended model: Exponential smoothing

```

OK--let's use exponential smoothing with the "automatic" estimation option. Also, let's use 36 months for the validation period, as we did in Statgraphics.

```Forecast Model for AUTOSALE
Automatic model selection
Multiplicative Winters: Linear trend, Multiplicative seasonality
Confidence limits proportional to indexes and level

Smoothing     Final
Component           Weight       Value
--------------------------------------
Level              0.29731      32.557
Trend              0.01889     0.12847
Seasonal           0.22900      1.1080

Seasonal Indexes
----------------------------------------------------------
January - March            0.87334     0.89532     1.06630
April - June               1.05658     1.10797     1.11531
July - September           1.06250     1.07803     1.01079
October - December         0.96825     0.90960     0.90090

Standard Diagnostics
-------------------------------------------------------------
Sample size 245                  Number of parameters 3
Mean 16.78                       Standard deviation 8.788
Durbin-Watson 1.91               Ljung-Box(18)=20.49 P=0.6942
Forecast error 1.198             BIC 1.232 (Best so far)
MAPE 0.04891                     RMSE 1.191

```

The Ljung-Box statistic is a test of the significance of the sum of squares of the first n residual autocorrelations--i.e., a test for the "total autocorrelation" in the residuals--where n=18 was used here. A "good" value of this statistic is a number not much larger than n, and the P-value as reported here ought to be less than 0.95. This test shows that the overall amount of autocorrelation in the residuals is acceptably low.

```Rolling simulation results
Cumulative        Cumulative
H   N       MAD      Average   MAPE    Average
---------------------------------------------------------------------
1  36      0.974      0.974    0.030    0.030
2  35      0.982      0.978    0.031    0.031
3  34      1.074      1.009    0.034    0.032
4  33      1.237      1.064    0.039    0.033
5  32      1.309      1.110    0.041    0.035
6  31      1.554      1.178    0.049    0.037
7  30      1.697      1.246    0.053    0.039
8  29      1.742      1.301    0.053    0.041
9  28      1.843      1.354    0.055    0.042
10  27      1.960      1.406    0.058    0.043
11  26      2.020      1.453    0.060    0.045
12  25      2.147      1.500    0.064    0.046
13  24      2.226      1.545    0.067    0.047
14  23      2.188      1.580    0.066    0.048
15  22      2.308      1.617    0.069    0.049
16  21      2.336      1.650    0.070    0.050
17  20      2.419      1.683    0.072    0.051
18  19      2.496      1.714    0.074    0.052
19  18      2.518      1.742    0.074    0.053
20  17      2.470      1.765    0.071    0.053
21  16      2.569      1.789    0.073    0.054
22  15      2.582      1.810    0.073    0.054
23  14      2.413      1.825    0.068    0.055
24  13      2.360      1.837    0.066    0.055
25  12      2.168      1.843    0.060    0.055
26  11      1.857      1.844    0.050    0.055
27  10      1.766      1.842    0.048    0.055
28   9      1.223      1.833    0.033    0.055
29   8      0.844      1.821    0.023    0.054
30   7      0.785      1.810    0.023    0.054
31   6      0.832      1.801    0.023    0.054
32   5      0.843      1.794    0.024    0.053
33   4      0.970      1.789    0.027    0.053
34   3      0.958      1.785    0.025    0.053
35   2      0.663      1.781    0.017    0.053
36   1      1.194      1.781    0.030    0.053

```

Notice that the estimated coefficients are similar but not identical to those found by Statgraphics--with nonlinear estimation, it's not guaranteed that you will get exactly the same results, and the two programs may perform backforecasting or otherwise initialize the models in different ways. (Fortunately, Forecast Pro prints out the final estimates of the seasonal indices, unlike Statgraphics. Neither program reveals the starting values it uses, which is one reason the Winters model is a bit of a black box.) The first row of the "rolling simulation rsults"--which shows the 1-step ahead error statistics in the validation period--agrees reasonably closely with what we obtained in Statgraphics (MAE=0.96 in SG versus MAD=0.974 here).

This model gets very lucky with its long-term forecasts near the end of the validation period, as shown here;

Now let's try Box-Jenkins with a log transform. (We need to return to the "tableau" to add the log transformation.) We'll start by using the "automatic" estimation option here too.

```Forecast Model for AUTOSALE (Log transform)
Automatic model selection
ARIMA(0,1,1)*(2,0,3)

Term          Coefficient  Std. Error  t-Statistic  Significance
---------------------------------------------------------------------
b[1]             0.4733       0.0571       8.2876       1.0000
A[12]            0.4403       0.2779       1.5844       0.8869
A[24]            0.5578       0.2776       2.0092       0.9555
B[12]            0.1197       0.2725       0.4393       0.3396
B[24]            0.5286       0.2115       2.4995       0.9876
B[36]            0.2467       0.0693       3.5622       0.9996

Embedded insignificant AR terms -- consider dynamic regression.

```

Yike--what happened here? How did we end up with SAR=2 and SMA=3 with NO seasonal difference? At first glance this model appears VERY strange, but upon closer inspection, some interesting facts emerge. Notice that the two SAR coefficients add up to 0.9981--i.e., almost exactly 1.0. This means there is a UNIT ROOT IN THE SAR PART OF THE MODEL!! Effectively the model is performing a seasonal difference after all--it's just buried in the SAR part of the model. What we ought to do (minimally) is add a seasonal difference while reducing SAR from 2 to 1--and I would try reducing the orders of seasonal terms even further. (I suspect the "expert system" in this program is trying too hard to eliminate the stubborn autocorrelation we saw around the seasonal period when we fitted the series in Statgraphics.) But let's continue for now...

```Standard Diagnostics
------------------------------------------------------------
Sample size 244                  Number of parameters 6
Mean 2.682                       Standard deviation 0.5428
Durbin-Watson 1.894              Ljung-Box(18)=28.8 P=0.9491
Forecast error 0.06107           BIC 0.9432 (Best so far)
MAPE 0.04579                     RMSE 1.165

Rolling simulation results
Cumulative        Cumulative
H   N       MAD      Average   MAPE    Average
---------------------------------------------------------------------
1  36      0.893      0.893    0.028    0.028
2  35      0.890      0.892    0.028    0.028
3  34      0.922      0.902    0.030    0.028
4  33      1.119      0.954    0.036    0.030
5  32      1.080      0.977    0.035    0.031
6  31      1.276      1.023    0.041    0.033
7  30      1.389      1.071    0.044    0.034
8  29      1.442      1.112    0.045    0.035
9  28      1.531      1.153    0.046    0.036
10  27      1.644      1.195    0.049    0.037
11  26      1.728      1.236    0.051    0.038
12  25      1.907      1.281    0.057    0.040
13  24      1.961      1.323    0.059    0.041
14  23      1.957      1.359    0.060    0.042
15  22      2.109      1.397    0.064    0.043
16  21      2.221      1.435    0.067    0.044
17  20      2.387      1.475    0.072    0.045
18  19      2.483      1.513    0.075    0.046
19  18      2.477      1.547    0.073    0.047
20  17      2.497      1.578    0.073    0.048
21  16      2.676      1.610    0.077    0.049
22  15      2.601      1.636    0.074    0.050
23  14      2.412      1.655    0.068    0.050
24  13      2.515      1.674    0.071    0.051
25  12      2.370      1.688    0.066    0.051
26  11      2.311      1.699    0.064    0.051
27  10      2.380      1.710    0.068    0.051
28   9      2.214      1.717    0.064    0.052
29   8      1.991      1.721    0.059    0.052
30   7      2.131      1.725    0.065    0.052
31   6      2.267      1.730    0.066    0.052
32   5      2.578      1.737    0.073    0.052
33   4      3.080      1.745    0.084    0.052
34   3      3.029      1.751    0.078    0.052
35   2      2.699      1.754    0.068    0.053
36   1      4.035      1.757    0.101    0.053

```

This model actually outperforms the Winters model in its 1-step-ahead forecasts in the validation period, but it doesn't get quite as lucky with its long-term forecasts. (It's Ljung-Box statistic is also a little worse.) The confidence intervals widen fairly rapidly because of the effect of the log transformation:

```
```

Now here's one of the more "standard" seasonal ARIMA models that we identified earlier in Statgraphics:

```Forecast Model for AUTOSALE (Log transform)
ARIMA(0,1,1)*(0,1,1)

Term          Coefficient  Std. Error  t-Statistic  Significance
---------------------------------------------------------------------
b[1]             0.4981       0.0575       8.6632       1.0000
B[12]            0.8896       0.0287      30.9505       1.0000

Standard Diagnostics
-------------------------------------------------------------
Sample size 232                  Number of parameters 2
Mean 2.733                       Standard deviation 0.5058
Durbin-Watson 1.896              Ljung-Box(18)=24.39 P=0.8575
Forecast error 0.06218           BIC 0.9751
MAPE 0.04774                     RMSE 1.252

Rolling simulation results
Cumulative        Cumulative
H   N       MAD      Average   MAPE    Average
---------------------------------------------------------------------
1  36      0.907      0.907    0.028    0.028
2  35      0.947      0.927    0.030    0.029
3  34      0.974      0.942    0.031    0.030
4  33      1.207      1.005    0.039    0.032
5  32      1.209      1.044    0.039    0.033
6  31      1.379      1.095    0.044    0.035
7  30      1.544      1.154    0.049    0.037
8  29      1.551      1.198    0.048    0.038
9  28      1.644      1.241    0.050    0.039
10  27      1.814      1.290    0.054    0.040
11  26      1.820      1.331    0.055    0.042
12  25      1.931      1.372    0.058    0.043
13  24      1.964      1.408    0.060    0.044
14  23      1.819      1.431    0.057    0.044
15  22      2.037      1.462    0.063    0.045
16  21      2.151      1.493    0.066    0.046
17  20      2.204      1.523    0.068    0.047
18  19      2.304      1.553    0.070    0.048
19  18      2.246      1.578    0.067    0.049
20  17      2.332      1.602    0.068    0.049
21  16      2.425      1.626    0.069    0.050
22  15      2.518      1.650    0.071    0.051
23  14      2.526      1.671    0.071    0.051
24  13      2.510      1.690    0.071    0.052
25  12      2.417      1.704    0.068    0.052
26  11      2.566      1.720    0.074    0.052
27  10      2.769      1.737    0.081    0.053
28   9      2.596      1.749    0.077    0.053
29   8      2.655      1.760    0.079    0.053
30   7      3.202      1.776    0.095    0.054
31   6      3.329      1.790    0.096    0.054
32   5      3.646      1.804    0.103    0.055
33   4      4.122      1.818    0.112    0.055
34   3      4.173      1.829    0.107    0.055
35   2      4.019      1.836    0.101    0.055
36   1      5.210      1.841    0.130    0.055

```

Notice that its 1-period-ahead error statistics agree reasonably well with what was obtained in Statgraphics (MAD=0.907 here versus MAE=0.912 in SG). The estimated coefficients are close too (b[1]=0.498 and b[12]=0.889 here versus MA(1)=.503 and SMA(1)=0.896 in SG.) This model is a littles less lucky in its long-term forecasts, but its average performance in the validation period is not much different from those of the previous models--and it is MUCH simpler!

```
```

Finally, here's the other ARIMA model we tried in Statgraphics, which substitutes an AR(1) term for the nonseasonal difference and thereby estimates the long-term trend in the series rather than the local trend:

```Forecast Model for AUTOSALE (Log transform)
ARIMA(1,0,1)*(0,1,1)

Term          Coefficient  Std. Error  t-Statistic  Significance
---------------------------------------------------------------------
a[1]             0.9198       0.0326      28.2486       1.0000
b[1]             0.4221       0.0735       5.7396       1.0000
B[12]            0.8924       0.0271      32.9531       1.0000
_CONST           0.0072       0.0030       2.3964       0.9834

Standard Diagnostics
-------------------------------------------------------------
Sample size 233                  Number of parameters 3
Mean 2.729                       Standard deviation 0.5097
Durbin-Watson 1.938              Ljung-Box(18)=24.78 P=0.8688
Forecast error 0.06123           BIC 0.9647
MAPE 0.04699                     RMSE 1.237

Rolling simulation results
Cumulative        Cumulative
H   N       MAD      Average   MAPE    Average
---------------------------------------------------------------------
1  36      1.207      1.207    0.038    0.038
2  35      1.683      1.442    0.053    0.045
3  34      2.160      1.674    0.068    0.053
4  33      2.694      1.918    0.085    0.060
5  32      3.143      2.149    0.098    0.067
6  31      3.595      2.372    0.112    0.074
7  30      4.033      2.587    0.125    0.081
8  29      4.374      2.787    0.134    0.087
9  28      4.699      2.973    0.143    0.092
10  27      5.027      3.149    0.151    0.097
11  26      5.257      3.309    0.158    0.102
12  25      5.519      3.460    0.166    0.106
13  24      5.931      3.612    0.178    0.111
14  23      6.148      3.754    0.185    0.115
15  22      6.443      3.890    0.194    0.119
16  21      6.637      4.016    0.200    0.123
17  20      6.861      4.136    0.206    0.126
18  19      7.101      4.250    0.212    0.129
19  18      7.310      4.357    0.216    0.132
20  17      7.485      4.457    0.217    0.135
21  16      7.771      4.554    0.223    0.138
22  15      8.058      4.648    0.229    0.140
23  14      8.090      4.732    0.230    0.142
24  13      8.194      4.808    0.233    0.144
25  12      8.296      4.878    0.236    0.146
26  11      8.402      4.942    0.241    0.148
27  10      8.633      5.001    0.249    0.150
28   9      8.555      5.052    0.246    0.151
29   8      8.759      5.098    0.252    0.152
30   7      9.081      5.142    0.262    0.153
31   6      9.190      5.179    0.260    0.154
32   5      9.571      5.212    0.266    0.155
33   4     10.061      5.242    0.271    0.156
34   3     10.389      5.265    0.265    0.156
35   2     10.192      5.280    0.256    0.157
36   1     11.312      5.289    0.283    0.157
```

Despite its good intentions in trying to estimate the long-term trend in a more stable manner than the other models, this model gets embarrassed in the validation period because it fails to respond to the cyclical downturn which happens to occur right at the end of the estimation period (around the beginning of '91). The other models picked this up in one way or another and therefore were luckier in their long-term forecasts for the next three years. ("The race is not always to the swift nor the battle to the strong....") This model's 1-step-ahead performance is not too bad, though.