Time series in OpenBudgets

TimeSeries.OBeu is used on OpenBudgets.eu data mininig tool platform with OpenCPU integration of R and JavaScript to estimate and return the needed parameters for visualizations designed for OpenBudgets.eu http://openbudgets.eu/ time series budget or expenditure data.

The vignette shows the way TimeSeries.OBeu (in R and OpenCPU environment) is fitted with datasets of OpenBudgets.eu according to the OpenBudgets.eu data model. Detailed documentation about OpenBudgets.eu data model can be found here

The input and the resulted object are in json format.

First you have to load the library

# load TimeSeries.OBeu
library(TimeSeries.OBeu)

Time Series analysis on OpenBudgets.eu platform

open_spending.ts is designed to estimate autocorrelation and partial autocorrelation of input time series data, autocorrelation and partial autocorrelation of the model residuals, trend, seasonal (if exists) and remainder components, model parameters such as arima order, arima coefficients etc. and the desired forecasts with their corresponding confidence intervals of OpenBudgets.eu time series data.

The input data must be a JSON link according to the OpenBudgets.eu data model. The user should specify the amount and time variables, future steps to be predicted (default is 1 step forward) and the arima order (if not specified the most appropriate model will be selected according to AIC value).

open_spending.ts input
Input Description
json_data The json string, URL or file from Open Spending API
time the time label of the json time series data
amount the amount label of the json time series data
order An integer vector of length 3 specifying the order of the Arima model
prediction_steps The number of prediction steps

Stationary test

TimeSeries.OBeu includes function that automatically tests the deterministic and stochastic trend of the input time series data. This function uses ACF and PACF functions from forecast package, Phillips Perron test, Augmented Dickey Fuller (ADF) test, Kwiatkowski Phillips Schmidt Shin (KPSS) test, from tseries package and Mann Kendall test for Monotonic Trend Cox Stuart trend test from trend package.

Decomposition

Decomposition of seasonal time series data through arima models is based on stlm from forecast package and returns a list with useful parameters for OBEU. For non-seasonal time series that there is no seasonal component, local regression and likelihood models (locfit package) are used in order to extract the trend and remaider components.

Model Fitting

Model fit of time series using arima models of seasonal time series data. The model with the lowest AIC value is selected for forecasts.

Forecast

Forecasts the model that fits the input data using the auto.arima function(see forecast package). The model selection depends on the results of some diagnostic tests (acf,pacf,pp adf and kpss). For short time series the selected arima model is among various orders of the AR part using the first differences and the first order moving average component, with the lower AIC value.

Output

Here is a sort description of the outputs of each function:

open_spending.ts return
Component Output Description
acf.parameters
  • acf
  • acf.lag
  • confidence.interval.up
  • confidence.interval.low
  • ACF values of the input time series
  • Lags at which the acf is estimated
  • Upper limit of the confidence interval
  • Lower limit of the confidence interval
pacf.parameters
  • pacf
  • pacf.lag
  • confidence.interval.up
  • confidence.interval.low
  • PACF values of the input time series
  • Lags at which the pacf is estimated
  • Upper limit of the confidence interval
  • Lower limit of the confidence interval
acf.residuals.parameters
  • acf.res
  • acf.res.lag
  • confidence.interval.up
  • confidence.interval.low
  • ACF values of the model residuals
  • Lags at which the acf is estimated of the model residuals
  • Upper limit of the confidence interval
  • Lower limit of the confidence interval
pacf.residuals.parameters
  • pacf.res
  • pacf.res.lag
  • confidence.interval.up
  • confidence.interval.low
  • Pacf values of the model residuals
  • Lags at which the pacf is estimated of the model residuals
  • Upper limit of confidence interval
  • Lower limit of confidence interval
stl.plot
  • trend
  • trend.ci.up
  • trend.ci.low
  • seasonal
  • remainder
  • time
  • Trend component
  • Up limit for trend component
  • Low limit for trend component
  • Seasonal component
  • Remainder component
  • Time of the series was sampled
stl.general
  • stl.degree
  • degfr
  • degfr.fitted
  • fitted
  • Degree of fit
  • Effective degrees of freedom
  • Fitted degrees of freedom
  • Model’s fitted values
residuals
  • residuals
  • Residuals of the model
compare
  • arima.order
  • arima.coef
  • arima.coef.se
  • covariance.coef
  • resid.variance
  • not.used.obs
  • used.obs
  • loglik
  • aic
  • bic
  • gcv
  • aicc
  • Arima order
  • AR, MA and regression coefficients
  • Standard error of the coefficients
  • Variance of the coefficients
  • Residuals variance
  • Number of not used observations
  • Used observations
  • Maximized log-likelihood,
  • AIC value
  • BIC value
  • Generalized cross-validation statistic
  • Second-order AIC
forecasts
  • ts.model
  • data_year
  • data
  • predict_time
  • predict_values
  • up80
  • low80
  • up95
  • low95
  • A string indicating the arima orders
  • Time of time series data
  • Time series values
  • Time of the predicted values
  • Predicted values
  • Upper 80% confidence limit
  • Lower 80% confidence limit
  • Upper 95% confidence limit
  • Lower 95% confidence limit

Examples

The dataset that is being used is a sample json string from OpenBudgets.eu platform.

In R environment

open_spending.ts function’s input are data as json link and described with OpenBudgets.eu data model.

#example openbudgets.eu time series data
sample.ts.data = 
'{"page":0,
"page_size": 30,
"total_cell_count": 15,
"cell": [],
"status": "ok",
"cells": [{
        "global__fiscalPeriod__28951.notation": "2002",
        "global__amount__0397f.sum": 290501420.64,
        "global__amount__0397f__CZK.sum": 9210928544.2325,
        "_count": 4805
    },
    {
        "global__fiscalPeriod__28951.notation": "2003",
        "global__amount__0397f.sum": 311242291.07,
        "global__amount__0397f__CZK.sum": 9832143974.9013,
        "_count": 4988
    },
    {
        "global__fiscalPeriod__28951.notation": "2004",
        "global__amount__0397f.sum": 5268500701.1,
        "global__amount__0397f__CZK.sum": 170688885714.24,
        "_count": 10055
    },
    {
        "global__fiscalPeriod__28951.notation": "2005",
        "global__amount__0397f.sum": 2542887761.01,
        "global__amount__0397f__CZK.sum": 77204615312.025,
        "_count": 2032
    },
    {
        "global__fiscalPeriod__28951.notation": "2006",
        "global__amount__0397f.sum": 14803951786.68,
        "global__amount__0397f__CZK.sum": 429758720367.32,
        "_count": 13632
    },
    {
        "global__fiscalPeriod__28951.notation": "2007",
        "global__amount__0397f.sum": 16188514346.44,
        "global__amount__0397f__CZK.sum": 445588857385.76,
        "_count": 22798
    },
    {
        "global__fiscalPeriod__28951.notation": "2008",
        "global__amount__0397f.sum": 18231035815.89,
        "global__amount__0397f__CZK.sum": 480643028250.12,
        "_count": 24176
    },
    {
        "global__fiscalPeriod__28951.notation": "2009",
        "global__amount__0397f.sum": 19079541164.68,
        "global__amount__0397f__CZK.sum": 511808691742.54,
        "_count": 26250
    },
    {
        "global__fiscalPeriod__28951.notation": "2010",
        "global__amount__0397f.sum": 22738650575.01,
        "global__amount__0397f__CZK.sum": 597685430364.14,
        "_count": 87667
    },
    {
        "global__fiscalPeriod__28951.notation": "2011",
        "global__amount__0397f.sum": 24961375670.57,
        "global__amount__0397f__CZK.sum": 626230992823.26,
        "_count": 134352
    },
    {
        "global__fiscalPeriod__28951.notation": "2012",
        "global__amount__0397f.sum": 261513607691.41,
        "global__amount__0397f__CZK.sum": 7030666436872.5,
        "_count": 147556
    },
    {
        "global__fiscalPeriod__28951.notation": "2013",
        "global__amount__0397f.sum": 268946402299.09,
        "global__amount__0397f__CZK.sum": 7226220232913.8,
        "_count": 150079
    },
    {
        "global__fiscalPeriod__28951.notation": "2014",
        "global__amount__0397f.sum": 255222816704.9,
        "global__amount__0397f__CZK.sum": 6907598086283.4,
        "_count": 176019
    },
    {
        "global__fiscalPeriod__28951.notation": "2015",
        "global__amount__0397f.sum": 22976062973.62,
        "global__amount__0397f__CZK.sum": 636276111928.46,
        "_count": 213777
    },
    {
        "global__fiscalPeriod__28951.notation": "2016",
        "global__amount__0397f.sum": 12051686541.16,
        "global__amount__0397f__CZK.sum": 325672725401.77,
        "_count": 161797
    }
],
"order": [
    ["global__fiscalPeriod__28951.fiscalPeriod", "asc"]
],
"aggregates": ["", "_count"],
"summary": {
    "global__amount__0397f.sum": 945126777743.27,
    "global__amount__0397f__CZK.sum": 25485085887878
},
"attributes": [""]
}'
## Registered S3 method overwritten by 'xts':
##   method     from
##   as.zoo.xts zoo
## Registered S3 method overwritten by 'quantmod':
##   method            from
##   as.zoo.data.frame zoo
## Registered S3 methods overwritten by 'forecast':
##   method             from    
##   fitted.fracdiff    fracdiff
##   residuals.fracdiff fracdiff
## Warning in tseries::kpss.test(tsdata): p-value greater than printed p-value
## {
##   "acf.param": {
##     "acf.parameters": {
##       "acf": [
##         1,
##         0.6083,
##         0.1674,
##         -0.1663,
##         -0.1295,
##         -0.0727,
##         -0.0925,
##         -0.1301,
##         -0.1615,
##         -0.1959,
##         -0.2115,
##         -0.1311
##       ],
##       "acf.lag": [
##         0,
##         1,
##         2,
##         3,
##         4,
##         5,
##         6,
##         7,
##         8,
##         9,
##         10,
##         11
##       ],
##       "confidence.interval.up": [
##         0.5061
##       ],
##       "confidence.interval.low": [
##         -0.5061
##       ]
##     },
##     "pacf.parameters": {
##       "pacf": [
##         0.6083,
##         -0.3215,
##         -0.1865,
##         0.25,
##         -0.1593,
##         -0.1764,
##         0.0869,
##         -0.1346,
##         -0.2117,
##         -0.0036,
##         0.0508
##       ],
##       "pacf.lag": [
##         1,
##         2,
##         3,
##         4,
##         5,
##         6,
##         7,
##         8,
##         9,
##         10,
##         11
##       ],
##       "confidence.interval.up": [
##         0.5061
##       ],
##       "confidence.interval.low": [
##         -0.5061
##       ]
##     },
##     "acf.residuals.parameters": {
##       "acf.residuals": [
##         1,
##         0.3097,
##         0.2296,
##         -0.2346,
##         -0.0115,
##         -0.069,
##         -0.0524,
##         -0.0981,
##         -0.0842,
##         -0.1215,
##         -0.0934,
##         -0.0868,
##         -0.0484,
##         -0.2128,
##         -0.115,
##         -0.1051,
##         0.2946
##       ],
##       "acf.residuals.lag": [
##         0,
##         1,
##         2,
##         3,
##         4,
##         5,
##         6,
##         7,
##         8,
##         9,
##         10,
##         11,
##         12,
##         13,
##         14,
##         15,
##         16
##       ],
##       "confidence.interval.up": [
##         0.5061
##       ],
##       "confidence.interval.low": [
##         -0.5061
##       ]
##     },
##     "pacf.residuals.parameters": {
##       "pacf.residuals": [
##         0.3097,
##         0.1479,
##         -0.3857,
##         0.1673,
##         0.0455,
##         -0.2432,
##         0.0379,
##         0.0137,
##         -0.2159,
##         0.0048,
##         0.0175,
##         -0.1445,
##         -0.2757,
##         0.0882,
##         -0.0175,
##         0.2238
##       ],
##       "pacf.residuals.lag": [
##         1,
##         2,
##         3,
##         4,
##         5,
##         6,
##         7,
##         8,
##         9,
##         10,
##         11,
##         12,
##         13,
##         14,
##         15,
##         16
##       ],
##       "confidence.interval.up": [
##         0.5061
##       ],
##       "confidence.interval.low": [
##         -0.5061
##       ]
##     }
##   },
##   "decomposition": {
##     "stl.plot": {
##       "trend": [
##         -823419544.0324,
##         1661560665.8427,
##         4624784832.814,
##         7878983908.9168,
##         9164365783.7901,
##         1249040775.5615,
##         -4351015667.1447,
##         6551641382.3009,
##         57664029716.7199,
##         135646130025.509,
##         199114831580.159,
##         212547970271.575,
##         183231679544.124,
##         110152904455.055,
##         -12061960507.0845
##       ],
##       "conf.interval.up": [
##         100039247757.031,
##         66576136730.7478,
##         60840745924.5652,
##         68328241466.4622,
##         72409579664.1255,
##         65432105294.9799,
##         59676059485.8763,
##         70171989437.0366,
##         121691104869.741,
##         199829194544.927,
##         262360045460.495,
##         272997227829.121,
##         239447640635.875,
##         175067480519.96,
##         88800706793.9786
##       ],
##       "conf.interval.low": [
##         -101686086845.095,
##         -63253015399.0623,
##         -51591176258.9372,
##         -52570273648.6285,
##         -54080848096.5454,
##         -62934023743.857,
##         -68378090820.1657,
##         -57068706672.4349,
##         -6363045436.3011,
##         71463065506.0904,
##         135869617699.824,
##         152098712714.03,
##         127015718452.373,
##         45238328390.1502,
##         -112924627808.148
##       ],
##       "seasonal": {
## 
##       },
##       "remainder": [
##         1113920964.6724,
##         -1350318374.7727,
##         643715868.286,
##         -5336096147.9068,
##         5639586002.8899,
##         14939473570.8785,
##         22582051483.0347,
##         12527899782.3791,
##         -34925379141.7099,
##         -110684754354.939,
##         62398776111.2508,
##         56398432027.5148,
##         71991137160.7759,
##         -87176841481.4353,
##         24113647048.2445
##       ],
##       "time": [
##         2002,
##         2003,
##         2004,
##         2005,
##         2006,
##         2007,
##         2008,
##         2009,
##         2010,
##         2011,
##         2012,
##         2013,
##         2014,
##         2015,
##         2016
##       ]
##     },
##     "stl.general": {
##       "degfr": [
##         5.288
##       ],
##       "degfr.fitted": [
##         4.9747
##       ],
##       "stl.degree": [
##         2
##       ]
##     },
##     "residuals_fitted": {
##       "residuals": [
##         1113920964.6724,
##         -1350318374.7727,
##         643715868.286,
##         -5336096147.9068,
##         5639586002.8899,
##         14939473570.8785,
##         22582051483.0347,
##         12527899782.3791,
##         -34925379141.7099,
##         -110684754354.939,
##         62398776111.2508,
##         56398432027.5148,
##         71991137160.7759,
##         -87176841481.4353,
##         24113647048.2445
##       ],
##       "fitted": [
##         -823419544.0324,
##         1661560665.8427,
##         4624784832.814,
##         7878983908.9168,
##         9164365783.7901,
##         1249040775.5615,
##         -4351015667.1447,
##         6551641382.3009,
##         57664029716.7199,
##         135646130025.509,
##         199114831580.159,
##         212547970271.575,
##         183231679544.124,
##         110152904455.055,
##         -12061960507.0845
##       ],
##       "time": [
##         2002,
##         2003,
##         2004,
##         2005,
##         2006,
##         2007,
##         2008,
##         2009,
##         2010,
##         2011,
##         2012,
##         2013,
##         2014,
##         2015,
##         2016
##       ],
##       "line": [
##         0
##       ]
##     },
##     "compare": {
##       "resid.variance": [
##         2.4902231028103e+021
##       ],
##       "used.obs": [
##         2002,
##         2016,
##         2009,
##         2005.5,
##         2012.5
##       ],
##       "loglik": [
##         -1.74315617196721e+022
##       ],
##       "aic": [
##         3.48631234393441e+022
##       ],
##       "bic": [
##         3.48631234393441e+022
##       ],
##       "gcv": [
##         5.54416871365202e+021
##       ]
##     }
##   },
##   "model.param": {
##     "model": {
##       "arima.order": [
##         2,
##         1,
##         0,
##         0,
##         1,
##         1,
##         0
##       ],
##       "arima.coef": [
##         0.8348,
##         -0.249,
##         -0.9999
##       ],
##       "arima.coef.se": [
##         0.2524,
##         0.2482,
##         0.5954
##       ]
##     },
##     "residuals_fitted": {
##       "residuals": [
##         290501.235,
##         18348492.233,
##         4388547106.9232,
##         -2696772619.0425,
##         12279728464.2613,
##         1663580423.4101,
##         5162045969.9383,
##         4109968753.4051,
##         6995758560.2579,
##         5772141482.9292,
##         231395399804.073,
##         31316280480.3554,
##         66705685366.0788,
##         -149540618863.384,
##         33819215962.2688
##       ],
##       "fitted": [
##         290210919.405,
##         292893798.837,
##         879953594.1768,
##         5239660380.0525,
##         2524223322.4187,
##         14524933923.0299,
##         13068989845.9517,
##         14969572411.2749,
##         15742892014.7521,
##         19189234187.6408,
##         30118207887.337,
##         237630121818.735,
##         188517131338.821,
##         172516681837.004,
##         -21767529421.1088
##       ],
##       "time": [
##         2002,
##         2003,
##         2004,
##         2005,
##         2006,
##         2007,
##         2008,
##         2009,
##         2010,
##         2011,
##         2012,
##         2013,
##         2014,
##         2015,
##         2016
##       ],
##       "line": [
##         0
##       ]
##     },
##     "compare": {
##       "resid.variance": [
##         7.52601939136356e+021
##       ],
##       "variance.coef": [
##         [
##           0.0637,
##           -0.034,
##           -0.0003
##         ],
##         [
##           -0.034,
##           0.0616,
##           -0.0002
##         ],
##         [
##           -0.0003,
##           -0.0002,
##           0.3545
##         ]
##       ],
##       "not.used.obs": [
##         0
##       ],
##       "used.obs": [
##         14
##       ],
##       "loglik": [
##         -371.6686
##       ],
##       "aic": [
##         751.3372
##       ],
##       "bic": [
##         753.8934
##       ],
##       "aicc": [
##         755.7816
##       ]
##     }
##   },
##   "forecasts": {
##     "ts.model": [
##       "ARIMA(2,1,1)"
##     ],
##     "data_year": [
##       2002,
##       2003,
##       2004,
##       2005,
##       2006,
##       2007,
##       2008,
##       2009,
##       2010,
##       2011,
##       2012,
##       2013,
##       2014,
##       2015,
##       2016
##     ],
##     "data": [
##       290501420.64,
##       311242291.07,
##       5268500701.1,
##       2542887761.01,
##       14803951786.68,
##       16188514346.44,
##       18231035815.89,
##       19079541164.68,
##       22738650575.01,
##       24961375670.57,
##       261513607691.41,
##       268946402299.09,
##       255222816704.9,
##       22976062973.62,
##       12051686541.16
##     ],
##     "predict_time": [
##       2017
##     ],
##     "predict_values": [
##       27966099866.1595
##     ],
##     "up80": [
##       142431304484.384
##     ],
##     "low80": [
##       -86499104752.0649
##     ],
##     "up95": [
##       203025523588.521
##     ],
##     "low95": [
##       -147093323856.202
##     ]
##   }
## }
##

In OpenCPU environment

Select library and function

  1. Go to: yourserver/ocpu/test

  2. Copy and paste the following function to the endpoint

  1. Select Method: Post

Add parameters

Click add parameters every time you want to add a new parameters and values.

  1. Define the input data:

    • Param Name: json_data
    • Param Value (URL of json data or json string): "http://next.openspending.org/api/3/cubes/21260d070eb5d63a121ea4c400dafbbb:apbn_fungsi_2016/facts?pagesize=20" (or any other json URL with the data)

  2. Define the time parameter:

    • Param Name: time
    • Param Value: "global__fiscalPeriod__28951.notation"

  3. Define the amount parameter:

    • Param Name: amount
    • Param Value: "global__amount__0397f.sum"

You add likewise x.order parameter to fit a specific arima order, see TimeSeries.OBeu reference manual for further details.

  1. Ready! Click on Ajax request!

Results

  1. copy the /ocpu/tmp/{this_id_number}/R/.val (second on the right panel)

  2. finally, paste yourserver/ocpu/tmp/{this_id_number}/R/.val on a new tab.

Further Details

Github