---
title: "Time Series Data Wrangling"
author: "Matt Dancho"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{Time Series Data Wrangling}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, echo = FALSE, message = FALSE, warning = FALSE}
knitr::opts_chunk$set(
    message = FALSE,
    warning = FALSE,
    fig.width = 8, 
    fig.height = 4.5,
    fig.align = 'center',
    out.width='95%', 
    dpi = 100
)

# devtools::load_all() # Travis CI fails on load_all()
```

__Time series data wrangling__ is an essential skill for any forecaster. `timetk` includes the essential data wrangling tools. In this tutorial, we'll cover:

- __Summarise by Time__ - For time-based aggregations
- __Filter by Time__ - For complex time-based filtering
- __Pad by Time__ - For filling in gaps and going from _low to high frequency_
- __Slidify__ - For turning any function into a sliding (rolling) function

Additional concepts covered:

- __Imputation__ - Needed for Padding (See Low to High Frequency)
- __Advanced Filtering__ - Using the new add time `%+time` infix operation (See _Padding Data: Low to High Frequency_)
- __Visualization__ - `plot_time_series()` for all visualizations

# Libraries

Load the following libraries. 

```{r}
library(tidyverse)
library(tidyquant) 
library(timetk)
```

# Data

This tutorial will use the `FANG` dataset: 

- Daily
- Irregular (missing business holidays and weekends)
- 4 groups (FB, AMZN, NFLX, and GOOG). 

```{r}
FANG
```

The adjusted column contains the adjusted closing prices for each day. 

```{r}
FANG %>%
  group_by(symbol) %>%
  plot_time_series(date, adjusted, .facet_ncol = 2, .interactive = FALSE)
```

The volume column contains the trade volume (number of times the stock was transacted) for the day. 

```{r}
FANG %>%
  group_by(symbol) %>%
  plot_time_series(date, volume, .facet_ncol = 2, .interactive = FALSE)
```



# Summarize by Time

`summarise_by_time()` aggregates by a period. It's great for:

- Period Aggregation - `SUM()`
- Period Smoothing - `AVERAGE()`, `FIRST()`, `LAST()`

## Period Summarization

Objective: Get the total trade volume by quarter

- Use `SUM()`
- Aggregate using `.by = "quarter"`

```{r}
FANG %>%
  group_by(symbol) %>%
  summarise_by_time(
    date, .by = "quarter",
    volume = SUM(volume)
  ) %>%
  plot_time_series(date, volume, .facet_ncol = 2, .interactive = FALSE, .y_intercept = 0)
```


## Period Smoothing

Objective: Get the first value in each month

- We can use `FIRST()` to get the first value, which has the effect of reducing the data (i.e. smoothing). We could use `AVERAGE()` or `MEDIAN()`. 
- Use the summarization by time: `.by = "month"` to aggregate by month. 

```{r}
FANG %>%
  group_by(symbol) %>%
  summarise_by_time(
    date, .by = "month",
    adjusted = FIRST(adjusted)
  ) %>%
  plot_time_series(date, adjusted, .facet_ncol = 2, .interactive = FALSE)
```



# Filter By Time

Used to quickly filter a continuous time range. 

## Time Range Filtering

Objective: Get the adjusted stock prices in the 3rd quarter of 2013. 

- `.start_date = "2013-09"`: Converts to "2013-09-01
- `.end_date = "2013"`: Converts to "2013-12-31
- A more advanced example of filtering using `%+time`  and `%-time` is shown in _"Padding Data: Low to High Frequency"_. 

```{r}
FANG %>%
  group_by(symbol) %>%
  filter_by_time(date, "2013-09", "2013") %>%
  plot_time_series(date, adjusted, .facet_ncol = 2, .interactive = FALSE)
```

# Padding Data

Used to fill in (pad) gaps and to go from from low frequency to high frequency. This function uses the awesome `padr` library for filling and expanding timestamps. 

## Fill in Gaps

Objective: Make an irregular series regular. 

- We will leave padded values as `NA`. 
- We can add a value using `.pad_value` or we can impute using a function like `ts_impute_vec()` (shown next). 

```{r, message=TRUE}
FANG %>%
  group_by(symbol) %>%
  pad_by_time(date, .by = "auto") # Guesses .by = "day"
```

## Low to High Frequency

Objective: Go from Daily to Hourly timestamp intervals for 1 month from the start date. Impute the missing values.

- `.by = "hour"` pads from daily to hourly
- Imputation of hourly data is accomplished with `ts_impute_vec()`, which performs linear interpolation when `period = 1`.
- Filtering is accomplished using:  
    - "start": A special keyword that signals the start of a series
    - `FIRST(date) %+time% "1 month"`: Selecting the first date in the sequence then using a special infix operation, `%+time%`, called "add time". In this case I add "1 month". 

```{r}
FANG %>%
  group_by(symbol) %>%
  pad_by_time(date, .by = "hour") %>%
  mutate_at(vars(open:adjusted), .funs = ts_impute_vec, period = 1) %>%
  filter_by_time(date, "start", FIRST(date) %+time% "1 month") %>%
  plot_time_series(date, adjusted, .facet_ncol = 2, .interactive = FALSE) 
```

# Sliding (Rolling) Calculations

We have a new function, `slidify()` that turns any function into a sliding (rolling) window function. It takes concepts from `tibbletime::rollify()` and it improves them with the R package `slider`.

## Rolling Mean

Objective: Calculate a "centered" simple rolling average with partial window rolling and the start and end windows. 

- `slidify()` turns the `AVERAGE()` function into a rolling average. 

```{r}
# Make the rolling function
roll_avg_30 <- slidify(.f = AVERAGE, .period = 30, .align = "center", .partial = TRUE)

# Apply the rolling function
FANG %>%
  select(symbol, date, adjusted) %>%
  group_by(symbol) %>%
  # Apply Sliding Function
  mutate(rolling_avg_30 = roll_avg_30(adjusted)) %>%
  pivot_longer(cols = c(adjusted, rolling_avg_30)) %>%
  plot_time_series(date, value, .color_var = name,
                   .facet_ncol = 2, .smooth = FALSE, 
                   .interactive = FALSE)
```

For simple rolling calculations (rolling average), we can accomplish this operation faster with `slidify_vec()` - A vectorized rolling function for simple summary rolls (e.g. `mean()`, `sd()`, `sum()`, etc)

```{r}
FANG %>%
  select(symbol, date, adjusted) %>%
  group_by(symbol) %>%
  # Apply roll apply Function
  mutate(rolling_avg_30 = slidify_vec(adjusted,  ~ AVERAGE(.), 
                                      .period = 30, .partial = TRUE))
```

## Rolling Regression

Objective: Calculate a rolling regression.

- This is a complex sliding (rolling) calculation that requires multiple columns to be involved. 
- `slidify()` is built for this.
- Use the multi-variable `purrr` `..1`, `..2`, `..3`, etc notation to setup a function

```{r}
# Rolling regressions are easy to implement using `.unlist = FALSE`
lm_roll <- slidify(~ lm(..1 ~ ..2 + ..3), .period = 90, 
                   .unlist = FALSE, .align = "right")


FANG %>%
  select(symbol, date, adjusted, volume) %>%
  group_by(symbol) %>%
  mutate(numeric_date = as.numeric(date)) %>%
  # Apply rolling regression
  mutate(rolling_lm = lm_roll(adjusted, volume, numeric_date)) %>%
  filter(!is.na(rolling_lm))
```

# Learning More

<p>
	<iframe width="100%" height="450" src="https://www.youtube.com/embed/elQb4VzRINg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" style="box-shadow: 0 0 5px 2px rgba(0, 0, 0, .5);"><span id="selection-marker-1" class="redactor-selection-marker"></span><span id="selection-marker-1" class="redactor-selection-marker"></span><span id="selection-marker-1" class="redactor-selection-marker"></span><span id="selection-marker-1" class="redactor-selection-marker"></span>
	</iframe>
</p>

_My Talk on High-Performance Time Series Forecasting_

Time series is changing. __Businesses now need 10,000+ time series forecasts every day.__ 

 __High-Performance Forecasting Systems will save companies MILLIONS of dollars.__ Imagine what will happen to your career if you can provide your organization a "High-Performance Time Series Forecasting System" (HPTSF System).

I teach how to build a HPTFS System in my __High-Performance Time Series Forecasting Course__. If interested in learning Scalable High-Performance Forecasting Strategies then [take my course](https://university.business-science.io/p/ds4b-203-r-high-performance-time-series-forecasting). You will learn:

- Time Series Machine Learning (cutting-edge) with `Modeltime` - 30+ Models (Prophet, ARIMA, XGBoost, Random Forest, & many more)
- NEW - Deep Learning with `GluonTS` (Competition Winners)
- Time Series Preprocessing, Noise Reduction, & Anomaly Detection
- Feature engineering using lagged variables & external regressors
- Hyperparameter Tuning
- Time series cross-validation
- Ensembling Multiple Machine Learning & Univariate Modeling Techniques (Competition Winner)
- Scalable Forecasting - Forecast 1000+ time series in parallel
- and more.

<p class="text-center" style="font-size:30px;">
<a href="https://university.business-science.io/p/ds4b-203-r-high-performance-time-series-forecasting">Unlock the High-Performance Time Series Forecasting Course</a>
</p>
