\documentclass[article,nojss]{jss}
\usepackage{thumbpdf,lmodern}
%\graphicspath{{Figures/}}

\usepackage{amsmath}

\newcommand{\bzero}{\boldsymbol{0}}
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\ba}{\mathbf{a}}
\newcommand{\bB}{\mathbf{B}}
\newcommand{\bc}{\mathbf{c}}
\newcommand{\bK}{\mathbf{K}}
\newcommand{\bs}{\mathbf{s}}
\newcommand{\bt}{\mathbf{t}}
\newcommand{\bv}{\mathbf{v}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\by}{\mathbf{y}}
\newcommand{\bY}{\mathbf{Y}}
\newcommand{\bu}{\mathbf{u}}
\newcommand{\bU}{\mathbf{U}}
\newcommand{\bX}{\mathbf{X}}
\newcommand{\bPhi}{\boldsymbol{\Phi}}
\newcommand{\bmu}{\boldsymbol{\mu}}
\newcommand{\bkappa}{\boldsymbol{\kappa}}

\author{Devin Francom\\Los Alamos National Laboratory \And
        Bruno Sans\'{o}\\University of California Santa Cruz}
\title{\pkg{BASS}: An \proglang{R} Package for Fitting and Performing Sensitivity Analysis of Bayesian Adaptive Spline Surfaces}

\Plainauthor{Devin Francom, Bruno Sans\'{o}}
\Plaintitle{BASS: An R Package for Sensitivity Analysis and Fitting of Bayesian Adaptive Spline Surfaces} %% without formatting
\Shorttitle{\pkg{BASS}: Bayesian Adaptive Spline Surfaces} %% a short title (if necessary)

\Abstract{ We present the \proglang{R} package \pkg{BASS} as a tool
  for nonparametric regression.  The primary focus of the package is
  fitting fully Bayesian adaptive spline surface (BASS) models and
  performing global sensitivity analyses of these models.  The BASS
  framework is similar to that of Bayesian multivariate adaptive
  regression splines (BMARS) from \cite{denison1998bayesian}, but with
  many added features.  The software is built to efficiently handle
  significant amounts of data with many continuous or categorical
  predictors and with functional response.  Under our Bayesian
  framework, most priors are automatic but these can be modified by
  the user to focus on parsimony and the avoidance of overfitting.  If
  directed to do so, the software uses parallel tempering to improve
  the reversible jump Markov chain Monte Carlo (RJMCMC) methods used
  to perform inference.  We discuss the implementation of these
  features and present the performance of \pkg{BASS} in a number of
  analyses of simulated and real data.  }

\Keywords{splines, functional data analysis, sensitivity analysis, nonparametric regression}

%% \Volume{50}
%% \Issue{9}
%% \Month{June}
%% \Year{2012}
%% \Submitdate{2012-06-04}
%% \Acceptdate{2012-06-04}

\Address{
  Devin Francom\\
  Statistical Sciences Group (CCS-6)\\
  Los Alamos National Laboratory\\
  Los Alamos, NM 87545\\
  E-mail: \email{dfrancom@lanl.gov}\\
  LA-UR-18-21548
}

%\VignetteIndexEntry{BASS_vignette}
%\VignetteEngine{knitr::knitr}


\begin{document}
<<setup, echo=F,include=FALSE, cache=FALSE>>=
library(knitr)
# set global chunk options
opts_chunk$set(prompt=TRUE, fig.pos = 't!', out.extra='trim = 0 15 0 50, clip')
options(replace.assign=TRUE, width=77, prompt="R> ")
opts_chunk$set(fig.width=6, fig.height=6, fig.align='center', out.width='.4\\linewidth')#, tidy=T, tidy.opts=list(width.cutoff=60))
knitr::render_sweave()
@


\section{Introduction}

The purpose of the \proglang{R} \citep{R} package \pkg{BASS} \citep{BASS} is to provide an easy-to-use implementation of Bayesian adaptive spline models for nonparametric regression.  It provides a combination of flexibility, scalability, interpretability and probabilistic accuracy that can be difficult to find in other nonparametric regression software packages.  The model form is flexible enough to capture local features that may be present in the data.  It is scalable to moderately large datasets in both the number of predictors and the number of observations.  It performs automatic variable selection.  It can build nonparametric functional regression models and incorporate categorical predictors.  The package can partition the variability of a resultant model using a nonlinear analysis of variance (ANOVA) decomposition, providing valuable interpretation to the predictors.  The Bayesian approach allows for model estimates and predictions that can be evaluated probabilistically. The package is  protected under the GNU General Public License, version 3 (GPL-3), and is available from the Comprehensive \proglang{R} Archive Network (CRAN) at \url{https://CRAN.R-project.org/package=BASS}.

The BASS framework builds on multivariate adaptive regression splines (MARS) from \cite{friedman1991multivariate}.  Well-developed software implementations of the MARS model are available in the \proglang{R} packages \pkg{earth} \citep{earth}, \pkg{polspline} \citep{polspline} and \pkg{mda} \citep{mda}.  The Bayesian version of MARS (BMARS) was first developed in \cite{denison1998bayesian}.  A \proglang{MATLAB} \citep{MATLABR2019a} implementation of BMARS is available from the software website accompanying \cite{denison2002bayesian}.

Our implementation is more similar to the BMARS implementation, though with some substantial changes to methodology as described in \cite{francom2018sensitivity} and \cite{francom2018inferring}.  The primary motivation for developing this software was building surrogate models (or emulators) for complex and computationally expensive simulators (or computer models).  In particular, we wanted to build a fast and accurate surrogate model to use for uncertainty quantification in the presence of a large number of simulations and where each simulation had functional output.  Attributing the variance in the response of the surrogate to different combinations of predictors, a practice known as global sensitivity analysis, is a valuable tool for determining which predictors and interactions are important.  One of the major benefits of polynomial spline surrogate models is that sensitivity analysis can be done analytically.  The \pkg{BASS} package has this functionality for scalar and functional response models with both continuous and categorical inputs.

There are a number of other \proglang{R} packages that use splines, such as \pkg{crs} \citep{crs}, \pkg{gss} \citep{gss}, \pkg{mgcv} \citep{mgcv1, mgcv2} and \pkg{R2BayesX} \citep{BayesX1, BayesX2}, the latter two of which include possible Bayesian inference methods.  These packages allow (or require) the user to specify which variables are allowed to interact in what way, as well as which variables are allowed to have nonlinear main effects. The \pkg{crs} package is more similar to the packages that fit MARS models in that it can learn the structure of the model from the data.  These packages report a single best model.  \pkg{BASS} reports an ensemble of models (posterior draws from the model space) that can be used to make probabilistic predictions.  In this way, it is more similar to Bayesian nonparametric regression packages like \pkg{BART} \citep{BART} and \pkg{tgp} \citep{gramacy2010categorical}.

We introduce the package as follows.  In Section~\ref{sec:BASS}, we describe the modeling framework, including our methods for posterior sampling, modeling functional responses, and incorporating categorical inputs.  In Section~\ref{sec:SA}, we describe the sensitivity analysis methods.  Then, in Section~\ref{sec:ex}, we walk through six examples of how to use the package.  These are done with \pkg{knitr} \citep{knitr} in order to be reproducible by the reader.  Finally, in Section~\ref{sec:summary}, we present a summary of the package capabilities.

\section{Bayesian adaptive spline surfaces}
\label{sec:BASS}

The BASS model, like the MARS and BMARS models, uses data to learn a set of data dependent basis functions that are tensor products of polynomial splines. The number of basis functions as well as the knots and variables used in each basis are chosen adaptively.  The BMARS approach uses reversible jump Markov chain Monte Carlo \citep[RJMCMC;][]{green1995reversible} to sample the posterior.  The BASS adaptation of BMARS includes the improvements of \cite{nott2005efficient} for more efficient posterior sampling as well as parallel tempering for better posterior exploration.  BASS also efficiently handles functional responses and allows for categorical variables.  We discuss each of these aspects below.  First, we introduce the BASS model and priors.

Let $y_i$ denote the dependent variable and $\bx_i$ denote a vector of $p$ independent variables, with $i=1,\ldots,n$.  Without loss of generality, let each independent variable be scaled to be between zero and one.  We model $y_i$ as
\begin{align}
y_i &=f(\bx_i) + \epsilon_i,~~\epsilon_i\sim N(0,\sigma^2) \label{equ:f}\\
f(\bx) &= a_0+\sum_{m=1}^{M}a_mB_m(\bx)\\
B_m(\bx) &= \prod_{k=1}^{K_m} g_{km} [s_{km}(x_{v_{km}}-t_{km})]_+^\alpha \label{equ:basis}
\end{align}
where $s_{km}\in \{-1,1\}$ is referred to as a sign, $t_{km}\in [0,1]$ is a knot, $v_{km}$ selects a variable, $K_m$ is the degree of interaction and $g_{km}=[(s_{km}+1)/2-s_{km}t_{km}]^\alpha$ is a constant that makes the basis function have a maximum of one.  The function $[\cdot]_+$ is defined as $\text{max}(0,\cdot)$.  The power $\alpha$ determines the degree of the polynomial splines.  We allow for no repeats in $v_{1m},\dots,v_{K_mm}$, meaning that a variable can be used only once in each basis function.  $M$ is the number of basis functions, and $\ba$ is the $M+1$ vector of basis coefficients (including the intercept).  The only difference between this setup and that of MARS and BMARS is the inclusion of the constant $g_{km}$ in each element of the tensor product.  This normalizes the basis functions so that the basis coefficients $a_1,\dots,a_M$ are on the same scale, making computations more stable.

In the course of fitting this model, we seek to estimate $\btheta=\{\sigma^2,M,\ba,\bK,\bs,\bt,\bv\}$ where $\bK$ is the $M$-vector of interaction degrees, $\bs$ is the vector of signs $\{\{s_{km}\}_{k=1}^{K_m}\}_{m=1}^M$, $\bt$ the vector of knots and $\bv$ the vector of variables used (with $\bt$ and $\bv$ defined similar to $\bs$).  Under the Bayesian formulation, we specify a prior distribution for $\btheta$.

First, consider the priors for $\sigma^2$ and $\ba$. Let $\bB$ be the $n\times (M+1)$ matrix of basis functions (including the intercept).  Then we use Zellner's $g$-prior \citep{liang2008mixtures} for $\ba$ with
\begin{align}
	\ba|\sigma^2,\tau,\bB &\sim N(\bzero,\sigma^2 (\bB^\top\bB)^{-1}/\tau)\\
	\sigma^2 &\sim InvGamma(g_1,g_2)\\
	\tau &\sim Gamma(a_\tau,b_\tau)
\end{align}
with default settings $a_\tau=1/2$ and $b_\tau=2/n$ (shape and rate), which is the Zellner–Siow Cauchy prior, and $g_1=g_2=0$ resulting in the non-informative prior $p(\sigma^2)\propto 1/\sigma^2$.  In practice, the default settings are sufficient for most cases, though it can be helpful to encode actual prior information into the prior for $\sigma^2$.

Now, consider the prior for the number of basis functions, $M$.  We use a Poisson prior for $M$, truncated to be between $0$ and $M_\text{max}$.  We give a Gamma hyperprior to the mean of the Poisson, $\lambda$.  If $c$ is the Poisson mass that has been truncated, i.e., $c=1-\sum_{m=0}^{M_\text{max}} e^{-\lambda}\lambda^m/m!$, then we have
\begin{align}
	p(M|\lambda) &= \frac{e^{-\lambda}\lambda^M}{cM!} 1(M\leq M_\text{max})\\
	\lambda &\sim Gamma(h_1,h_2)
\end{align}
where $1(\cdot)$ is the indicator function and the default settings of $h_1=h_2=10$ (shape and rate) in most cases induce a small number of basis functions.  In practice, these hyperparameters can be key in order to prevent overfitting.  More specifically, we increase $h_2$ (by orders of magnitude in some cases) to bring the prior for $\lambda$ close to zero in an effort to thin out the tails of the Poisson and have fewer basis functions.  We use $M_\text{max}$ to give an upper bound to the computational cost, rather than to prevent overfitting.  This strategy results in better fitting models since setting $M_\text{max}$ too small can result in poor RJMCMC mixing.  Poor mixing of this sort is due to the fact that the primary way to search the model space with our RJMCMC algorithm is through adding a new basis function, deleting a current basis function, or modifying a current basis function.  If we are at the maximum number of basis functions, we can only explore further by modifying the current set of basis functions or deleting basis functions.  However, deleting basis functions can be difficult, because they may all be useful enough that deleting one causes a significant drop in the model likelihood.  On the other hand, if we are allowed to add a few new basis functions, we may be able to traverse the model space to a different posterior mode, at which point we may be able to delete some of the old basis functions.

The priors for $\bK$, $\bs$, $\bt$ and $\bv$ are uniform over a constrained space as described in \cite{francom2018sensitivity}.  The constraint in this prior makes sure basis functions have more than $b$ non-zero values.  Note that a basis function, as shown in Equation~\ref{equ:basis}, is likely to have many zeros in it depending on how close the knot is to the edge of the space.  If a knot is too close to the edge of the space, there might only be a few non-zero values in the basis function.  A basis function with only a few non-zero values corresponds to very local fitting and usually results in edge effects (i.e., extreme variance at the edges of the space). If we calculate the number of non-zero points in basis function $m$ to be $b_m$, this prior requires that $b_m>b$.  This is the BASS equivalent of specifying a minimum number of points in each partition in recursive partitioning.  In addition to specifying $b$, we also specify $K_\text{max}$, the maximum degree of interaction for each basis function.

Table~\ref{table:t1} shows the parameters used in the function \code{bass} that we have discussed thus far, and their corresponding mathematical symbols.

\begin{table}[t!]
\centering
\begin{tabular}{ccccccccccc}
	\hline
	Symbol & $K_\text{max}$ & $b$ & $h_1$ & $h_2$ & $g_1$ & $g_2$ & $\alpha$ & $M_\text{max}$ & $a_\tau$ & $b_\tau$\\
	\hline
	\code{bass} input & \code{maxInt} & \code{npart} & \code{h1} & \code{h2} & \code{g1} & \code{g2} & \code{degree} & \code{maxBasis} & \code{a.tau} & \code{b.tau}\\
	\hline
\end{tabular}
\caption{Translation from mathematical symbols to parameters used in function \code{bass}.}
\label{table:t1}
\end{table}

\subsection{Efficient posterior sampling}
Posterior sampling is complicated by the fact that the model is transdimensional (since $M$ is unknown).  Our RJMCMC scheme allows us to add, delete, or change a basis function consistent with the approach of \cite{nott2005efficient}.  That is, instead of proposing to add a completely random new basis function in a reversible jump step, we use a proposal generating distribution that favors the variables and degrees of interaction already included in the model.  For example, say there were four basis functions already in the model, each with degree of interaction two.  Say the maximum degree of interaction was three.  Then if we were proposing a new basis function we would sample the degree of interaction from $\{1,2,3\}$ with weights proportional to $\{w_1,w_1+4,w_1\}$, thus favoring two-way interactions since we have seen more of them.  If the nominal weight $w_1$ is large compared to the number of basis functions, this distribution looks more uniform.  The value $w_2$ is the equivalent nominal weight for sampling variables to be included in a candidate basis function.  Both $w_1$ and $w_2$ default to five. If there are a large number of unimportant variables in the data, a small value of $w_2$ (relative to $M$) helps to make posterior sampling more efficient by not proposing basis functions that include the unimportant variables.

We extend the framework of \cite{nott2005efficient} to allow for more than two-way interactions.  This ends up being non-trivial, since the RJMCMC acceptance ratio requires us to calculate the probability of sampling the proposed basis function.  The difficulty comes when we try to calculate the probability of sampling the particular variables, as this requires calculating the probability of a weighted sample without replacement (weighted because we do not sample variables uniformly, without replacement because variables cannot be used more than once in the same basis function).  This is equivalent to sampling from the multivariate Wallenius' noncentral hypergeometric distribution.  In earlier versions of \pkg{BASS}, we determined the probability of such a sample using a function from the \proglang{R} package \pkg{BiasedUrn} \citep{BiasedUrn}.  Since the CRAN version of \pkg{BiasedUrn} allows for only 32 possible variables, we included a slightly altered version of the function in \pkg{BASS} to quickly evaluate the approximate density function of the multivariate Wallenius' noncentral hypergeometric distribution.  In the current version of \pkg{BASS}, we have implemented a simplified multivariate Wallenius' noncentral hypergeometric distribution better fit for this specific purpose.

The computation behind posterior sampling becomes much more efficient when we recognize that each RJMCMC iteration only allows slight changes to our set of basis functions.  Thus, quantities like $\bB^\top\bB$, $\bB\ba$ and $\bB^\top\by$ can easily be updated rather than recalculated, as shown in \cite{francom2018sensitivity}.

We perform $N_\text{MCMC}$ RJMCMC iterations and discard the first $N_\text{burn}$, after which every $N_\text{thin}$ iteration is kept.  This results in $(N_\text{MCMC}-N_\text{burn})/N_\text{thin}$ posterior samples.
Table~\ref{table:t2} shows the parameters used in function \code{bass} discussed in this section, as well as their mathematical symbols.

\begin{table}[t!]
	\centering
		\begin{tabular}{cccccc}
			\hline
			Symbol & $w_1$ & $w_2$ & $N_\text{MCMC}$ & $N_\text{burn}$ & $N_\text{thin}$ \\
			\hline
			\code{bass} input & \code{w1} & \code{w2} & \code{nmcmc} & \code{nburn} & \code{thin} \\
			\hline
		\end{tabular}
                \caption{Translation from mathematical symbols to
                  parameters used to specify nominal weights of
                  proposal distributions and number of RJMCMC
                  iterations in function \code{bass}.}
	\label{table:t2}
\end{table}

\subsection{Parallel tempering}
Posterior sampling with RJMCMC is prone to mixing problems (i.e., problems exploring all of the parameter space).  In our case, this is because only slight changes to the basis functions can be made in each iteration.  Thus, once we start sampling from one mode of the posterior, it can be hard to move to another mode if it requires changing many of the basis functions \citep{gramacy2010importance}.

We are able to achieve better mixing by using parallel tempering.  This requires the specification of a temperature ladder, $1=t_1<t_2<\dots<t_T<\infty$.  For each temperature in the temperature ladder, a RJMCMC chain samples the posterior raised to the inverse temperature (i.e., if $\pi(\btheta|\by)$ is the posterior of interest, we sample from $\pi(\btheta|\by)^{1/t_i}$).  The chains at neighboring temperatures are allowed to swap states according to a Metropolis-Hastings acceptance ratio (see \citealt{francom2018sensitivity} and references therein).  Only samples in the lowest temperature chain ($t_1$) are used for inference.  The high temperature chains mix over many posterior modes, allowing diverse models to be propagated to the low temperature chain.  We allow the chains to run without swapping for $N_{st}$ iterations at the beginning of the run to allow them to get close to their stationary distributions.

Specifying a temperature ladder can be difficult.  Temperatures need to be close enough to each other to allow for frequent swaps (with acceptance rates between 20 and 60\%; \citealt{altekar2004parallel}), and the highest temperature ($t_T$) needs to be high enough to be able to explore all the modes.  Future versions of this package may make some attempt at automatically specifying and altering a temperature ladder.  Further, a message passing interface (MPI) approach to handling the multiple chains could result in substantial speedup, and may be implemented in future versions of the package.

Table~\ref{table:t3} shows the translation from parameters used for parallel tempering in function \code{bass} to symbols we have used in this section.

\begin{table}[t!]
\centering
		\begin{tabular}{ccc}
			\hline
			Symbol & $(t_1,\dots,t_T)$ & $N_{st}$ \\
			\hline
			\code{bass} input & \code{temp.ladder} & \code{start.temper} \\
			\hline
		\end{tabular}
	\caption{Translation from mathematical symbols to parameters used for parallel tempering in function \code{bass}.}
	\label{table:t3}
\end{table}

\subsection{Functional response}
The \pkg{BASS} package has two implementations of methods to use for functional response modeling.

\subsubsection{Augmentation approach}
The augmentation approach handles functional responses as though the variable indexing the functional response, like time or location, is one of the independent variables.  When the functional response is output onto the same functional variable grid for all samples, this results in more efficient calculations involving basis functions because of the Khatri-Rao product structure \citep{francom2018sensitivity}.  For example, this software is well suited to fit a model where the data are such that a combination of independent variables results in a time series and the grid of times (say, $r_1,\dots, r_q$) is the same for each combination.

If there are multiple functional variables, we must specify a maximum degree of interaction for them, $K_{\text{max}}^F$.  For instance, if the functional output was a spatiotemporal field (a function of three variables) and we specify a maximum degree of functional interaction of two, we would not allow for interactions between both spatial dimensions and time.  We would specify the grid of spatial locations and time points as a matrix with three columns rather than a vector like we did in the time series example above.  We can also specify a value $b_F$, possibly different from $b$, that indicates the number of non-zero values required in the functional part of basis functions.  When functional responses are included, the values of $b$ and $b_F$ should be relative to the sample size and the size of the functional grid, respectively.

Table~\ref{table:t4} shows parameters necessary to model functional responses with the augmentation approach in function \code{bass}.  The response \code{y} should be specified as a matrix when the response is functional.

\begin{table}[t!]
	\centering
		\begin{tabular}{cccc}
			\hline
			Symbol & $(r_1,\dots,r_q)$ & $K_{\text{max}}^F$ & $b_F$  \\
			\hline
			\code{bass} input & \code{xx.func} & \code{maxInt.func} & \code{npart.func} \\
			\hline
		\end{tabular}
	\caption{Translation from mathematical symbols to parameters used in function \code{bass} when modeling functional data.}
	\label{table:t4}
\end{table}

\subsubsection{Basis expansion approach}
The second approach that the \pkg{BASS} package can use for modeling functional responses requires decomposing the functional response onto a set of basis functions $\bU = [\bu_1,\dots,\bu_{N_u}]$, so that the matrix of functional responses $\bY = \bU \bPhi$, where each column of $\bY$ is a (flattened) functional response, and $\bY$ has $n$ columns.  Hence $\bPhi$ is the $N_u \times n$ matrix of coefficients in the basis decomposition.  Often, we will center and scale the functional responses using the pointwise mean ($\bmu$) and standard deviation ($\bkappa$) of the functional responses before taking the basis decomposition.  For dimension reduction purposes, we often use only a subset of $n_u\leq N_u$ basis functions, denoted $\bU^*$ with corresponding subset coefficients $\bPhi^*$. Our approach to functional response modeling is then to fit independent models for the coefficients in the basis decomposition \citep{francom2018inferring}. If $Y_{ij}$ is the response for the $j^\text{th}$ functional variable setting using inputs $\bx_i$, this approach models $Y_{ij}$ with
\begin{align}
Y_{ij} = \sum_{l=1}^{n_u} U^*_{lj} \phi^*_l(\bx_i) \\
\phi^*_l(\bx_i) \sim \text{univariate BASS model}
\end{align}
where $\phi^*_l(\bx_i)\equiv \phi^*_{li}$ is modeled with the same specification given in Equation~\ref{equ:f}, so that $\phi^*_{li} =f_l(\bx_i) + \epsilon_{li},~~\epsilon_{li}\sim N(0,\sigma^2_l)$. These models are fit independently, and thus can be done in parallel.  In other words, this approach makes $n_u$ independent calls to the \code{bass} function.

This approach can often provide more accurate results than the augmentation approach, and if the number of available threads is greater than or equal to $n_u$, this can be done in the same amount of time it takes to fit a single univariate model.

Table~\ref{table:t5} shows parameters necessary to model functional responses with the general basis function approach in function \code{bassBasis} (these parameters are passed as a list).  Note that the shortcut function \code{bassPCA} calculates a principal component basis given the matrix \code{y} and fits the corresponding models, and thus requires only a subset of these parameters (not passed as a list).

\begin{table}[t!]
	\centering
		\begin{tabular}{ccccccc}
			\hline
			Symbol & $\bX$ & $n_u$ & $\bU^*$ & $\Phi^*$ & $\bmu$ & $\bkappa$  \\
			\hline
			\code{bassBasis} input & \code{xx} & \code{n.pc} & \code{basis} & \code{newy} & \code{y.m} & \code{y.s} \\
			\hline
		\end{tabular}
	\caption{Translation from mathematical symbols to parameters used in function \code{bassBasis} when modeling functional data. A subset of these parameters are used in function \code{bassPCA}.}
	\label{table:t5}
\end{table}

\subsection{Categorical inputs}
We include categorical variables by allowing for basis functions to include indicators for categorical variables being in certain categories.  Our approach is the Bayesian version of \cite{friedman1991estimating} and is described in \cite{francom2018inferring}.  If a set of independent variables is separated into continuous variables $\bx$ and categorical variables $\bc$, then the $m${th} basis function equivalent of Equation~\ref{equ:basis} can be written as
\begin{align}
	B_m(\bx,\bc)&= \prod_{k=1}^{K_m} g_{km} [s_{km}(x_{v_{km}}-t_{km})]_+^\alpha \prod_{l=1}^{K^c_m} 1\left(c_{v_{lm}^c} \in C_{lm}\right),
\end{align}
where $K^c_m$ is the degree of interaction for the categorical predictors, $1(\cdot)$ is the indicator function, $v^c_{lm}$ indexes the categorical variables and $C_{lm}$ is a subset of the categories for variables $c_{v_{lm}^c}$.  We now allow for $K_m$ or $K_m^c$ to be zero, and specify a $K_\text{max}^c$ (\code{maxInt.cat} in function \code{bass}).

The priors we use for the degree of interaction, variables used and categories used are, in combination with the priors we used above, the same constrained uniform ones.  Thus, basis function $(B_m(\bx_1,\bc_1),\dots,B_m(\bx_n,\bc_n))$ is required to have at least $b$ non-zero values.

\section{Sensitivity analysis}
\label{sec:SA}

Global sensitivity analysis for nonlinear models using the Sobol' decomposition \citep{sobol2001global} is well developed, but often requires large numbers of evaluations of the models for Monte Carlo approximation of integrals \citep{saltelli2008global}.  The benefit of polynomial spline models is that Monte Carlo approximation is unnecessary because the integrals can be calculated analytically.

The method decomposes a function $f(\bx)$ into main effects, two-way interactions, and so on, up to $p$-way interactions so that
\begin{align}
	f(\bx) &=f_0 + \sum_{i=1}^pf_i(x_i) + \sum_{i=1}^{p}\sum_{j>i}f_{ij}(x_i,x_j) + \dots + f_{1\cdots p}(x_1,\dots,x_p) \label{equ:sob1}.
\end{align}
Each term in the sum above is constructed so that it is centered at zero and orthogonal to all the other terms.  This can be done by calculating the centered (conditional) expectations
\begin{align}
	f_0 &= \int f(\bx)p(\bx)d\bx\\
	f_i(x_i) &= \int f(\bx)p(\bx)d\bx_{-i}-f_0\\
	f_{ij}(x_i,x_j) &= \int f(\bx)p(\bx)d\bx_{-ij} - f_i(x_i) - f_j(x_j) - f_0
\end{align}
etc., for all the terms in Equation~\ref{equ:sob1}.  It is often assumed that $\bx$ is independent uniform distributed, though the \pkg{BASS} package allows for more complex distributions, still with the assumption of independence.  Since the terms in Equation~\ref{equ:sob1} are orthogonal and zero-centered,
\begin{align}
	\E(f^2(\bx)) &= f_0^2 + \sum_{i=1}^p\E\left(f_i^2(x_i)\right) + \sum_{i=1}^{p}\sum_{j>i}\E\left(f^2_{ij}(x_i,x_j)\right) + \dots + \E\left(f^2_{1\cdots p}(x_1,\dots,x_p)\right).
\end{align}
Using the fact that $\VAR(f(\bx))=\E(f^2(\bx))-f_0^2$ and that $\E(f_{i_1\cdots i_s}(x_{i_1\cdots i_s})) = 0$ for all terms except $f_0$,
\begin{align}
	\VAR(f(\bx)) &= \sum_{i=1}^pVar(f_i(x_i)) + \sum_{i=1}^{p}\sum_{j>i}\VAR(f_{ij}(x_i,x_j)) + \dots + \VAR(f_{1\cdots p}(x_1,\dots,x_p)).
\end{align}
This is a decomposition of the variance of the model into variance due to each main effect, each two-way interaction (after accounting for the associated main effects), etc.  Under independent uniform $p(\bx)$ (as well as some others), all of these integrals are analytical, with solutions given in \cite{francom2018sensitivity}.  Sensitivity indices for main effects and interactions are then defined as proportions of the total variance.  Total sensitivity for a particular variable can then be gauged by adding the main effect and all interactions associated with that variable and comparing to the total sensitivity indices for other variables.

We can obtain this variance decomposition for each posterior sample to get posterior distributions of sensitivity indices.  This can be time consuming, so the \code{sobol} function has an argument \code{mcmc.use} to specify which RJMCMC iterations should be used.  Calculations of the integrals above can be vectorized when basis functions are the same and only basis function coefficients change.  This is the case for many of the RJMCMC iterations, and the \code{sobol} function automatically determines this and accounts for it. (As a side note, this is also the case for the \code{predict} function).

\subsection{Functional response}
There are a few ways to think about sensitivity analysis for models with functional response.  One way is to get the sensitivity indices for the functional variables in the same way we get the sensitivity indices for the rest of the variables.  This results in a total variance decomposition.  Another approach is to obtain functional sensitivity indices, which would tell us how important a variable or interaction is as we change the functional variable.  This can be done by following the procedure just mentioned, but simply not integrating over the functional variable.  Hence, all of the expectations above would be conditional on the functional variable.  These approaches are explored in \cite{francom2018sensitivity} and \cite{francom2018inferring}.

By default, function \code{sobol} gets sensitivity indices for the functional variables the same way it does for the other variables.  Setting \code{func.var = 1} gets the sensitivity indices as functions of the first (possibly only) functional variable (if there are multiple functional variables, this refers to the first column of the matrix \code{xx.func} passed to function \code{bass}).

On the other hand, the \code{sobolBasis} function (used on an object obtained using \code{bassBasis} or \code{bassPCA}) gets functional sensitivity indices by default.  Because of the complexity of these integrals across the models for the different basis function coefficients, this function can often be sped up significantly by performing the integrals in parallel (using the \code{n.cores} parameter).

\subsection{Categorical inputs}
Under our categorical input extension, the necessary expectations to obtain the Sobol' decomposition are still analytical, as described in \cite{francom2018inferring}.  For the categorical variables, we replace the integrals with sums over categories.

\section{Examples}
\label{sec:ex}
We now demonstrate the capabilities of the package on a few examples.  For each example, we start by setting the seed (\code{set.seed(0)}) so that readers can replicate the results. First we load the package
<<>>=
library("BASS")
@
which we use for all the examples.

\subsection{Curve fitting}
<<child='child/ex1-2.Rnw'>>=
@


\subsection{Friedman function}
<<child='child/ex2-2.Rnw'>>=
@

\subsection{Friedman function with a categorical variable}
<<child='child/ex3-2.Rnw'>>=
@

\subsection{Friedman function with functional response}
<<child='child/ex4-2.Rnw'>>=
@

\subsection{Air foil data}
\label{sec:ex5}
<<child='child/ex5-2.Rnw'>>=
@

\subsection{Pollutant spill model}
<<child='child/ex6-2.Rnw'>>=
@

\section{Summary}
\label{sec:summary}
Our proposed BASS framework provides a powerful general tool for nonparametric regression settings.  It can be used for modeling with many continuous and categorical inputs, large sample size and functional response.  It provides posterior sensitivity analyses without integration error.  The MCMC approach to inference, especially using parallel tempering, yields posterior samples that can be used for probabilistic prediction. The \pkg{BASS} package makes these features accessible to users with minimal exposure.  These capabilities have been demonstrated with a set of examples involving different dimensions, categorical variables, functional responses, and large datasets.



%\section{Acknowledgments}
%This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. LLNL-JRNL-xxxxxx.

\bibliography{bibl}


\end{document}
