---
title: "Harrison River Female Chinook 2011 Data"
author: "Carl James Schwarz"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    number_sections: yes
    toc: yes
vignette: >
  %\VignetteIndexEntry{Harrison River Female Chinook 2011 Data} 
  %\VignetteEngine{knitr::rmarkdown} 
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

# Introduction
The experiment consisted of daily sampling, but condensed to weeks.
Returning adult salmon were tagged and released at the upstream trap. They were recaptured
during carcass surveys

A stratified-Petersen estimator (Darroch, 1961; Plante et al. 1996) is found for the unpooled data and some pooling of the rows/columns.

# Load the SPAS package
```{r libraries}
library(SPAS)
```
This will load the SPAS fitting functions and any associated packages needed by SPAS.

# Import the data.
The data  should be stored as an $s+1$ by $t+1$ (before pooling) matrix where the upper $s \times t$ matrix consists of the 
The $s \times t$ upper left matrix is the number of animals released in row stratum $i$ and recovered in
column stratum $j$. 
Row $s+1$ contains the total number of UNMARKED animals recovered in column stratum $j$.
Column $t+1$ contains the number of animals marked in each row stratum but not recovered in any column stratum.
The $rawdata[s+1, t+1]$ is not used and can be set to 0 or NA.
The sum of the entries in each of the first $s$ rows is then the number of animals marked in each row stratum.
The sum of the entries in each of the first $t$ columns is then the number of animals captured (marked and unmarked) in each column stratum.
The row/column names of the matrix may be set to identify the entries in the output.

Here is the raw data for the Conne River. Notice that very small number of releases and recoveries in week 6. 

```{r loaddata}
harrison.2011.chinook.F.csv <- textConnection("
  4   ,      2   ,      1   ,     1   ,     0   ,     0   ,   130
 12   ,      7   ,     14   ,     1   ,     3   ,     0   ,   330
  7   ,     11   ,     41   ,     9   ,     1   ,     1   ,   790
  1   ,     13   ,     40   ,    12   ,     9   ,     1   ,   667
  0   ,      1   ,      8   ,     8   ,     3   ,     0   ,   309
  0   ,      0   ,      0   ,     0   ,     0   ,     1   ,    65
744   ,   1187   ,   2136   ,   951   ,   608   ,   127   ,     0")

har.data <- as.matrix(read.csv(harrison.2011.chinook.F.csv, header=FALSE))
har.data
```
A total of `r sum(har.data[1,])` fish were tagged and released in week 1. Of these fish, `r har.data[1,1]` were recovered down stream in column stratum 1;
`r har.data[1,2]` were recovered in column stratum 2; and `r har.data[1,7]` were never seen again.
A total of `r har.data[7,1]` UNMARKED fish were recovered in column stratum 1.

You can add row and column names to the matrix which will used when the data are printed.

# Fitting the Stratified-Petersen model
The Stratified-Petersen model is fit to the above data object.
The very sparse last row will
make fitting the model to the original data difficult with a very flat likelihood surface and so we relax the convergence
criteria:
```{r fit1,results="hide"}
mod..1 <- SPAS.fit.model(har.data,
                       model.id="No restrictions",
                       row.pool.in=1:6, col.pool.in=1:6)
```
The `row.pool.in` and `col.pool.in` inform the function on which rows or columns to pool before the analysis proceeds. 
Both parameters use a vector of codes (length $s$ for row pooing and length $t$ for column pooling) where rows/columns
are pooled that share the same value.

For example. `row.pool.in=c(1,2,3,4,5,6)` would imply that no rows are pooled, while
`row.pool.in=c('a','a','a','b','b','b')` would imply that the first three rows and last three rows are pooled.
The entries in the vector can be numeric or character; however, using character entries implies that the final
pooled matrix is displayed in the order of the character entries. I find that using entries such as `'123'` to represent
pooling rows 1, 2, and 3 to be easiest to use.

**The SPAS system only fits models were the number of rows after pooling is less than or equal to the number of columns after pooling.**

## Results from model 1 (no pooling).
The results of the model fit is a LARGE list. But the function `SPAS.print.model` produces a nice report
```{r mod1p}
SPAS.print.model(mod..1)
```

The original data, the data after pooling, estimates and their standard errors are shown.
Here the stratified-Petersen estimate of the total number of smolts passing the at the first sampling station is
`r formatC(round(mod..1$est$real$N), format='f', digits=0, big.mark=',')` with a standard error of 
`r formatC(round(mod..1$se $real$N), format='f', digits=0, big.mark=',')`.


Each entry in the output is available in fitted model object. You will need to explore the
structure
```{r str1}
cat("Names of objects at highest level\n")
names(mod..1)
cat("\n\nNames of estimates (both beta and real)\n")
names(mod..1$est)
cat("\n\nNames of real estimates\n")
names(mod..1$est$real)
```
The `N` entries refer to the  population size; the `N.stratum` entries refer to the individual stratum population sizes;
the `cap` entries refer to the estimated probability of capture in each row stratum;
the `exp.factor` entries refer to `(1-cap)/cap`, or the expansion factor for each row;
the `psi` entries refer to the number of animals tagged but never seen again (the right most column in the input data);
and the `theta` entries refer to the expected number of animals that were tagged in row stratum $i$ and recovered in column stratum $j$ (after pooling).

# Pooling some rows and columns
As noted by Darroch (1961), the stratified-Petersen will fail if the matrix of movements is close to singular. This often happens if two rows are proportional to each other. In this case, there is no unique MLE for the probability of capture in the two rows, and they should be pooled.
A detailed discussion of pooling is found in Schwarz and Taylor (1998).

## Physically pooling first two rows and last two rows
Let us now pool the first two rows and the last two rows, but leave rows 3 and 4 alone.
Similar, let us pool the last two columns.

The code is
```{r fit2,results="hide"}
mod..2 <- SPAS.fit.model(har.data, model.id="Pooling some rows",
                       row.pool.in=c("12","12","3","4","56","56"),
                       col.pool.in=c(1,2,3,4,56,56))
```

Notice how we specify the pooling for rows and columns and the choice of entries for the two corresponding vectors.
I used character entries for the row pooling to ensure that the pooled matrix is sorted properly (see below);
the numeric entries for the columns is ok as is.

The results of the model fit is
```{r mod2p}
SPAS.print.model(mod..2)
```

Here the stratified-Petersen estimate of the total number of smolts passing the at the first sampling station is
`r formatC(round(mod..2$est$real$N), format='f', digits=0, big.mark=',')` with a standard error of 
`r formatC(round(mod..2$est$real$N), format='f', digits=0, big.mark=',')`.
which is a slight reduction
from the unpooled estimates.

## Physical pooling to a single row and complete pooling
You can physically pool to a single row (and multiple columns) or a single row and single column (which is equivalent to the pooled Petersen estimator).
The code and output follow:

```{r mod3, echo=TRUE}
mod..3 <- SPAS.fit.model(har.data, model.id="A single row",
                       row.pool.in=rep(1, nrow(har.data)-1),
                       col.pool.in=c(1,2,3,4,56,56))
SPAS.print.model(mod..3)
```

```{r }
mod..4 <- SPAS.fit.model(har.data, model.id="Pooled Peteren",
                       row.pool.in=rep(1, nrow(har.data)-1),
                       col.pool.in=rep(1, ncol(har.data)-1))
SPAS.print.model(mod..4)
```
Both models have an estimated abundance of
`r formatC(round(mod..3$est$real$N), format='f', digits=0, big.mark=',')` with a standard error of 
`r formatC(round(mod..3$se $real$N), format='f', digits=0, big.mark=',')`.
which is a slight reduction
from the unpooled estimates.

## Logically pooling the first two rows and last two rows.
Logical pooling is specified the same way as physical pooling except you set
the additional argument *raw.physical.pool=FALSE*.

The code and output are:
```{r }
mod..5 <- SPAS.fit.model(har.data, model.id="Logical Pooling some rows",
                       row.pool.in=c("12","12","3","4","56","56"),
                       row.physical.pool=FALSE,
                       col.pool.in=c(1,2,3,4,56,56))
SPAS.print.model(mod..5)
```


## Logical pooling to a single row and complete pooling
You can also logically pool to a single row (and multiple columns) or a single row and single column (which is equivalent to the pooled Petersen estimator).

Logical pooling is specified the same way as physical pooling except you set
the additional argument *raw.physical.pool=FALSE*.

The code and output follow:

```{r }
mod..6 <- SPAS.fit.model(har.data, model.id="A single row - Logical Pool",
                       row.pool.in=rep(1, nrow(har.data)-1), row.physical.pool=FALSE,
                       col.pool.in=c(1,2,3,4,56,56))

SPAS.print.model(mod..6)

mod..7 <- SPAS.fit.model(har.data, model.id="Pooled Peteren - Logical Pool",
                       row.pool.in=rep(1, nrow(har.data)-1), row.physical.pool=FALSE,
                       col.pool.in=rep(1, ncol(har.data)-1))

SPAS.print.model(mod..7)
```
Both models have an estimated abundance of
`r formatC(round(mod..5$est$real$N), format='f', digits=0, big.mark=',')` with a standard error of 
`r formatC(round(mod..5$se $real$N), format='f', digits=0, big.mark=',')`.
which is a slight reduction
from the unpooled estimates.  

Note that the estimates from physical and logical pooling are the same.


## Logically pooling every successive pair of rows
Looking at the results from the first 6 models, it appears that rows
3 and 4 are also approximately proportional and should be pooled.

The code and output are:
```{r }
mod..8 <- SPAS.fit.model(har.data, model.id="Logical Pooling pairs rows",
                       row.pool.in=c("12","12","34","34","56","56"),
                       row.physical.pool=FALSE,
                       col.pool.in=c(1,2,3,4,56,56))
SPAS.print.model(mod..8)
```



# Comparing the different estimates:
We extract the results from the models and make a summary report:

```{r echo=TRUE}
model.list <- mget( ls()[grepl("^mod\\.\\.",ls())])
names(model.list)

report <- plyr::ldply(model.list, function(x){
   #browser()
   data.frame(#version=x$version,
              date   = as.Date(x$date),
              model.id         = x$model.info$model.id,
              s.a.pool         =-1+nrow(x$fit.setup$pooldata),
              t.p.pool         =-1+ncol(x$fit.setup$pooldata),
              logL.cond        = x$model.info$logL.cond,
              np               = x$model.info$np,
              AICc             = x$model.info$AICc,
              gof.chisq        = round(x$gof$chisq,1),
              gof.df           = x$gof$chisq.df,
              gof.p            = round(x$gof$chisq.p,3),
              kappa.after.lp   = round(x$kappa.after.lp),
              Nhat             = round(x$est$real$N),
              Nhat.se          = round(x$se $real$N))
  
})
report

```




**Unfortunately, because pooling actually changes the data (internally), it is not possible to
do likelihood ratio testing or to use AICc to compare different physical poolings.**

Different logical poolings can be compared without problems. In this case, models 1, 5, 6 and 8 can be compared because they involve no pooling, or logical pooling of the rows. Model 8 has the (arithmetically) smallest AIC but is not much preferred over
model 6 because the difference in AIC is small.

Both of these model have adequate goodness of fit.
The *kappa.after.lp* for both models is adequate.

In this case, all of methods that lead to a single row give the same estimated population size and standard error and all are equivalent to the Pooled Petersen estimator. 
The Chapman modification to the Petersen estimator is also given in the output and
results are similar.


Different column pooling will give the same fit and so cannot be compared. Refer
to the vignette on why column poolings cannot be compared for more details.


# References
Darroch, J. N. (1961). The two-sample capture-recapture census when tagging and sampling are stratified. Biometrika, 48, 241–260.
https://www.jstor.org/stable/2332748

Plante, N., L.-P Rivest, and G. Tremblay. (1988). Stratified Capture-Recapture Estimation of the Size of a Closed Population. Biometrics 54, 47-60.
https://www.jstor.org/stable/2533994

Schwarz, C. J., & Taylor, C. G. (1998). The use of the stratified-Petersen estimator in fisheries management with an illustration of estimating the number of pink salmon (Oncorhynchus gorbuscha) that return to spawn in the Fraser River. Canadian Journal of Fisheries and Aquatic Sciences, 55, 281–296.
https://doi.org/10.1139/f97-238

