ncdc* functions - response header content
type changed - we had a check for proper content type - that check is
now more general so that any json content type will be okay (#390)ghcnd() now accepts more than 1 station identifier
(#373) PR from @eliocampersst(): use new v5 version of their service - see
?ersst docs for details (#381) thanks @vonStadarhraun for
the tipbuoy() docs to state that a special value of
9999 passsed to the year parameter will give
the most up to date data (aka current data) - and an example added using
it (#377)ncdc() with add_units = TRUE
(#378) (#379) PR from @amcdavidghcnd() handling of unknown/bad/invalid station
identifiers: now returns an empty data.frame and gives back empty
strings for the two attributes source and
file_modified (#374)storm_data(),
storm_meta(), storm_shp(), and
storm_shp_read(). associated package datasets
storm_columns and storm_names removed
(#306)lcd() returns a tibble now instead of a tibble with S3
class lcd attached (#369)stormevents_cachesf from Suggests, only used in an examplearc2() fix: when using bbox parameter, it
would not have worked as intended, fixed now (#372)argo functions that were not working because of a
down API are working again, see ?argo (#358)?rnoaa-defunct manual file.
gefs functions are still in the package as those functions
may come back at some point. (#359)arc2(): 1) now accepts more than 1 date;
2) gains new parameter box to accept a bounding box to
spatially filter results (uses dplyr::filter on the
data.frame of spatial data) (#351)Regarding the documentation site at https://docs.ropensci.org/rnoaa
ghncd() (and all functions that build on
ghcnd()) can now be altered to use a specific base URL for
requests. See the “Base URL” section of the ?ghcnd docs
(#353)ghcnd_splitvars() speedup, using
data.table instead of dplyr for manipulation (#352)
(#355)tibble::as_tibble throughout package instead of
dplyr::tbl_df (#354)ghcnd_stations(): internal method
get_inventory() was not creating a directory first before
trying to download a file into that directory (#349) (#350)rnoaa_options() to toggle package level
options; only option for now is cache_messages, a boolean
to toggle whether the user gets messages about cached files or not.
along with this change, messages about cached files and file sizes and
locations are now consistently used across all functions that cache
files on disk (#331)?rnoaa_caching - with information on
how to access and manage cached files for each of the rnoaa functions
that caches files on disk (#346)isd() moved to using hoardr caching - see
?isd_cache for details (#347)path parameter and telling them it’s no longer used;
been defunct for quite a whileghcnd_stations() now caching data - first time requests
should now take just over 1 minute, with subsequent requests (assuming
cached data isn’t deleted) taking ~ 3 seconds (#164)autoplot method meteo_coverage() fix to
visually display gaps in data (#314) (#333) thanks @philipshirkmeteo_nearby_stations() fix: coerce input data.frame to
the function to a data.frame before remainder of steps - in case user
inputs a tibble (#340)coops_search() fix: when
product=predictions, we get no metadata back - so just dont
adjust times (#342)lcd() changes: gains lcd_cache for
managing cached files; use a new internal function for safely reading
each csv file, with more informative error messages; (#344)meteo_pull_monitors() fix: changed internals of
meteo_tidy_ghcnd() to set -9999 values to NA slightly
differently to avoi failing (#348)lcd() function was unfortunately pulling data from
https://www.ncei.noaa.gov/data/global-hourly/access -
whereas it should have been pulling data from
https://www.ncei.noaa.gov/data/local-climatological-data/access
- fixed now; additionaly, lcd_cleanup is defunct because
lcd data coming from the appropriate link has all variable names spelled
out and data split up (#334) thanks @sayon000 !gefs* functions are now defunct - they are being
taken out for now until fixed - see the issues for the details (#335)
(#336)gefs_dimensions and
gefs_ensembles (#327) (#328)gefs function fixes: fixed failing test on CRAN having
to do with a date mismatch; gefs now cleans up temporary
files (#327) (#328)argo_search, argo_files,
argo_qwmo, argo_plan) no longer work, while
the functions that use the FTP server still work
(argo_buoy_files, argo) (#333)ens and time,
that will eventually replace the deprecated parameters
ens_idx and time_idx (#321) (#324)isd() now using fetching data using http instead of
ftptornadoes(): the URL had changed yet again
(#322) (#323) thanks @mbjoseph !sea_ice_tabular() function for fetching tabular
.csv sea ice files instead of using the shp files in
sea_ice() (#194)seaice() fxn name has changed to sea_ice()
(#313)sea_ice() gains option to fetch GeoTIFF format data in
addition to shp files (#219) (#313)lcd_cleanup() - takes output of call
to lcd(), parsing additional columns that contain comma
separated strings (#283)isd() docs to highlight that cached files
downloaded with the fxn will be used until deleted by the user! See
?isd docs for details (#205)gefs only mentioned
longitude, now both vars discussed (#317) (#318)ncdc() regarding units, and in readme
and vignette as well (#265) (#315) from @amoebacpc_prcp(): should have allowed dates back
to 1948, but only allowed back to 1979 (#300)buoy() fxn: datasets that did not have lat/lon
variables were failing to be parsed by the fxn; now when lat/lon vars
missing, we just give. back ncdf4 object for the user to deal with
themselves (#303) (#304)gefs(): longitude on the (-180, 180) scale
worked but not on the (0,360) scale (#316) (#318) (#319)tornadoes(): the URL for the data had changed
(#311) (#312) thanks @mbjosephncdc() parameters
startdate/enddate weren’t handling dates as
input values; now handle date and character inputs (#307)ghcnd_stations(); there was an encoding
issue with the data returned from NOAA (#305)This very long US federal government shutdown has allowed time for building in nicer failure behavior and more documentation for government shutdowns. There’s a number of related changes:
swdi() function (#298)lcd() function (#295)ncdc*() functions
(#293) (#297)swdi(): changed from downloading data with
download.file to crul (#298)arc2() tests to not have hard-coded dates
(#294)arc2 tests to not be sensitive to the real year
that the test is run in, reported in CRAN checks and via email
(#294)bsw() for Blended Sea Winds data
(#246)ghcnd_read() - to read .dly files
directly, e.g., files already downloaded (#223) thanks @shabbychef for the
feature requestlcd() for Local Climatological Data
(#212)se_data() and se_files()
for the Storm Events Database (#282)ghcnd() and ghcnd_search() gain a
refresh parameter to refresh data for the query even if
it’s already cached locally. in addition, these functions now print
messages to tell the user what file path the data is locally cached in,
and the min and max dates when using ghcnd_search() (#269)
thanks @kgmccannncdc() gains add_units parameter (boolean)
to toggle adding units to the output data.frame. default is
add_units=FALSE. if add_units=TRUE we match
dataset id and data type id and return units if we have them. do be in
touch if you see a problem with these units! ncdc() now
returns tibbles in the data slot (#233) (#266) (#289)
(#287)coops docs (#228) thanks @jstaghcnd_search() to arrange data by day instead of
by month (#247) thanks @asrivas3 for reportingswdi() to use xml2 and
crul packages instead of XML and
httr (#275)swdi() tests (#239) thanks @kevin-ht-hoisd_stations_search() no longer renames lat and lon
column names (#238) thanks @kevin-ht-hohttr with crul throughout package
(#186)vcr now for caching, more to do waiting
on vcr being able to handle direct to disk use cases and
binary files like pdfs (#284)gefs(). was incorrectly repeating time values
within ensembles when it should have repeated time values across
ensembles so that each ensemble has a time value for each time period
(#230) (#231) thanks for report from @lcsma and fix by @potterzotncdc() - fix to internal function
parse_ncdc(), which was failing on strsplit()
call if attributes was NULL (#232) thanks for reporting
@andypickecpc_prcp(): URLs were changed at some point,
fixes for this (#242)cpc_prcp(): to read .gz files
correctly with gzfile instead of file (#248)homr() - NOAA server gives back a 200 OK
response even if that’s not the case - but we can check content type to
see if there was likely an error (#250)autoplot.meteo_coverage (#258)buoy_stations(): was getting the wrong station
ids - fixed; and use async HTTP requests to get data faster (#261)
thanks @johnharley
for the bug reportghcnd_stations(): was returning an extra empty
row (#267) thanks @joeroe for the bug reportghcnd() was giving a trailing row of
NA’s - fixed (#270)swdi(): radius parameter doesn’t work,
update docs to tell users not to use it (#243)ncdc() fix: fix to internal function
parse_ncdc(), errored when more than 1 flag returned (#279)
thanks @ghaines3 for
the bug reportstorm_shp() fix: update to use new URL patterns
(#263)swdi() fail better (#274) thanks @OrionDarleymeteo_nearby_stations() to coerce character to
numeric lat and lon values (#257) thanks @mondorescue for the bug reportmeteo_nearby_stations() - internally ignored the
user supplied column names for lat and lon (#286) thanks @ghaines3 for the bug
reporttornadoes() for Windows OS’s:
utils::untar() was failing on windows, changed to using
utils::unzip() (#203)argo_buoy_files(): use fill=TRUE in
the read.table call - was erroring on some Windows OS’s
(#235) thanks @jonmcalder for reportingNote that some NOAA datasets have changed names:
GHCNDMS is now GSOM (Global Summary of the
Month)ANNUAL is now GSOY (Global Summary of the
Year)isd() gains new parameters additional to
toggle whether the non-mandatory ISD fields (additional + remarks) are
parsed and returned & force to toggle whether download
new version or use cached version. isd_read() gains new
parameter additional (see description above) (#190)cpc_prcp() (#193)arc2() to get data from Africa Rainfall
Climatology version 2 (#201)https - changed
internal code to use https from http for
coops, swdi, ersst, and tornadoes data sources (#187)coops_search() to handle requests better: only
certain date combinations allowed for certain COOPS products (#213)
(#214) thanks @tphilippi !hoardr package to manage caching in some
functions. Will roll out to all functions that cache soon (#191)GHCNDMS is now GSOM and
ANNUAL is now GSOY - added to docs and
examples of using GSOM and GSOY (#189)isd() (#168)coops_search() to fix time zone problems
(#184) thanks @drf5nghcnd() - fix some column types that were of
inappropriate type before (#211)ghcnd(): we were coercing factors to integers,
which caused nonsense output - first coercing to character now, then
integer (#221)ncdc() function. Added metadata to the package to help
parse flags (#199)isd() now using a new package isdparser to
parse NOAA ISD files. We still fetch the file within rnoaa,
but the file parsing is done by isdparser (#176) (#177)
(#180) thanks @mrubayet for the pushmeteo_* functions
(#178) thanks @mrubayetghcnd() where internal unexported function
was not found (#179)isd_stations() and
isd_stations_search() to work correctly on Windows (#181)
thanks @GuodongZhuncdc) to https from http (#182)
thanks @maspottshomr) to https from http
(#183)meteo_clear_cache gains parameter force to
control force parameter in unlink()lubridate usage in seaiceurls()
function, just using base R functions.isd_read() to read ISD output from
isd() manually instead of letting isd() read
in the data. This is useful when you use isd() but need to
read the file in later when it’s already cached. (#169)rnoaa cache files that are downloaded
from various NOAA web services. File caching is usually done when data
comes from FTP servers. In some of these functions where we cache data,
we used to write to your home directory, but have now changed all these
functions to write to a proper cache directory in a platform independent
way. We determine the cache directory using
rappdirs::user_cache_dir(). Note that this may change your
workflow if you’d been depending on cached files to be a in particular
place on your file system. In addition, the path parameter
in the changed functions is now defunct, but you get an informative
warning about it (#171)storm_data() now returns a tibble/data.frame not inside
of a list. We used to return a list with a single slot data
with a data.frame, but this was unnecessary.ghcnd_stations() now outputs a data.frame
(tbl_df) by itself, instead of a data.frame nested in a
list. This may change how you access data from this function.
(#163)ncdc_*()) (#167)isd() docs that when you get an error
similar to
Error: download failed for ftp://ftp.ncdc.noaa.gov/pub/data/noaa/1955/011490-99999-1955.gz,
the file does not exist on NOAA’s ftp servers. If your internet is down,
you’ll get a different error saying as much (#170)meteo_*, and are meant to find
weather monitors near locations (meteo_nearby_stations),
find all monitors within a radius of a location
(meteo_distance), calculate the distances between a
location and all available stations
(meteo_process_geographic_data), calculate the distance
between two locations (meteo_spherical_distance), pull
GHCND weather data for multiple weather monitors
(meteo_pull_monitors), create a tidy GHCND dataset from a
single monitor (meteo_tidy_ghcnd), and determine the
“coverage” for a station data frame (meteo_coverage()). In
addition, vis_miss() added to visualize missingness in a
data.frame. See the PR diff against
master for all the changes. (#159) Thanks a ton to @geanders et al.
(@hrbrmstr, @maelle, @jdunic, @njtierney, @leighseverson, @RyanGan, @mandilin, @jferreri, @cpatrizio88, @ryan-hicks, @Ewen2015, @mgutilla, @hakessler, @rodlammers)isd_stations_search() changed internal structure. We
replaced usage of geojsonio and lawn for
faster dplyr::filter for bbox inputs, and
meteo_distance() for lat/long/radius inputs .
This speeds up this function significantly. Thanks to @lukas-rokka
(#157)isd_stations_search() and isd_stations()
now return tibble’s instead of data.frame’sisd_stations_search() now caches using
rappdirs (#161)is() uses to inherits()seaiceeurls() function that’s used to generate
urls for the seaice() function - due to change in NOAA urls
(#160)ghncd_split_vars() to not fail on
dplyr::contains call (#156) thanks @lawinslow !httr version to call encoding explicitly
(#135)gefs functions (#121)isd() function - it’s a
time consuming task as we have to parse a nasty string of characters
line by line - more speed ups to come in future versions (#146)dplyr::rbind_all() with
dplyr::bind_rows() as the former is being deprecated
(#152)isd() function - was failing on some station
names that had leading zeros. (#136)ncdc_stations() - used to allow more than one
station id to be passed in, but internally only handled one. This is a
restriction due to the NOAA NCDC API. Documentation now shows an example
of how to deal with many station ids (#138)ncdc_*() functions to allow
multiple inputs to those parameters where allowed (#139)ncdc_plot() due to new
ggplot2 version (#153)argo() functions: a) with new
httr, box input of a vector no longer works, now manually
make a character vector; b) errant file param being passed into the http
request, removed (#155)argo() (#123) for more, see http://www.argo.ucsd.edu/coops_search() (#111) for idea from @fmichonneau (#124) for implementing
@jsta also (#126)
(#128)rgdal moved to Suggests to make usage easier
(#125)ncdc_plot() - made default brakes to just
default to what ggplot2 does, but you can still pass in
your own breaks (#131)gefs(),
gefs_dimension_values(), gefs_dimensions(),
gefs_latitudes(), gefs_longitudes(), and
gefs_variables() (#106) (#119) thanks @potterzot - he’s now an
author tooersst() (#96)isd_stations() to get ISD station
data.ncdf package for ncdf4 package.
Windows binaries weren’t availiable for ncdf4 prior to now.
(#117)isd() function to do transformations of
certain variables to give back data that makes more sense (#115)leaflet, geojsonio, and lawn
added in Suggests, used in a few functions.swdi() function man page that the
nldn dataset is available to military users only
(#107)buoy() function to accept character class inputs
for the buoyid parameter. the error occurred because
matching was not case-insensitive, now works regardless of case
(#118)ggplot2 version (#113)GET request retries for ghncd
functions as some URLs fail unpredictably (#110)utils, methods, and stats
(#103)?rnoaa-defunct for more information (#104)radius parameter removed from
ncdc_stations() function (#102), was already removed
internally within the function in the last version, now not in the
function definition, see also (#98) and (#99)plyr and data.table from imports.
plyr::rbind.fill() and data.table::rbindlist()
replaced with dplyr::bind_rows().httr v1 where empty
list not allowed to pass to the query parameter in
GET (#101)ghcnd(), ghcnd_clear_cache(),
ghcnd_countries(), ghcnd_search(),
ghcnd_splitvars() ghcnd_states(),
ghcnd_stations(), and ghcnd_version() (#85)
(#86) (#87) (#88) (#94)isd() functions, including better
man file.callopts parameter changed to ... in
function swdi().ncdc() requires that users do their own paging -
previously this was done internally (#77)RCurl,
maptools, stringr, digest. A few
new ones added: dplyr, tidyr.erddap functions now defunct - see the package rerddap, a general
purpose R client for ERDDAP servers. (#51) (#73) (#90) (#95)extent function in noaa_stations()
used to accept either a bounding box or a point defined by lat/long. The
lat/long option dropped as it required two packages, one of which is a
pain to install for many users (#98) (#99)ncdc_legacy(). (#54)isd() to get ISD data from NOAA FTP
server. (#76)erddap_table(), while gridded datasets are available
via erddap_grid(). Helper function
erddap_search() was modified to search for either tabledap
or griddap datasets, and erddap_info() gets and prints
summary information differently for tabledap and griddap datasets.
(#63)erddap_data() defunct, now as functions
erddap_table() and erddap_grid(), uses new
store parameter which takes a function, either
disk(path, overwrite) to store on disk or
memory() to store in R memory.assertthat library removed, replaced with
stopifnot()tornadoes(). (#56)storm_*(). (#57)homr_*() (#59)dplyr-like outputs with a summary of the data.frame, as
appropriate.ncdc_* functions changed
callopts parameter to .... This parameter
allow you to pass in options to httr::GET to modify curl
requests. (#61)check_key() looks for one of two
stored keys, as an environment variable under the name
NOAA_KEY, or an option variable under the name
noaakey. Environment variables can be set during session
like Sys.setenv(VAR = "..."), or stored long term in your
.Renviron file. Option variables can be set during session
like options(var = "..."), or stored long term in your
.Rprofile file.is.* and print.* functions no longer have
public man files, but can be seen via rnoaa::: if
needed.sp, rgeos,
assertthat, jsonlite, and ncdf4,
and new package Suggests: knitr, taxizenoaa*() functions for
NCDC data changed to ncdc*(). noaa_buoy()
changed to buoy(). noaa_seaice() changed to
seaice(). When you call the old versions an error is
thrown, with a message pointing you to the new function name. See
?rnoaa-defunct.erddap_info(), erddap_data(), and
erddap_search().buoy(),
including a number of helper functions.ncdc() now splits apart attributes. Previously, the
attributes were returned as a single column, but now there is column for
each attribute so data can be easily retrieved. Attribute columns differ
for each different datasetid.buoy() function has been removed from the CRAN version
of rnoaa. Install the version with buoy() and
associated functions via
devtools::install_github("ropensci/rnoaa", ref="buoy")noaa_swdi() (function changed to swdi())
gains new parameter filepath to specify path to write a
file to if format=kmz or format=shp. Examples
added for using format= csv, shp, and kmz.plyr::compact.ncdc() gains new parameter
includemetadata. If TRUE, includes metadata, if not, does
not, and response should be faster as does not take time to calculate
metadata.noaa_stations() gains new parameter
radius. If extent is a vector of length 4 (for
a bounding box) then radius is ignored, but if you pass in two points to
extent, it is interpreted as a point, and then
radius is used as the distance upon which to construct a
bounding box. radius default is 10 km.datasetid, startdate, and
enddate are often required parameters, and changes were
made to help users with this.