options(cansim.suppress_repair_warnings=TRUE) (#169)StatCan being unavailable no longer aborts with an error. Timeouts, connection failures and error
responses are now reported with a loud warning and the function returns NULL, so that a script or a
document can decide for itself what to do when the servers are down. This applies to every function
that talks to StatCan, and it also covers the two calls that previously bypassed the retry helper,
get_cansim_table_last_release_date() and get_cansim_series_info_cube_coord(). Set
options(cansim.error_on_unavailable=TRUE) to get the previous behaviour of raising an error
examples that make a single lightweight API call are now \donttest{} rather than \dontrun{}, so
they are checked rather than merely displayed. Examples that download a full table or the cube list
stay \dontrun{} because of their run time, and cansim_old_to_new() needs no network at all so its
example now always runs
data retrieved by vector or by table/coordinate now carries UOM and UOM_ID columns, taken from the
cube metadata. StatCan flags a single dimension of each cube as carrying the unit of measure and the unit
varies by member of that dimension, so the unit is resolved per coordinate. Tables that have no unit of
measure, for example census tables, get no unit columns, matching the full table download (#170).
With the unit known, percentage values retrieved by vector or by coordinate are now normalized the
same way as full table downloads: val_norm carries the value divided by 100 and the unit of
measure is relabelled to Rate (Taux in French). Previously the same series normalized
differently depending on whether it was retrieved as a full table or by vector, so scripts that
fetch percentage vectors will see val_norm change by a factor of 100
non-breaking spaces and control characters in names returned by StatCan are now replaced with regular
spaces. These characters render as an ordinary space or as nothing at all, so a column whose name
contained one could not be reached by typing or copy-pasting what the console displayed. The repair
covers table downloads, vector and coordinate calls, cube metadata, table templates and the cube list,
and emits a warning that shows the offending characters by code point, for example
Performance<U+00A0> strategy, together with a count of how many names were repaired. Set
options(cansim.suppress_repair_warnings=TRUE) to silence the warning. The warning also says that
the characters are in the data StatCan publishes rather than anything the user did, and that it
will disappear on its own once StatCan stops sending them, pointing at the issue where that is
tracked. Column names of tables cached before this release keep the original characters
until the table is downloaded again, get_cansim_connection() warns when it finds such a cache (#169)
the same repair now also covers the member labels in the data itself, not just the names of the
columns holding them. These characters turn out to be more common in member labels than in dimension
names, 53 of 500 sampled tables carry at least one. Repairing only the metadata side would have left
the labels in the data unable to match their own factor levels, so every row carrying an affected
label would have become NA. Labels are now also identical whichever way the data is retrieved, so
a table can be joined to template, vector or coordinate data on its dimension columns (#169)
an internal scan_statcan_character_problems() reads the cube metadata straight off the API,
without the repair applied, and reports every title, dimension name and member name StatCan
publishes with a non-breaking space or a control character in it, by table, level and language.
summarize_statcan_character_problems() aggregates that by survey. Neither is exported, they
exist to track whether the upstream problem is shrinking, and both go away along with the repair
once it has (#169)
cached tables now record the package version they were parsed under alongside the download
timestamp, in a single .Rda_info file that replaces the .Rda_time file the timestamp used to
have to itself. The timestamp says whether StatCan has newer data, the version says whether this
release still reads those files the same way. list_cansim_cached_tables() reports it in a new
cansimVersion column, empty for anything cached before this release. The old timestamp file is
still read, so existing caches keep their download date, and is replaced when a table is
refreshed. get_cansim_connection() uses the version to check whether a cache predates the repair
of non-breaking spaces and control characters, and if so reads the metadata cached alongside the
table to see whether its dimension names or member labels actually carry any. Only then does it
warn, naming the offending label and pointing at refresh=TRUE (#169)
every call that sends StatCan a list of vectors, coordinates or tables is now split into batches of
at most 300 items. StatCan refuses a longer list outright with an HTTP 416, which had gone unnoticed
because most of these calls already batched. get_cansim_vector_info() and the cube metadata
download did not, so asking either for more than 300 items at a time failed rather than returning
data
a vector or coordinate StatCan cannot answer for is no longer passed off as data. StatCan signals a
bad item two different ways depending on the method, either marking the record FAILED or answering
SUCCESS and putting the reason in responseStatusCode, and the package only checked the first.
That let an invalid vector through get_cansim_vector_info() as a row of NAs indistinguishable
from real metadata. Both are now checked everywhere, and the items that carry no data are dropped and
reported by reason, naming the vectors or coordinates concerned
vector calls that come back with nothing now warn and return an empty table. Previously the empty
answer travelled on to the metadata join and surfaced there as Column 'cansimTableNumber' doesn't exist, which said nothing about what had happened. The warning names all three things that produce
it: vectors that do not exist, vectors with no data in the requested time frame, and the daily window
from midnight to 8:30am Eastern in which StatCan does not serve vector data
when StatCan refuses a request it explains why in the response body, and that explanation is now shown alongside the status code instead of being discarded. An HTTP 409 says whether the product is simply not released yet, and an HTTP 416 names the limit the request went past. Those two status codes also got the plain-language translation the other codes already had. The HTTP 504 message now says that StatCan builds a whole response before sending any of it, so the way past a gateway timeout is to ask for less at once rather than to retry the same request
three new functions expose StatCan's changed series methods, which report what changed at a finer
grain than get_cansim_changed_tables() does. get_cansim_changed_series_list() lists the series
StatCan changed today as vectors, with the table and coordinate each belongs to, and
get_cansim_changed_series_data_for_vectors() and
get_cansim_changed_series_data_for_coordinates() retrieve the changed data points themselves, in
the same shape and with the same metadata as the corresponding get_cansim_vector() and
coordinate calls. Series that did not change simply contribute no rows, and if none of the ones
asked about changed the answer is an empty table rather than an error. Like the other vector
methods the two data ones batch requests of more than 300 items.
How long the list method takes is entirely a matter of how much StatCan released that morning. It
takes no parameters, so a busy day cannot be asked about in smaller pieces, and on one heavy enough
the request has been seen to outlive StatCan's own gateway and come back as an HTTP 504 after some
nine minutes of silence. Its timeout defaults high to let that answer arrive as StatCan's own
rather than as a vaguer local abort, but the limit is at StatCan's end and raising it further will
not help; get_cansim_changed_tables() is the question to ask on such a day
the package now talks to StatCan through httr2 rather than httr. Requests that fail on a status
StatCan recovers from within seconds, an HTTP 429, 500, 502 or 504, are now retried with
exponential backoff and jitter and honour a Retry-After header, where the previous retries went
out back to back and stood a good chance of arriving while the server was still busy. Requests are
also throttled to the 25 per second StatCan documents as its per-IP limit, so a script asking for
many tables or vectors no longer risks being turned away for asking too quickly, and they now
identify themselves with a cansim/<version> user agent. Statuses that will not improve on a retry
are deliberately not retried: an HTTP 416 carries more items than StatCan accepts however often it
is sent, an HTTP 409 is the nightly update window, and an HTTP 503 is StatCan being down for
maintenance or an outage, which lasts far longer than any retry budget worth spending. The 503 case
says so, and says to try again later, rather than appearing to hang while retries run down. Users
behind a proxy should note that httr2 reads the standard http_proxy and https_proxy
environment variables instead of taking an httr::set_config() call
the timeout argument now bounds how long StatCan may go without sending anything, rather than how
long the whole transfer may take. As a cap on the total it could not tell a connection StatCan had
stopped answering on from a large table that was simply taking a while, and cut both off alike, so a
slow download could fail after two hundred seconds with most of the data already in hand. A transfer
that keeps delivering is now left alone however long it runs, and one that goes quiet for timeout
seconds is dropped, which is the distinction the argument was always described as making. Note that
StatCan works out a whole response before sending any of it, taking roughly a tenth of a second per
vector, so a request for a full batch of 300 is silent for something like thirty five seconds before
the first byte arrives; the default of two hundred seconds leaves ample room for that, but a much
smaller value passed by hand will cut off large requests. Establishing the connection is bounded
separately and briefly, so an unreachable host now fails in ten seconds instead of waiting out the
full timeout
get_cansim() and get_cansim_connection() now ask StatCan where a table lives instead of
assembling the download address from the table number. The address the package built was a guess at
a layout StatCan is free to change, and the extra call that replaces it is small next to the table
download it precedes
get_cansim_sqlite(), list_cansim_sqlite_cached_tables() and remove_cansim_sqlite_cached_table() are now
also documented as deprecated, matching the deprecation warnings they already emit. Use
get_cansim_connection(..., format="sqlite"), list_cansim_cached_tables() and
remove_cansim_cached_tables(..., format="sqlite") insteaddisconnect_cansim_sqlite() is deprecated in favour of the new disconnect_cansim_connection(),
which does the same thing under a name that does not claim a format. It closes a sqlite connection
and leaves parquet and feather connections alone, so a connection can be closed without knowing
which format it came from. It was the last function still named for sqlite that was not itself
deprecated, and its own example demonstrated the deprecated get_cansim_sqlite()get_cansim_sqlite(), list_cansim_sqlite_cached_tables(), remove_cansim_sqlite_cached_table(),
disconnect_cansim_sqlite(), list_cansim_tables() and search_cansim_tables() are scheduled for
removal in a future releaseget_cansim_vector_for_latest_periods() or get_cansim_data_for_table_coord_periods() for
all periods no longer sends an arbitrary round number as the period count. StatCan takes latestN
as a signed 32-bit integer, rejecting zero or less and anything past 2147483647, and quietly clamps
a count longer than the series to the whole series, so the default is now that bound as the
API itself enforces it, rather than a guess that happened to exceed the longest series. A period count larger than
the bound, or an infinite one, is capped instead of being silently coerced to NA and sent to
StatCan as "latestN":NA, and a count below one now fails immediately with a message instead of
earning an HTTP 406. This also applies to the per-coordinate periods column of a table templatelanguage argument is now an error naming what was passed, instead of an NA that
travelled on into a cache directory name or the tail of a StatCan URL and surfaced later as a
download failure or a missing column. Either language can be named in either language, so
"english", "en", "eng" and "anglais" all select English and "french", "fr", "fra" and
"français" all select French, along with their longer and shorter forms; case, surrounding
whitespace and accents are ignored. get_cansim_table_url() and get_cansim_table_notes() now
default to "english" like every other function that takes a language, which selects the same
language their previous "en" default did (#152)if (TRUE) ... else ... in metadata parsing. The else branch held the
readr::read_delim() implementation that utils::read.delim() replaced in February 2025 and had
since fallen behind the live branch, so it was no longer a working fallback (#151)case_when() deprecation warning emitted by dplyr 1.2.0 on every table readget_cansim_changed_tables() passing "days" to difftime() as a time zone instead of a unitget_cansim_changed_tables() now takes both the current date and the cutoff after which the day's
changes are available in Eastern time. It used to compare against 9am, half an hour after StatCan
actually closes its nightly update window, and to take "today" from the local clock, so a machine
set west of Eastern could ask StatCan about a day that had not started there yetget_cansim_connection() no longer fails when the release date of a table cannot be determined, the
staleness check is skipped with a message insteadget_cansim_cube_metadata() and get_cansim_table_template() for vectors of table numbers, metadata for
all tables is still retrieved in a single API call and cached per tableget_cansim_cube_metadata() adds a cansimTableNumber column for the "members", "notes" and "corrections" typesRate, or Taux in French tables, as the
documentation always described; the comparison doing the relabelling could never match before, so the
unit columns used to keep their original Percent... labels after the values had been divided by 100add_cansim_vectors_to_template() now finds vectors for coordinates whose member ids end in a zero,
trimming of trailing .0 positions used to eat into member ids like 10 and the affected rows came
back with an NA vectorcollect_and_normalize(),
an internal language comparison never matched the French settingget_cansim_connection() now falls
back to the previously cached version with a warning instead of returning NULL. Refreshing cube
metadata degrades the same way, so the notes, column, overview and template functions keep working
from previously seen metadata when the servers are downadd_provincial_abbreviations that could lead to mislabelling of provinces in some casesview_cansim_webpage to view vector information from statcan in the browserget_cansim_vector_info()list_cansim_tables and serach_cansim_tables and fallback to corresponding "_cube" methods as Open Data Canada API has changed and similar functionality is available through the "_cube" methods that tie directly into StatCan APISlist_cansim_sqlite_cached_tablesadjust_cansim_values_by_variable functionnormalize_cansim_values into the default table and vector output, in particular always add a scaled variable column called val_norm and an imputed Date column and covert categories to factors by default.get_cansim_sqlite function that stores tables in an SQLite database and facilitates access and management of data.getOption("unzip") is set to enable unzip for files larger than 4GB on unix-like systemsget_cansim_table_notes() functionality