One of the limitations of web browsers is inability to execute general purpose languages like R and Python. On the other hand, the limitation of Python and R (data science languages) is that they need a medium for generating graphics. The increasing number of packages written in JS means data scientists and developers can choose to run some tasks in the browser (clients) and other tasks in R/Python on operating systems, given a client-server architecture (Berson 1996).
The aim of this vignette is to show the way TGVE has been developed
and showcase the difference between this approach and other approaches
where R is used to generate HTML, CSS and/or JavaScript (JS) code to
generate visualizations to run in browsers. Although the front-end
application is written in ReactJS, the tgver
package
bundles a production build of a web application that uses the @tgve/tgvejs
npm package.
Let us do some processing in R and then view the result in the TGVE.
For this purpose we will use these R packages: edgebundle
,
igraph
and ggplot2
first. The following is
adapted from here.
library(igraph)
#>
#> Attaching package: 'igraph'
#> The following objects are masked from 'package:stats':
#>
#> decompose, spectrum
#> The following object is masked from 'package:base':
#>
#> union
library(edgebundle)
library(ggplot2)
library(ggraph)
# this environment variable would return true if run on
# github actions therefore, we can stop running TGVE if so.
# See GitHub docs: https://docs.github.com/en/actions/learn-github-actions/environment-variables
= Sys.getenv("GITHUB_ACTIONS") != ""
is.actions
= us_flights
g = cbind(V(g)$longitude, V(g)$latitude)
xy = data.frame(x = V(g)$longitude, y = V(g)$latitude)
verts
= map_data("state")
states
= edge_bundle_path(g, xy, max_distortion = 12, weight_fac = 2, segments = 50)
pbundle # caching the heavy processing
# saveRDS(pbundle, "vignettes/pbundle.Rds")
# pbundle = readRDS("pbundle.Rds")
# edge list
= get.edgelist(g)
el colnames(el) = c("from", "to")
# get names of airports in verts
= data.frame(x = V(g)$longitude, y = V(g)$latitude, V(g)$name)
verts.with.names colnames(verts.with.names) = c("x", "y", "name")
# intermediate
= verts.with.names[match(el[,"from"], verts.with.names$name), c("x", "y")]
from = verts.with.names[match(el[,"to"], verts.with.names$name), c("x", "y")]
to # create matrix for sf
= cbind(from, to)
m library(sf)
#> Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 6.3.1; sf_use_s2() is TRUE
= lapply(1:nrow(m), function(x) st_linestring(matrix(unlist(m[x,]), ncol = 2, byrow = TRUE)))
sfc = st_sfc(sfc, crs = 4326) sfc
So far, we have done some pre-processing of an igraph
object, created some lines, processed the lines using
edgebundle::edge_bundle_path
(please see the
edgebundle
package for details of the algorithm), and
created an sfc
object.
First, let us have a look at the lines in our sfc
object
as they are using plot
function:
plot(sfc)
Secondly, let us have a look at the bundled paths using
ggplot
:
ggplot() +
geom_path(data = pbundle, aes(x, y, group = group),
col = "orange", size = 0.05) +
geom_path(data = pbundle, aes(x, y, group = group),
col = "white", size = 0.005) +
labs(title = "Edge-Path Bundling") +
::theme_graph(background = "black") +
ggraphtheme(plot.title = element_text(color = "white"))
Using the TGVE we can embed an instance of the TGVE in the output of
a HTML rendered RMarkdown file. Please see the documentations of
knitr
package for details of embedding HTML.
Let us first view the raw data (no bundling), we can build an
instance of the TGVE using tgver::setup
. The package relies
on passing API variables using a URL to the TGVE instance:
library(tgver)
#>
#> Attaching package: 'tgver'
#> The following object is masked from 'package:utils':
#>
#> help
#> The following object is masked from 'package:base':
#>
#> version
= getwd() # for reproducibilty this needs to be a persistent location
p = file.path(p, "tgve")
tp unlink(tp, recursive = TRUE)
# some random line width
# df = data.frame(lw=runif(length(sfc), min=1, max=5))
= function(layerName="line") {
run_tgve setup(p)
= st_as_sf(sfc)
sf = explore_sf(sf, static = TRUE, path = tp)
p ::include_url(file.path(p, "index.html"))
knitr
}
= function(img.url) {
img_or_warning if(!curl::has_internet()) {
warning("Rmd was rendered with no connection!")
else {
} paste0("<img src='", img.url, "' />")
}
}if(!is.actions) {
run_tgve()
else {
} img_or_warning("https://user-images.githubusercontent.com/408568/144712831-7d2aec72-0af4-4ca9-b2e0-2316d7533753.png")
}#> A TGVE instance has been created at: /tmp/RtmpzKYLmF/Rbuildc02973306c430/tgver/vignettes/tgve
#> Attempting to browse TGVE from: /tmp/RtmpzKYLmF/Rbuildc02973306c430/tgver/vignettes/tgve
As of version 1.4.0
release of the npm package, the TGVE
has a simple filter built in for lines. This is designed to only work
with a line
layer where clicking on a line would filter out
all other lines that are not originating from the source of the line
clicked. In future versions, this could be a much richer set of
functionalists for filtering “features” such as paths, boundaries etc.
Indeed, the column filters built into the TGVE could also be improved
and become more sophisticated.
The TGVE basic “click to filter” of lines from an origin-destination data.
Notice, that as the package is hosted on GitHub
and rely
on actions to run the package checks, we have checks to decide whether
this Rmd renders the TGVE instance on your local machine or not. If you
are reading this Rmd on GitHub, chances are you cannot see the TGVE
instance. You may have to open the output of the rendered Rmd on a
browser tab to see the TGVE instances.
Let us see how the bundled data looks on the TGVE, we need to use the
path
layer for now, as the TGVE can still do with more
advanced layer detection in future:
# convert the pbundle into sf
= lapply(1:length(unique(pbundle$group)), function(x) st_linestring(matrix(unlist(pbundle[pbundle$group == x, 1:2]), ncol = 2)))
sfc = st_sfc(sfc, crs = 4326)
sfc # plot(sfc)
= file.path(p, "edge-tgve")
p = file.path(p, "tgve")
tp unlink(tp, recursive = TRUE)
if(!is.actions) {
run_tgve(layerName = "path")
else {
} img_or_warning("https://user-images.githubusercontent.com/408568/144713110-245cf94e-826d-4525-bc48-ac7c97220c71.png")
}#> A TGVE instance has been created at: /tmp/RtmpzKYLmF/Rbuildc02973306c430/tgver/vignettes/edge-tgve/tgve
#> Attempting to browse TGVE from: /tmp/RtmpzKYLmF/Rbuildc02973306c430/tgver/vignettes/edge-tgve/tgve
Notice again, if you are reading this vignette on GitHub, you will not see the interactive result but a screenshot of the final result. Also, if you like to clean up after running this Rmarkdown document, then:
unlink(file.path(getwd(), "tgve"), recursive = TRUE)
unlink(file.path(getwd(), "edge-tgve"), recursive = TRUE)
The CDRC (Consumer Data Research Centre) provides “unique insight in to a diverse range of societal and economic challenges in collaboration with a wide range of consumer data providers”(CDRC 2021). On the website researchers can find consumer datasets. One of these is IMD historical data for UK’s Lower Super Output Areas (LSOA). The LSOA geography is a “large” boundary data and you can find out more about it and download different datasets here or elsewhere.
A typical scenario for a researcher, including senior data scientists
is to learn about how geographic data is processed in data science
languages such as R and Python. Let us imagine our researcher is an R
expert but has not been exposed to geographic data. They need to first
learn about OGC standards such as Simple Features. They will then need
to learn about packages like sf
and underlying system
requirements without mentioning examples.
What the TGVE does for our researcher in question is, whilst they are learning about these data structures and how they are processed in R, they can start exploring them using the TGVE. Provided they can find the right geography for the dataset in question, and having made sure that there is a matching geography column name between the dataset and the geography data file, the TGVE can combine the two.
We needs to download an IMD dataset such as this one. They would then need to be aware that this is only greater London with some ~5k LSOA areas. Once they can filter the large LSOA file from source down to those in the CSV file as follows:
= file.path(tempdir(), "cdrc")
dir dir.create(dir)
# download the csv data which is behind authorization from cdrc
# https://data.cdrc.ac.uk/system/files/c1_english_imd_2019_rebased_for_london.csv
# via https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imd
# renamed data.csv
= read.csv(file.path(dir, "data.csv"))
csv
# download the geography (a .geojson file for our purposes using sf)
# from here https://github.com/gausie/LSOA-2011-GeoJSON
library(sf)
= st_read("https://raw.githubusercontent.com/gausie/LSOA-2011-GeoJSON/master/lsoa.geojson")
geo
# filter it to those in the CSV file
= match(csv$ls11cd, geo$LSOA11CD)
m # notice the different column names for the same geography codes
= geo[m, ]
geo # rename column as TGVE package is young
colnames(geo)[1] = "ls11cd" # same as column one in CSV
# make sure both files are in the ~/Downloads/cdrc directory
# named lsoa.geojson
st_write(geo, file.path(dir, "lsoa.geojson"))
# Finally, let TGVE serve it!
::explore_dir(dir) tgver
Now, the package (tgver
) has a function called
explore_dir
which searches through at a directory for two
files: a CSV and a GeoJSON file. It reads them, generates the API using
the plumber
package behind the scenes and serves them to
the front TGVE which puts them together.
The result will be like the following image.
The TGVE accepts data source URLs to be passed as URL parameters. This means, data scientists can do their processing in R, once they have uploaded (or are able to serve locally) data, they can pass such sources to the TGVE.
Check the following instance being customized with data served from a GitHub repository:
::include_url(
knitrpaste0("https://tgve.github.io/app/?",
"defaultURL=https://raw.githubusercontent.com",
"/tgve/example-data/main/casualties_100.geojson&layerName=heatmap")
)
We can, of course, pass the data (bundled or not) as a json (geojson)
using the data
variable of the TGVE. This is how you could
do this in this code sample:
# same URL as the instance above
::include_url(
knitrpaste0(
# use the 1data1 parameter
"https://tgve.github.io/app?data=",
# see above
gj
) )
The TGVE is built with a vision that R should do what R does best and
so should JS/browser. Their power lies in what they do best and in their
own environments. At this early releases of the front-end (npm package)
the TGVE can be used embedded in the tgver
package or as a
remote instance. The aim is to bring R + JS together using their latest
releases and libraries to carry out scalable geospatial data processing
and visualization.