In this document, we show how to use the scimesh software renderer instead of the default rgl backend to produce publication-ready static images with fsbrain.

fsbrain visualizes data on brain surfaces. By default it uses the rgl package, which provides fast, hardware-accelerated 3D graphics based on OpenGL. As an alternative, fsbrain also supports the scimesh software renderer, a headless, GPU-free C++ renderer that produces identical static images without requiring X11, OpenGL, or a GPU.

This vignette explains why and when using scimesh makes sense, how to switch to it, and demonstrates the most important ways to produce output with it. All code blocks below are shown for you to copy and adapt, but they are not executed during the build of this vignette (to keep it fast and independent of a network connection). If you want to browse a fully rendered version of this document, including all resulting images, have a look at the online notebook with pre-rendered figures.

Why and when to use the scimesh backend

The scimesh backend is a great fit whenever you need static, publication-ready images and cannot or do not want to rely on a GPU, OpenGL, or an X11 display:

Headless environments: HPC clusters, remote servers, and CI runners typically have no display and often no GPU. scimesh renders everything in software, so it works anywhere.
macOS systems with broken X11/XQuartz: Recent macOS versions (e.g., Tahoe and Sonoma) often ship with broken or missing X11 support, which breaks interactive rgl windows. The scimesh backend is not affected, because it never opens a window.
Reproducibility: Because scimesh is a deterministic software renderer, the same data and code produce identical output images across machines, which is convenient for scripts and automated reporting.
Simpler system dependencies: With the scimesh backend, none of the rgl system dependencies (X11, OpenGL/GLU, etc.) are required.

There are a few things scimesh does not do: it cannot show interactive 3D windows that you can rotate with the mouse, it does not support the browser-based vis.rglwidget() WebGL widgets, and it does not produce animated GIFs. These features remain available through the default rgl backend. In practice, most users only need static images for presentations and publications, for which scimesh works perfectly.

How to switch to the scimesh backend

Install the scimesh package (it is in the Suggests field of fsbrain, so it is optional):

install.packages("scimesh");

Then activate the scimesh backend for the current R session with a single option. This affects all static image export via export() and vislayout.from.coloredmeshes():

# Switch to the scimesh software renderer for headless environments:
options(fsbrain.renderer_backend = "scimesh");

# Check the current backend:
get.fsbrain.renderer.backend();

The default output resolution is 1920x1080 pixels. You can change it with the fsbrain.scimesh.output_dims option:

options(fsbrain.scimesh.output_dims = c(1600, 900));

Note that fsbrain.set.default.figsize(), which controls the rgl window size, has no effect on scimesh rendering.

You can switch back to the default rgl backend at any time:

options(fsbrain.renderer_backend = "rgl");

Again, please keep in mind that interactive viewing (e.g., views = "si", "sr", "t4", "t9" passed to a vis* function) always uses rgl, even when the scimesh backend is active. To render static multi-view figures with scimesh, obtain the coloredmeshes with views = NULL and pass them to the export() function, as shown in the rest of this document.

Example data

We will use the example data that comes with fsbrain (downloaded on demand, see ?download_optional_data) and the FreeSurfer fsaverage template subject (see ?download_fsaverage, and note that you have to accept the FreeSurfer license to download it):

library('fsbrain');

download_optional_data();
download_fsaverage(accept_freesurfer_license = TRUE);

sjd = get_optional_data_filepath("subjects_dir");
sj = 'subject1';

Part 1: Different ways to use the export API

The export() function provides a high-level interface to create publication-ready figures from a coloredmeshes instance. A coloredmeshes instance is returned by all live visualization functions in fsbrain and represents the two brain meshes along with per-vertex data, a colormap, and the resulting per-vertex colors. We obtain it by running a vis* function with views = NULL, which suppresses the interactive rendering while still returning the coloredmeshes. We then pass it to export().

A basic export

Standard-space cortical thickness, with a legend above the colorbar:

cm = vis.subject.morph.standard(sjd, sj, 'thickness', fwhm='10', cortex_only=TRUE, views=NULL);
img = export(cm, colorbar_legend='Cortical thickness [mm]', output_img='thickness_t4.png');

Custom view angles

The default layout is the t4 view (four tiles). You can select any subset of the available views via the view_angles parameter (see get.view.angle.names() for all valid view names). Here we show only the two medial views, for an annotation:

cm = vis.subject.annot(sjd, sj, 'aparc', views=NULL);
img = export(cm, view_angles = c("sd_medial_lh", "sd_medial_rh"));

A larger grid of views

The layout of the tiles is determined automatically from the number of requested views: 4 views become a 2x2 grid, 8 views a 2x4 grid, and any other number produces a vertical strip. Here we use all 8 views and clip the tails of the mean curvature data:

cm = vis.subject.morph.native(sjd, sj, 'curv', cortex_only=TRUE, views=NULL, rglactions=list('trans_fun'=clip.data));
img = export(cm, view_angles = get.view.angle.names(angle_set = "t8"), colorbar_legend='Mean curvature [mm^-1]');

Forcing a vertical strip instead of a grid

You can always force a strip layout by setting grid_like = FALSE:

cm = vis.subject.morph.standard(sjd, sj, 'sulc', fwhm='10', cortex_only=TRUE, views=NULL);
img = export(cm, view_angles = get.view.angle.names("t4"), grid_like = FALSE, colorbar_legend='Sulcal depth [mm]');

The colorbar

The position of the colorbar is controlled with the draw_colorbar parameter: it can be "horizontal" (the default), "vertical", or disabled with FALSE. Note that a colorbar is only drawn if the coloredmeshes contain colormap information (this is the case for morphometry-like data, but not for annotations or custom colors):

cm = vis.subject.morph.standard(sjd, sj, 'sulc', fwhm='10', cortex_only=TRUE, views=NULL);
img = export(cm, colorbar_legend='Sulcal depth [mm]', draw_colorbar = 'vertical');

Changing the background color

Use hex color strings (like '#000000') rather than color names (like 'black'), because color names are ambiguous and their interpretation differs between R and ImageMagick:

cm = vis.subject.morph.standard(sjd, sj, 'sulc', fwhm='10', cortex_only=TRUE, views=NULL);
img = export(cm, view_angles = c("sd_medial_lh", "sd_medial_rh"), background_color = '#000000', draw_colorbar = FALSE);

Transparent background

For figures with a transparent background (very convenient for publications), use the transparency_color parameter. It works by rendering on a temporary background color and later mapping that color to transparency in the output image:

cm = vis.subject.morph.standard(sjd, sj, 'sulc', fwhm='10', cortex_only=TRUE, views=NULL);
img = export(cm, view_angles = c("sd_medial_lh", "sd_medial_rh"), transparency_color = '#FFFFFF');

Part 2: Plotting region-based and vertex-based results

When you perform atlas-based analyses, you typically get one result (significance, effect size, etc.) per atlas region. Vertex-wise analyses in standard space, in contrast, produce one result per vertex (e.g., a t-map). Both kinds of results are commonly displayed on the fsaverage template subject.

Results: region-based

The function vis.region.values.on.subject() visualizes one value per atlas region. Here we show an example on the Desikan atlas, using dummy values:

atlas = 'aparc';   # Desikan atlas

# For the left hemisphere, we just assign a subset of the atlas regions.
# The others will get the default value.
lh_region_value_list = list("bankssts"=0.9, "precuneus"=0.7, "postcentral"=0.8, "lingual"=0.6);

# For the right hemisphere, we retrieve the full list of regions for the
# atlas and assign random values to all of them.
atlas_region_names = get.atlas.region.names(atlas, template_subjects_dir = sjd, template_subject = sj);
rh_region_value_list = rnorm(length(atlas_region_names), 0.8, 0.2);
names(rh_region_value_list) = atlas_region_names;

cm = vis.region.values.on.subject(sjd, sj, atlas, lh_region_value_list, rh_region_value_list, views=NULL);
img = export(cm, colorbar_legend='Effect size (dummy data)');

Results: vertex-based

For symmetric data around zero, e.g., t-values, you should use a symmetric colormap. The function vis.symmetric.data.on.subject() does this, and can also draw a gray-scale background computed from the mean curvature of the cortex. Here we use the demo cluster data that ships with fsbrain and render it on fsaverage:

subjects_dir = get_optional_data_filepath("subjects_dir");
subject_id = 'fsaverage';

lh_demo_cluster_file = system.file("extdata", "lh.clusters_fsaverage.mgz", package = "fsbrain", mustWork = TRUE);
rh_demo_cluster_file = system.file("extdata", "rh.clusters_fsaverage.mgz", package = "fsbrain", mustWork = TRUE);

lh_clust = freesurferformats::read.fs.morph(lh_demo_cluster_file);   # a single positive cluster (activation), the other values are 0
rh_clust = freesurferformats::read.fs.morph(rh_demo_cluster_file);   # two negative clusters

cm = vis.symmetric.data.on.subject(subjects_dir, subject_id, lh_clust, rh_clust, bg="curv_light", views=NULL);
img = export(cm, colorbar_legend='t-value (dummy data)');

Part 3: A workflow with manually loaded meshes and data

fsbrain is not limited to data arranged in the standard FreeSurfer directory structure. As long as you have a brain mesh and per-vertex data (from FreeSurfer files, GIFTI, CIFTI, or any other source supported by freesurferformats), you can visualize them. You do not need a subjects_dir at all.

The example below mirrors this workflow: we load the surface and morphometry files manually from arbitrary paths (here, from the fsbrain example data via get_optional_data_filepath(), but they could come from anywhere) using the low-level freesurferformats::read.fs.surface() and freesurferformats::read.fs.morph() functions. We then build coloredmeshes with coloredmesh.from.preloaded.data() and render them with export(). Note that we never call a vis.subject.* function and never rely on automatic subjects_dir detection.

# 1. Paths to the mesh and data files (arbitrary locations, no subjects_dir needed):
lh_surf_file = get_optional_data_filepath(file.path("subjects_dir", "subject1", "surf", "lh.white"));
rh_surf_file = get_optional_data_filepath(file.path("subjects_dir", "subject1", "surf", "rh.white"));
lh_thick_file = get_optional_data_filepath(file.path("subjects_dir", "subject1", "surf", "lh.thickness"));
rh_thick_file = get_optional_data_filepath(file.path("subjects_dir", "subject1", "surf", "rh.thickness"));

# 2. Load the meshes and the per-vertex data:
lh_surf = freesurferformats::read.fs.surface(lh_surf_file);
rh_surf = freesurferformats::read.fs.surface(rh_surf_file);
lh_thick = freesurferformats::read.fs.morph(lh_thick_file);
rh_thick = freesurferformats::read.fs.morph(rh_thick_file);

# 3. Build coloredmeshes from the preloaded data and export a multi-view figure:
cm_lh = coloredmesh.from.preloaded.data(lh_surf, morph_data = lh_thick, hemi = "lh");
cm_rh = coloredmesh.from.preloaded.data(rh_surf, morph_data = rh_thick, hemi = "rh");

img = export(list("lh" = cm_lh, "rh" = cm_rh), colorbar_legend = 'Cortical thickness [mm]', output_img = 'manual_thickness_t4.png');

This is the same pattern that is used to visualize e.g. CIFTI grayordinate data on fs_LR 32k surfaces, which do not live in a FreeSurfer subjects directory at all.

fsbrain with the scimesh rendering backend

Tim Schäfer