R Internals ¶

1.1 SEXPs ¶

What R users think of as variables or objects are symbols which are bound to a value. The value can be thought of as either a SEXP (a pointer), or the structure it points to, a SEXPREC (and there are alternative forms used for vectors, namely VECSXP pointing to VECTOR_SEXPREC structures). So the basic building blocks of R objects are often called nodes, meaning SEXPRECs or VECTOR_SEXPRECs.

Note that the internal structure of the SEXPREC is not made available to R Extensions: rather SEXP is an opaque pointer, and the internals can only be accessed by the functions provided.

Both types of node structure have as their first three fields a 64-bit sxpinfo header and then three pointers (to the attributes and the previous and next node in a doubly-linked list), and then some further fields. On a 32-bit platform a node¹ occupies 32 bytes: on a 64-bit platform typically 56 bytes (depending on alignment constraints).

The first five bits of the sxpinfo header specify one of up to 32 SEXPTYPEs.

• SEXPTYPEs
• Rest of header
• The ‘data’
• Allocation classes

#define NAMED_BITS 16
struct sxpinfo_struct {
    SEXPTYPE type      :  5;  /* discussed above */
    unsigned int scalar:  1;  /* is this a numeric vector of length 1?
    unsigned int obj   :  1;  /* is this an object with a class attribute? */
    unsigned int alt   :  1;  /* is this an ALTREP object? */
    unsigned int gp    : 16;  /* general purpose, see below */
    unsigned int mark  :  1;  /* mark object as ‘in use’ in GC */
    unsigned int debug :  1;
    unsigned int trace :  1;
    unsigned int spare :  1;  /* debug once and with reference counting */
    unsigned int gcgen :  1;  /* generation for GC */
    unsigned int gccls :  3;  /* class of node for GC */
    unsigned int named : NAMED_BITS; /* used to control copying */
    unsigned int extra : 32 - NAMED_BITS;
}; /*		    Tot: 64 */

b <- a

dim(a) <- c(7, 2)

a <- `dim<-`(a, c(7, 2))

union {
    struct primsxp_struct primsxp;
    struct symsxp_struct symsxp;
    struct listsxp_struct listsxp;
    struct envsxp_struct envsxp;
    struct closxp_struct closxp;
    struct promsxp_struct promsxp;
} u;

1.2 Environments and variable lookup ¶

What users think of as ‘variables’ are symbols which are bound to objects in ‘environments’. The word ‘environment’ is used ambiguously in R to mean either the frame of an ENVSXP (a pairlist of symbol-value pairs) or an ENVSXP, a frame plus an enclosure.

There are additional places that ‘variables’ can be looked up, called ‘user databases’ in comments in the code. These seem undocumented in the R sources, but apparently refer to the RObjectTable package formerly available at https://www.omegahat.net/RObjectTables/.

The base environment is special. There is an ENVSXP environment with enclosure the empty environment R_EmptyEnv, but the frame of that environment is not used. Rather its bindings are part of the global symbol table, being those symbols in the global symbol table whose values are not R_UnboundValue. When R is started the internal functions are installed (by C code) in the symbol table, with primitive functions having values and .Internal functions having what would be their values in the field accessed by the INTERNAL macro. Then .Platform and .Machine are computed and the base package is loaded into the base environment followed by the system profile.

The frames of environments (and the symbol table) are normally hashed for faster access (including insertion and deletion).

By default R maintains a (hashed) global cache of ‘variables’ (that is symbols and their bindings) which have been found, and this refers only to environments which have been marked to participate, which consists of the global environment (aka the user workspace), the base environment plus environments⁴ which have been attached. When an environment is either attached or detached, the names of its symbols are flushed from the cache. The cache is used whenever searching for variables from the global environment (possibly as part of a recursive search).

• Search paths
• Namespaces
• Hash table

1.3 Attributes ¶

As we have seen, every SEXPREC has a pointer to the attributes of the node (default R_NilValue). The attributes can be accessed/set by the macros/functions ATTRIB and SET_ATTRIB, but such direct access is normally only used to check if the attributes are NULL or to reset them. Otherwise access goes through the functions getAttrib and setAttrib which impose restrictions on the attributes. One thing to watch is that if you copy attributes from one object to another you may (un)set the "class" attribute and so need to copy the object and S4 bits as well. There is a macro/function DUPLICATE_ATTRIB to automate this.

Note that the ‘attributes’ of a CHARSXP are used as part of the management of the CHARSXP cache: of course CHARSXP’s are not user-visible but C-level code might look at their attributes.

The code assumes that the attributes of a node are either R_NilValue or a pairlist of non-zero length (and this is checked by SET_ATTRIB). The attributes are named (via tags on the pairlist). The replacement function attributes<- ensures that "dim" precedes "dimnames" in the pairlist. Attribute "dim" is one of several that is treated specially: the values are checked, and any "names" and "dimnames" attributes are removed. Similarly, you cannot set "dimnames" without having set "dim", and the value assigned must be a list of the correct length and with elements of the correct lengths (and all zero-length elements are replaced by NULL).

The other attributes which are given special treatment are "names", "class", "tsp", "comment" and "row.names". For pairlist-like objects the names are not stored as an attribute but (as symbols) as the tags: however the R interface makes them look like conventional attributes, and for one-dimensional arrays they are stored as the first element of the "dimnames" attribute. The C code ensures that the "tsp" attribute is an REALSXP, the frequency is positive and the implied length agrees with the number of rows of the object being assigned to. Classes and comments are restricted to character vectors, and assigning a zero-length comment or class removes the attribute. Setting or removing a "class" attribute sets the object bit appropriately. Integer row names are converted to and from the internal compact representation.

Care needs to be taken when adding attributes to objects of the types with non-standard copying semantics. There is only one object of type NILSXP, R_NilValue, and that should never have attributes (and this is enforced in installAttrib). For environments, external pointers and weak references, the attributes should be relevant to all uses of the object: it is for example reasonable to have a name for an environment, and also a "path" attribute for those environments populated from R code in a package.

When should attributes be preserved under operations on an object? Becker, Chambers & Wilks (1988, pp. 144–6) give some guidance. Scalar functions (those which operate element-by-element on a vector and whose output is similar to the input) should preserve attributes (except perhaps class, and if they do preserve class they need to preserve the OBJECT and S4 bits). Binary operations normally call copyMostAttrib to copy most attributes from the longer argument (and if they are of the same length from both, preferring the values on the first). Here ‘most’ means all except the names, dim and dimnames which are set appropriately by the code for the operator.

Subsetting (other than by an empty index) generally drops all attributes except names, dim and dimnames which are reset as appropriate. On the other hand, subassignment generally preserves such attributes even if the length is changed. Coercion drops all attributes. For example:

> x <- structure(1:8, names=letters[1:8], comm="a comment")
> x[]
a b c d e f g h
1 2 3 4 5 6 7 8
attr(,"comm")
[1] "a comment"
> x[1:3]
a b c
1 2 3
> x[3] <- 3
> x
a b c d e f g h
1 2 3 4 5 6 7 8
attr(,"comm")
[1] "a comment"
> x[9] <- 9
> x
a b c d e f g h
1 2 3 4 5 6 7 8 9
attr(,"comm")
[1] "a comment"

1.4 Contexts ¶

Contexts are the internal mechanism used to keep track of where a computation has got to (and from where), so that control-flow constructs can work and reasonable information can be produced on error conditions (such as via traceback), and otherwise (the sys.xxx functions).

Execution contexts are a stack of C structs:

typedef struct RCNTXT {
    struct RCNTXT *nextcontext; /* The next context up the chain */
    int callflag;               /* The context ‘type’ */
    JMP_BUF cjmpbuf;            /* C stack and register information */
    int cstacktop;              /* Top of the pointer protection stack */
    int evaldepth;              /* Evaluation depth at inception */
    SEXP promargs;              /* Promises supplied to closure */
    SEXP callfun;               /* The closure called */
    SEXP sysparent;             /* Environment the closure was called from */
    SEXP call;                  /* The call that effected this context */
    SEXP cloenv;                /* The environment */
    SEXP conexit;               /* Interpreted on.exit code */
    void (*cend)(void *);       /* C on.exit thunk */
    void *cenddata;             /* Data for C on.exit thunk */
    char *vmax;                 /* Top of the R_alloc stack */
    int intsusp;                /* Interrupts are suspended */
    SEXP handlerstack;          /* Condition handler stack */
    SEXP restartstack;          /* Stack of available restarts */
    struct RPRSTACK *prstack;   /* Stack of pending promises */
} RCNTXT, *context;

plus additional fields for the byte-code compiler. The ‘types’ are from

enum {
    CTXT_TOPLEVEL = 0,  /* toplevel context */
    CTXT_NEXT     = 1,  /* target for next */
    CTXT_BREAK    = 2,  /* target for break */
    CTXT_LOOP     = 3,  /* break or next target */
    CTXT_FUNCTION = 4,  /* function closure */
    CTXT_CCODE    = 8,  /* other functions that need error cleanup */
    CTXT_RETURN   = 12, /* return() from a closure */
    CTXT_BROWSER  = 16, /* return target on exit from browser */
    CTXT_GENERIC  = 20, /* rather, running an S3 method */
    CTXT_RESTART  = 32, /* a call to restart was made from a closure */
    CTXT_BUILTIN  = 64  /* builtin internal function */
};

where the CTXT_FUNCTION bit is on wherever function closures are involved.

Contexts are created by a call to begincontext and ended by a call to endcontext: code can search up the stack for a particular type of context via findcontext (and jump there) or jump to a specific context via R_JumpToContext. R_ToplevelContext is the ‘idle’ state (normally the command prompt), and R_GlobalContext is the top of the stack.

Note that whilst calls to closures set a context, internal functions never do and primitive builtins only set it when profiling or when they are interfaces to foreign functions.

The byte-code compiler generates a map of instructions to source references and expressions at compile time, which allows to produce information on error conditions. As an optimization, the byte-code interpreter then does not set a context in some cases, such as in simple loops or when inlining simple builtins or wrappers for internal functions.

Dispatching from a S3 generic (via UseMethod or its internal equivalent) or calling NextMethod sets the context type to CTXT_GENERIC. This is used to set the sysparent of the method call to that of the generic, so the method appears to have been called in place of the generic rather than from the generic.

The R sys.frame and sys.call functions work by counting calls to closures (type CTXT_FUNCTION) from either end of the context stack.

Note that the sysparent element of the structure is not the same thing as sys.parent(). Element sysparent is primarily used in managing changes of the function being evaluated, i.e. by Recall and method dispatch.

CTXT_CCODE contexts are currently used in cat(), load(), scan() and write.table() (to close the connection on error), by PROTECT, serialization (to recover from errors, e.g. free buffers) and within the error handling code (to raise the C stack limit and reset some variables).

1.5 Argument evaluation ¶

As we have seen, functions in R come in three types, closures (SEXPTYPE CLOSXP), specials (SPECIALSXP) and builtins (BUILTINSXP). In this section we consider when (and if) the actual arguments of function calls are evaluated. The rules are different for the internal (special/builtin) and R-level functions (closures).

For a call to a closure, the actual and formal arguments are matched and a matched call (another LANGSXP) is constructed. This process first replaces the actual argument list by a list of promises to the values supplied. It then constructs a new environment which contains the names of the formal parameters matched to actual or default values: all the matched values are promises, the defaults as promises to be evaluated in the environment just created. That environment is then used for the evaluation of the body of the function, and promises will be forced (and hence actual or default arguments evaluated) when they are encountered. (Evaluating a promise sets NAMED = NAMEDMAX on its value, so if the argument was a symbol its binding is regarded as having multiple references during the evaluation of the closure call.) [The NAMED mechanism has been replaced by reference counting.]

If the closure is an S3 generic (that is, contains a call to UseMethod) the evaluation process is the same until the UseMethod call is encountered. At that point the argument on which to do dispatch (normally the first) will be evaluated if it has not been already. If a method has been found which is a closure, a new evaluation environment is created for it containing the matched arguments. (Note that this means changes to the values of the formal arguments in the body of the generic are discarded when calling the method, but actual argument promises which have been forced retain the values found when they were forced. On the other hand, missing arguments have values which are promises to use the default supplied by the method and not by the generic.) If the method found is a primitive it is called with the matched argument list of promises (possibly already forced) used for the generic.

The essential difference⁵ between special and builtin functions is that the arguments of specials are not evaluated before the C code is called, and those of builtins are. Note that being a special/builtin is separate from being primitive or .Internal: quote is a special primitive, + is a builtin primitive, cbind is a special .Internal and grep is a builtin .Internal.

Many of the internal functions are internal generics, which for specials means that they do not evaluate their arguments on call, but the C code starts with a call to DispatchOrEval. The latter evaluates the first argument, and looks for a method based on its class. (If S4 dispatch is on, S4 methods are looked for first, even for S3 classes.) If it finds a method, it dispatches to that method with a call based on promises to evaluate the remaining arguments. If no method is found, the remaining arguments are evaluated before return to the internal generic.

The other way that internal functions can be generic is to be group generic. Most such functions are builtins (so immediately evaluate all their arguments), and all contain a call to the C function DispatchGeneric. There are some peculiarities over the number of arguments for the "Math" group generic, with some members allowing only one argument, some having two (with a default for the second) and trunc allows one or more but the default method only accepts one.

• Missingness
• Dot-dot-dot arguments

argument "foobar" is missing, with no default

1.6 Autoprinting ¶

Whether the returned value of a top-level R expression is printed is controlled by the global boolean variable R_Visible. This is set (to true or false) on entry to all primitive and internal functions based on the eval column of the table in file src/main/names.c: the appropriate setting can be extracted by the macro PRIMPRINT.

The R primitive function invisible makes use of this mechanism: it just sets R_Visible = FALSE before entry and returns its argument.

For most functions the intention will be that the setting of R_Visible when they are entered is the setting used when they return, but there need to be exceptions. The R functions identify, options, system and writeBin determine whether the result should be visible from the arguments or user action. Other functions themselves dispatch functions which may change the visibility flag: examples⁶ are .Internal, do.call, eval, withVisible, if, NextMethod, Recall, recordGraphics, standardGeneric, switch and UseMethod.

‘Special’ primitive and internal functions evaluate their arguments internally after R_Visible has been set, and evaluation of the arguments (e.g. an assignment as in PR#9263) can change the value of the flag.

The R_Visible flag can also get altered during the evaluation of a function, with comments in the code about warning, writeChar and graphics functions calling GText (PR#7397). (Since the C-level function eval sets R_Visible, this could apply to any function calling it. Since it is called when evaluating promises, even object lookup can change R_Visible.) Internal and primitive functions force the documented setting of R_Visible on return, unless the C code is allowed to change it (the exceptions above are indicated by PRIMPRINT having value 2).

The actual autoprinting is done by PrintValueEnv in file print.c. If the object to be printed has the S4 bit set and S4 methods dispatch is on, show is called to print the object. Otherwise, if the object bit is set (so the object has a "class" attribute), print is called to dispatch methods: for objects without a class the internal code of print.default is called.

1.7 The write barrier and the garbage collector ¶

R has long had a generational garbage collector, and bit gcgen in the sxpinfo header is used in the implementation of this. This is used in conjunction with the mark bit to identify two previous generations.

There are three levels of collections. Level 0 collects only the youngest generation, level 1 collects the two youngest generations and level 2 collects all generations. After 20 level-0 collections the next collection is at level 1, and after 5 level-1 collections at level 2. Further, if a level-n collection fails to provide 20% free space (for each of nodes and the vector heap), the next collection will be at level n+1. (The R-level function gc() performs a level-2 collection.)

A generational collector needs to efficiently ‘age’ the objects, especially list-like objects (including STRSXPs). This is done by ensuring that the elements of a list are regarded as at least as old as the list when they are assigned. This is handled by the functions SET_VECTOR_ELT and SET_STRING_ELT, which is why they are functions and not macros. Ensuring the integrity of such operations is termed the write barrier and is done by making the SEXP opaque and only providing access via functions (which cannot be used as lvalues in assignments in C).

All code in R extensions is behind the write barrier. R extensions cannot directly access the internals of the SEXPRECs. Base code can access internals if ‘USE_RINTERNALS’ is defined. This is normally defined in Defn.h when R is compiled. To enable a check on the way that the access is used, R can be compiled with flag --enable-strict-barrier which ensures that header Defn.h does not define ‘USE_RINTERNALS’ and hence that SEXP is opaque in most of R itself. (There are some necessary exceptions: foremost in file memory.c where the accessor functions are defined and also in file size.c which needs access to the sizes of the internal structures.)

For background papers see https://homepage.stat.uiowa.edu/~luke/R/barrier.html and https://homepage.stat.uiowa.edu/~luke/R/gengcnotes.html.

1.8 Serialization Formats ¶

Serialized versions of R objects are used by load/save and also at a slightly lower level by saveRDS/readRDS (and their earlier ‘internal’ dot-name versions) and serialize/unserialize. These differ in what they serialize to (a file, a connection, a raw vector) and whether they are intended to serialize a single object or a collection of objects (typically the workspace). save writes a header at the beginning of the file (a single LF-terminated line) which the lower-level versions do not.

save and saveRDS allow various forms of compression, and gzip compression is the default (except for ASCII saves). Compression is applied to the whole file stream, including the headers, so serialized files can be uncompressed or re-compressed by external programs. Both load and readRDS can read gzip, bzip2 and xz forms of compression when reading from a file, and gzip compression when reading from a connection.

R has used the same serialization format called ‘version 2’ from R 1.4.0 in December 2001 until R 3.5.3 in March 2019. It has been expanded in back-compatible ways since its inception, for example to support additional SEXPTYPEs. Earlier formats are still supported via load and save but such formats are not described here. The current default serialization format is called ‘version 3’, and has been introduced in R 3.5.0.

save works by writing a single-line header (typically RDX2\n for a binary save: the only other current value is RDA2\n for save(files=TRUE)), then creating a tagged pairlist of the objects to be saved and serializing that single object. load reads the header line, unserializes a single object (a pairlist or a vector list) and assigns the elements of the object in the specified environment. The header line serves two purposes in R: it identifies the serialization format so load can switch to the appropriate reader code, and the newline \n allows the detection of files which have been subjected to a non-binary transfer which re-mapped line endings. It can also be thought of as a ‘magic number’ in the sense used by the file program (although R save files are not yet by default known to that program).

Serialization in R needs to take into account that objects may contain references to environments, which then have enclosing environments and so on. (Environments recognized as package or name space environments are saved by name.) There are ‘reference objects’ which are not duplicated on copy and should remain shared on unserialization. These are weak references, external pointers and environments other than those associated with packages, namespaces and the global environment. These are handled via a hash table, and references after the first are written out as a reference marker indexed by the table entry.

Version-2 serialization first writes a header indicating the format (normally ‘X\n’ for an XDR format binary save, but ‘A\n’, ASCII, and ‘B\n’, native word-order binary, can also occur) and then three integers giving the version of the format and two R versions (packed by the R_Version macro from Rversion.h). (Unserialization interprets the two versions as the version of R which wrote the file followed by the minimal version of R needed to read the format.) Serialization then writes out the object recursively using function WriteItem in file src/main/serialize.c.

Some objects are written as if they were SEXPTYPEs: such pseudo-SEXPTYPEs cover R_NilValue, R_EmptyEnv, R_BaseEnv, R_GlobalEnv, R_UnboundValue, R_MissingArg and R_BaseNamespace.

For all SEXPTYPEs except NILSXP, SYMSXP and ENVSXP serialization starts with an integer with the SEXPTYPE in bits 0:7⁷ followed by the object bit, two bits indicating if there are any attributes and if there is a tag (for the pairlist types), an unused bit and then the gp field⁸ in bits 12:27. Pairlist-like objects write their attributes (if any), tag (if any), CAR and then CDR (using tail recursion): other objects write their attributes after themselves. Atomic vector objects write their length followed by the data: generic vector-list objects write their length followed by a call to WriteItem for each element. The code for CHARSXPs special-cases NA_STRING and writes it as length -1 with no data. Lengths no more than 2^31 - 1 are written in that way and larger lengths (which only occur on 64-bit systems) as -1 followed by the upper and lower 32-bits as integers (regarded as unsigned).

Environments are treated in several ways: as we have seen, some are written as specific pseudo-SEXPTYPEs. Package and namespace environments are written with pseudo-SEXPTYPEs followed by the name. ‘Normal’ environments are written out as ENVSXPs with an integer indicating if the environment is locked followed by the enclosure, frame, ‘tag’ (the hash table) and attributes.

In the ‘XDR’ format integers and doubles are written in big-endian order: however the format is not fully XDR (as defined in RFC 1832) as byte quantities (such as the contents of CHARSXP and RAWSXP types) are written as-is and not padded to a multiple of four bytes.

The ‘ASCII’ format writes 7-bit characters. Integers are formatted with %d (except that NA_integer_ is written as NA), doubles formatted with %.16g (plus NA, Inf and -Inf) and bytes with %02x. Strings are written using standard escapes (e.g. \t and \013) for non-printing and non-ASCII bytes.

Version-3 serialization extends version-2 by support for custom serialization of ALTREP framework objects. It also stores the current native encoding at serialization time, so that unflagged strings can be converted if unserialized in R running under different native encoding.

1.9 Encodings for CHARSXPs ¶

Character data in R are stored in the SEXPTYPE CHARSXP.

There is support for encodings other than that of the current locale, in particular UTF-8 and the multi-byte encodings used on Windows for CJK languages. A limited means to indicate the encoding of a CHARSXP is via two of the ‘general purpose’ bits which are used to declare the encoding to be either Latin-1 or UTF-8. (Note that it is possible for a character vector to contain elements in different encodings.) Both printing and plotting notice the declaration and convert the string to the current locale (possibly using <xx> to display in hexadecimal bytes that are not valid in the current locale). Many (but not all) of the character manipulation functions will either preserve the declaration or re-encode the character string.

Strings that refer to the OS such as file names need to be passed through a wide-character interface on some OSes (e.g. Windows).

When are character strings declared to be of known encoding? One way is to do so directly via Encoding. The parser declares the encoding if this is known, either via the encoding argument to parse or from the locale within which parsing is being done at the R command line. (Other ways are recorded on the help page for Encoding.)

It is not necessary to declare the encoding of ASCII strings as they will work in any locale. ASCII strings should never have a marked encoding, as any encoding will be ignored when entering such strings into the CHARSXP cache.

The rationale behind considering only UTF-8 and Latin-1 was that most systems are capable of producing UTF-8 strings and this is the nearest we have to a universal format. For those that do not (for example those lacking a powerful enough iconv), it is likely that they work in Latin-1, the old R assumption. Then the parser can return a UTF-8-encoded string if it encounters a ‘\uxxxx’ escape for a Unicode point that cannot be represented in the current charset. (This needs MBCS support, and in the past was only enabled⁹ on Windows.) Now this is enabled for all platforms, and a ‘\uxxxx’ or ‘\Uxxxxxxxx’ escape ensures that the parsed string will be marked as UTF-8.

Most of the character manipulation functions now preserve UTF-8 encodings: there are some notes as to which at the top of file src/main/character.c and in file src/library/base/man/Encoding.Rd.

Graphics devices are offered the possibility of handing UTF-8-encoded strings without re-encoding to the native character set, by setting hasTextUTF8 to be ‘TRUE’ and supplying functions textUTF8 and strWidthUTF8 that expect UTF-8-encoded inputs. Normally the symbol font is encoded in Adobe Symbol encoding, but that can be re-encoded to UTF-8 by setting wantSymbolUTF8 to ‘TRUE’. The Windows’ port of cairographics has a rather peculiar assumption: it wants the symbol font to be encoded in UTF-8 as if it were encoded in Latin-1 rather than Adobe Symbol: this is selected by wantSymbolUTF8 = NA_LOGICAL.

Windows with MSVCRT as the C runtime has no UTF-8 locales, but rather expects to work with UCS-2¹⁰ strings. R (being written in standard C) would not work internally with UCS-2 without extensive changes. The Rgui console¹¹ uses UCS-2 internally, but communicates with the R engine in the native encoding. To allow UTF-8 strings to be printed in UTF-8 in Rgui.exe, an escape convention is used (see header file rgui_UTF8.h) by cat, print and autoprinting.

‘Unicode’ (UCS-2LE) files are common in the Windows world, and readLines and scan will read them into UTF-8 strings on Windows if the encoding is declared explicitly on an unopened connection passed to those functions.

Windows have multiple notions of the current locale encoding, one is in the C runtime (C library) and another is the active code page (system locale). The active code page is used when calling non-UTF-16 variants of Windows API functions (earlier referred to as ANSI calls), either directly or indirectly via POSIX wrappers inside MinGW-w64, from R, R packages and libraries they link to. While R has handled many cases by calling directly the UTF-16 variants of the Windows API, it still may sometimes use the non-UTF-16 ones, and external libraries also primarily developed for POSIX systems typically do that. Therefore, for R to reliably work with (non-ASCII) strings on Windows, both the C locale encoding and the active code page on Windows must be the same, and by default they are.

The Windows UCRT C runtime supports UTF-8 locales. Historically, the active code page was a system-wide setting, changing which required a reboot, and UTF-8 was not supported. Later a "BETA: Use Unicode UTF-8 for worldwide language support" feature has been added to set the active code page to UTF-8, but this still required a reboot and impacted all applications, many of which would not work correctly with that unexpected setting, so it could not be used in practice.

Windows since Windows 10 (version 1903), Windows Server 2022 (LTSC), and Windows Server 1903 (semi-annual channel) allow setting the active code page to UTF-8 in the application manifest. This changes the active code page only for the given application and does so together with changing also the UCRT C locale to UTF-8. R 4.2 for Windows uses this feature to get UTF-8 as the native encoding on Windows. To make that possible, R had to switch to UCRT, which in turn required creation of Rtools42.

Older versions of Windows still rely on the previous encoding support where the native encoding cannot be UTF-8. R 4.2 requires UCRT to work, but UCRT can be installed on Windows since Vista SP2 and Windows Server 2008 SP2. It is shipped with Windows since Windows 10 and Windows Server 2016.

1.10 The CHARSXP cache ¶

There is a global cache for CHARSXPs created by mkChar — the cache ensures that most CHARSXPs with the same contents share storage (‘contents’ including any declared encoding). Not all CHARSXPs are part of the cache – notably ‘NA_STRING’ is not. CHARSXPs reloaded from the save formats of R prior to 0.99.0 are not cached (since the code used is frozen and very few examples still exist).

The cache records the encoding of the string as well as the bytes: all requests to create a CHARSXP should be via a call to mkCharLenCE. Any encoding given in mkCharLenCE call will be ignored if the string’s bytes are all ASCII characters.

1.11 Warnings and errors ¶

Each of warning and stop have two C-level equivalents, warning, warningcall, error and errorcall. The relationship between the pairs is similar: warning tries to fathom out a suitable call, and then calls warningcall with that call as the first argument if it succeeds, and with call = R_NilValue if it does not. When warningcall is called, it includes the deparsed call in its printout unless call = R_NilValue.

warning and error look at the context stack. If the topmost context is not of type CTXT_BUILTIN, it is used to provide the call, otherwise the next context provides the call. This means that when these functions are called from a primitive or .Internal, the imputed call will not be to primitive/.Internal but to the function calling the primitive/.Internal . This is exactly what one wants for a .Internal, as this will give the call to the closure wrapper. (Further, for a .Internal, the call is the argument to .Internal, and so may not correspond to any R function.) However, it is unlikely to be what is needed for a primitive.

The upshot is that warningcall and errorcall should normally be used for code called from a primitive, and warning and error should be used for code called from a .Internal (and necessarily from .Call, .C and so on, where the call is not passed down). However, there are two complications. One is that code might be called from either a primitive or a .Internal, in which case probably warningcall is more appropriate. The other involves replacement functions, where the call was once of the form

> length(x) <- y ~ x
Error in "length<-"(`*tmp*`, value = y ~ x) : invalid value

which is unpalatable to the end user. For replacement functions there will be a suitable context at the top of the stack, so warning should be used. (The results for .Internal replacement functions such as substr<- are not ideal.)

1.12 S4 objects ¶

[This section is currently a preliminary draft and should not be taken as definitive. The description assumes that R_NO_METHODS_TABLES has not been set.]

• Representation of S4 objects
• S4 classes
• S4 methods
• Mechanics of S4 dispatch

 length(nM <- asNamespace("Matrix") )                    # 941 for Matrix 1.2-6
 length(meth <- grep("^[.]__T__", names(nM), value=TRUE))# 107 generics with methods
 length(meth.Ops <- nM$`.__T__Ops:base`) # 71 methods for the 'Ops' (group)generic
 head(sort(names(meth.Ops))) ## "abIndex#abIndex" ... "ANY#ddiMatrix" "ANY#ldiMatrix" "ANY#Matrix"

1.13 Memory allocators ¶

R’s memory allocation is almost all done via routines in file src/main/memory.c.

The rest of R should where possible make use of the allocators made available by file src/main/memory.c, which are also the methods recommended in Memory allocation in Writing R Extensions for use in R packages, namely the use of R_alloc, R_Calloc, R_Realloc and R_Free. Memory allocated by R_alloc is freed by the garbage collector once the ‘watermark’ has been reset by calling vmaxset. This is done automatically by the wrapper code calling primitives and .Internal functions (and also by the wrapper code to .Call and .External), but vmaxget and vmaxset can be used to reset the watermark from within internal code if the memory is only required for a short time.

All of the methods of memory allocation mentioned so far are relatively expensive. All R platforms support alloca, and in almost all cases¹⁴ this is managed by the compiler, allocates memory on the C stack and is very efficient.

There are two disadvantages in using alloca. First, it is fragile and care is needed to avoid writing (or even reading) outside the bounds of the allocation block returned. Second, it increases the danger of overflowing the C stack. It is suggested that it is only used for smallish allocations (up to tens of thousands of bytes), and that

    R_CheckStack();

is called immediately after the allocation (as R’s stack checking mechanism will warn far enough from the stack limit to allow for modest use of alloca). (do_makeunique in file src/main/unique.c provides an example of both points.)

There is an alternative check,

    R_CheckStack2(size_t extra);

to be called immediately before trying an allocation of extra bytes.

An alternative strategy has been used for various functions which require intermediate blocks of storage of varying but usually small size, and this has been consolidated into the routines in the header file src/main/RBufferUtils.h. This uses a structure which contains a buffer, the current size and the default size. A call to

    R_AllocStringBuffer(size_t blen, R_StringBuffer *buf);

sets buf->data to a memory area of at least blen+1 bytes. At least the default size is used, which means that for small allocations the same buffer can be reused. A call to R_FreeStringBufferL releases memory if more than the default has been allocated whereas a call to R_FreeStringBuffer frees any memory allocated.

The R_StringBuffer structure needs to be initialized, for example by

static R_StringBuffer ex_buff = {NULL, 0, MAXELTSIZE};

which uses a default size of MAXELTSIZE = 8192 bytes. Most current uses have a static R_StringBuffer structure, which allows the (default-sized) buffer to be shared between calls to e.g. grep and even between functions: this will need to be changed if R ever allows concurrent evaluation threads. So the idiom is

static R_StringBuffer ex_buff = {NULL, 0, MAXELTSIZE};
...
    char *buf;
    for(i = 0; i < n; i++) {
        compute len
        buf = R_AllocStringBuffer(len, &ex_buff);
        use buf
    }
    /*  free allocation if larger than the default, but leave
        default allocated for future use */
   R_FreeStringBufferL(&ex_buff);

• Internals of R_alloc

1.14 Internal use of global and base environments ¶

This section notes known use by the system of these environments: the intention is to minimize or eliminate such uses.

• Base environment
• Global environment

1.15 Modules ¶

R makes use of a number of shared objects/DLLs stored in the modules directory. These are parts of the code which have been chosen to be loaded ‘on demand’ rather than linked as dynamic libraries or incorporated into the main executable/dynamic library.

For the remaining modules the motivation has been the amount of (often optional) code they will bring in via libraries to which they are linked.

internet

The internal HTTP and FTP clients and socket support, which link to system-specific support libraries. This may load libcurl and on Windows will load wininet.dll and ws2_32.dll.

lapack

The code which makes use of the LAPACK library, and is linked to libRlapack or an external LAPACK library.

X11

(Unix-alikes only.) The X11(), jpeg(), png() and tiff() devices. These are optional, and links to some or all of the X11, pango, cairo, jpeg, libpng and libtiff libraries.

1.16 Visibility ¶

• Hiding C entry points
• Variables in Windows DLLs

nm -g libR.so | grep ' [BCDT] ' | cut -b20-

1.17 Lazy loading ¶

Lazy loading is always used for code in packages but is optional (selected by the package maintainer) for datasets in packages. When a package/namespace which uses it is loaded, the package/namespace environment is populated with promises for all the named objects: when these promises are evaluated they load the actual code from a database.

There are separate databases for code and data, stored in the R and data subdirectories. The database consists of two files, name.rdb and name.rdx. The .rdb file is a concatenation of serialized objects, and the .rdx file contains an index. The objects are stored in (usually) a gzip-compressed format with a 4-byte header giving the uncompressed serialized length (in XDR, that is big-endian, byte order) and read by a call to the primitive lazyLoadDBfetch. (Note that this makes lazy-loading unsuitable for really large objects: the unserialized length of an R object can exceed 4GB.)

The index or ‘map’ file name.rdx is a compressed serialized R object to be read by readRDS. It is a list with three elements variables, references and compressed. The first two are named lists of integer vectors of length 2 giving the offset and length of the serialized object in the name.rdb file. Element variables has an entry for each named object: references serializes a temporary environment used when named environments are added to the database. compressed is a logical indicating if the serialized objects were compressed: compression is always used nowadays. We later added the values compressed = 2 and 3 for bzip2 and xz compression (with the possibility of future expansion to other methods): these formats add a fifth byte to the header for the type of compression, and store serialized objects uncompressed if compression expands them.

Source references are treated specially for performance reasons: bindings lines and parseData from srcfile environments are loaded lazily. This uses a mechanism that allows loading selected bindings from an environment lazily. The key for such environment is a list with two elements: eagerKey gives the length-two integer key for the bindings loaded eagerly and lazyKeys gives a vector of length-two integer keys, one for each lazily loaded binding.

The loader for a lazy-load database of code or data is function lazyLoad in the base package, but note that there is a separate copy to load base itself in file R_HOME/base/R/base.

Lazy-load databases are created by the code in src/library/tools/R/makeLazyLoad.R: the main tool is the unexported function makeLazyLoadDB and the insertion of database entries is done by calls to .Call("R_lazyLoadDBinsertValue", ...).

Lazy-load databases of less than 10MB are cached in memory at first use: this was found necessary when using file systems with high latency (removable devices and network-mounted file systems on Windows).

Lazy-load databases are loaded into the exports for a package, but not into the namespace environment itself. Thus they are visible when the package is attached, and also via the :: operator. This was a deliberate design decision, as packages mostly make datasets available for use by the end user (or other packages), and they should not be found preferentially from functions in the package, surprising users who expected the normal search path to be used. (There is an alternative mechanism, sysdata.rda, for ‘system datasets’ that are intended primarily to be used within the package.)

The same database mechanism is used to store parsed Rd files. One or all of the parsed objects is fetched by a call to tools:::fetchRdDB.

2.1 Special primitives ¶

A small number of primitives are specials rather than builtins, that is they are entered with unevaluated arguments. This is clearly necessary for the language constructs and the assignment operators, as well as for && and || which conditionally evaluate their second argument, and ~, .Internal, call, expression, missing, on.exit, quote and substitute which do not evaluate some of their arguments.

rep and seq.int are special as they evaluate some of their arguments conditional on which are non-missing.

log, round and signif are special to allow default values to be given to missing arguments.

The subsetting, subassignment and @ operators are all special. (For both extraction and replacement forms, $ and @ take a symbol argument, and [ and [[ allow missing arguments.)

UseMethod is special to avoid the additional contexts added to calls to builtins.

2.2 Special internals ¶

There are also special .Internal functions: NextMethod, Recall, withVisible, cbind, rbind (to allow for the deparse.level argument), eapply, lapply and vapply.

2.3 Prototypes for primitives ¶

Prototypes are available for the primitive functions and operators, and these are used for printing, args and package checking (e.g. by tools::checkS3methods and by package codetools). There are two environments in the base package (and namespace), ‘.GenericArgsEnv’ for those primitives which are internal S3 generics, and ‘.ArgsEnv’ for the rest. Those environments contain closures with the same names as the primitives, formal arguments derived (manually) from the help pages, a body which is a suitable call to UseMethod or NULL and environment the base namespace.

The C code for print.default and args uses the closures in these environments in preference to the definitions in base (as primitives).

The QC function undoc checks that all the functions prototyped in these environments are currently primitive, and that the primitives not included are better thought of as language elements (at the time of writing

$  $<-  &&  (  :  @  @<-  [  [[  [[<-  [<-  {  ||  ~  <-  <<-  =
break  for function  if  next  repeat  return  while

). One could argue about ~, but it is known to the parser and has semantics quite unlike a normal function. And : is documented with different argument names in its two meanings.

The QC functions codoc and checkS3methods also make use of these environments (effectively placing them in front of base in the search path), and hence the formals of the functions they contain are checked against the help pages by codoc. However, there are two problems with the generic primitives. The first is that many of the operators are part of the S3 group generic Ops and that defines their arguments to be e1 and e2: although it would be very unusual, an operator could be called as e.g. "+"(e1=a, e2=b) and if method dispatch occurred to a closure, there would be an argument name mismatch. So the definitions in environment .GenericArgsEnv have to use argument names e1 and e2 even though the traditional documentation is in terms of x and y: codoc makes the appropriate adjustment via tools:::.make_S3_primitive_generic_env. The second discrepancy is with the Math group generics, where the group generic is defined with argument list (x, ...), but most of the members only allow one argument when used as the default method (and round and signif allow two as default methods): again fix-ups are used.

Those primitives which are in .GenericArgsEnv are checked (via tests/primitives.R) to be generic via defining methods for them, and a check is made that the remaining primitives are probably not generic, by setting a method and checking it is not dispatched to (but this can fail for other reasons). However, there is no certain way to know that if other .Internal or primitive functions are not internally generic except by reading the source code.

2.4 Adding a primitive ¶

[For R-core use: reverse this procedure to remove a primitive. Most commonly this is done by changing a .Internal to a primitive or vice versa.]

Primitives are listed in the table R_FunTab in src/main/names.c: primitives have ‘Y = 0’ in the ‘eval’ field.

There needs to be an ‘\alias’ entry in a help file in the base package, and the primitive needs to be added to one of the lists at the start of this section.

Some primitives are regarded as language elements (the current ones are listed above). These need to be added to two lists of exceptions, langElts in undoc() (in file src/library/tools/R/QC.R) and lang_elements in tests/primitives.R.

All other primitives are regarded as functions and should be listed in one of the environments defined in src/library/base/R/zzz.R, either .ArgsEnv or .GenericArgsEnv: internal generics also need to be listed in the character vector .S3PrimitiveGenerics. Note too the discussion about argument matching above: if you add a primitive function with more than one argument by converting a .Internal you need to add argument matching to the C code, and for those with a single argument, add argument-name checking.

Do ensure that make check-devel has been run: that tests most of these requirements.

3.1 R code ¶

Internationalization for R code is done in exactly the same way as for extension packages. As all standard packages which have R code also have a namespace, it is never necessary to specify domain, but for efficiency calls to message, warning and stop should include domain = NA when the message is constructed via gettextf, gettext or ngettext.

For each package, the extracted messages and translation sources are stored under package directory po in the source package, and compiled translations under inst/po for installation to package directory po in the installed package. This also applies to C code in packages.

3.2 Main C code ¶

The main C code (e.g. that in files src/*/*.c and in the modules) is where R is closest to the sort of application for which ‘gettext’ was written. Messages in the main C code are in domain R and stored in the top-level directory po with compiled translations under share/locale.

The list of files covered by the R domain is specified in file po/POTFILES.in.

The normal way to mark messages for translation is via _("msg") just as for packages. However, sometimes one needs to mark passages for translation without wanting them translated at the time, for example when declaring string constants. This is the purpose of the N_ macro, for example

{ ERROR_ARGTYPE,           N_("invalid argument type")},

from file src/main/errors.c.

The P_ macro

#ifdef ENABLE_NLS
#define P_(StringS, StringP, N) ngettext (StringS, StringP, N)
#else
#define P_(StringS, StringP, N) (N > 1 ? StringP: StringS)
#endif

may be used as a wrapper for ngettext: however in some cases the preferred approach has been to conditionalize (on ENABLE_NLS) code using ngettext.

The macro _("msg") can safely be used in directory src/appl; the header for standalone ‘nmath’ skips possible translation. (This does not apply to N_ or P_).

3.3 Windows-GUI-specific code ¶

Messages for the Windows GUI are in a separate domain ‘RGui’. This was done for two reasons:

• The translators for the Windows version of R might be separate from those for the rest of R (familiarity with the GUI helps), and
• Messages for Windows are most naturally handled in the native charset for the language, and in the case of CJK languages the charset is Windows-specific. (It transpires that as the iconv we ported works well under Windows, this is less important than anticipated.)

Messages for the ‘RGui’ domain are marked by G_("msg"), a macro that is defined in header file src/gnuwin32/win-nls.h. The list of files that are considered is hardcoded in function update_RGui_po in file src/library/tools/R/translations.R, which is invoked via the update-RGui target of file po/Makefile.win: note that this includes devWindows.c as the menus on the windows device are considered to be part of the GUI. (There is also GN_("msg"), the analogue of N_("msg").)

The template and message catalogs for the ‘RGui’ domain are in the top-level po directory of package base.

3.4 macOS GUI ¶

This is handled separately: see https://developer.r-project.org/Translations30.html.

3.5 Updating ¶

See file po/README for how to update the message templates and catalogs.

4.1 Metadata ¶

Directory Meta contains several files in .rds format, that is serialized R objects written by saveRDS. All packages have files Rd.rds, hsearch.rds, links.rds, features.rds, and package.rds. Packages with namespaces have a file nsInfo.rds, and those with data, demos or vignettes have data.rds, demo.rds or vignette.rds files.

The structure of these files (and their existence and names) is private to R, so the description here is for those trying to follow the R sources: there should be no reference to these files in non-base packages.

File package.rds is a dump of information extracted from the DESCRIPTION file. It is a list of several components. The first, ‘DESCRIPTION’, is a character vector, the DESCRIPTION file as read by read.dcf. Further elements ‘Depends’, ‘Suggests’, ‘Imports’, ‘Rdepends’ and ‘Rdepends2’ record the ‘Depends’, ‘Suggests’ and ‘Imports’ fields. These are all lists, and can be empty. The first three have an entry for each package named, each entry being a list of length 1 or 3, which element ‘name’ (the package name) and optional elements ‘op’ (a character string) and ‘version’ (an object of class ‘"package_version"’). Element ‘Rdepends’ is used for the first version dependency on R, and ‘Rdepends2’ is a list of zero or more R version dependencies—each is a three-element list of the form described for packages. Element ‘Rdepends’ is no longer used, but it is still potentially needed so R < 2.7.0 can detect that the package was not installed for it.

File nsInfo.rds records a list, a parsed version of the NAMESPACE file.

File Rd.rds records a data frame with one row for each help file. The columns are ‘File’ (the file name with extension), ‘Name’ (the ‘\name’ section), ‘Type’ (from the optional ‘\docType’ section), ‘Title’, ‘Encoding’, ‘Aliases’, ‘Concepts’ and ‘Keywords’. All columns are character vectors apart from the last three, which are lists of character vectors.

File hsearch.rds records the information to be used by ‘help.search’. This is a list of four unnamed elements which are character matrices for help files, aliases, keywords and concepts. All the matrices have columns ‘ID’ and ‘Package’ which are used to tie the aliases, keywords and concepts (the remaining column of the last three elements) to a particular help file. The first element has further columns ‘LibPath’ (stored as "" and filled in what the file is loaded), ‘name’, ‘title’, ‘topic’ (the first alias, used when presenting the results as ‘pkgname::topic’) and ‘Encoding’.

File links.rds records a named character vector, the names being aliases and the values character strings of the form

"../../pkgname/html/filename.html"

File data.rds records a two-column character matrix with columns of dataset names and titles from the corresponding help file. File demo.rds has the same structure for package demos.

File vignette.rds records a data frame with one row for each ‘vignette’ (.[RS]nw file in inst/doc) and with columns ‘File’ (the full file path in the sources), ‘Title’, ‘PDF’ (the pathless file name of the installed PDF version, if present), ‘Depends’, ‘Keywords’ and ‘R’ (the pathless file name of the installed R code, if present).

4.2 Help ¶

All installed packages, whether they had any .Rd files or not, have help and html directories. The latter normally only contains the single file 00Index.html, the package index which has hyperlinks to the help topics (if any).

Directory help contains files AnIndex, paths.rds and pkgname.rd[bx]. The latter two files are a lazy-load database of parsed .Rd files, accessed by tools:::fetchRdDB. File paths.rds is a saved character vector of the original path names of the .Rd files, used when updating the database.

File AnIndex is a two-column tab-delimited file: the first column contains the aliases defined in the help files and the second the basename (without the .Rd or .rd extension) of the file containing that alias. It is read by utils:::index.search to search for files matching a topic (alias), and read by scan in utils:::matchAvailableTopics, part of the completion system.

File aliases.rds is the same information as AnIndex as a named character vector (names the topics, values the file basename), for faster access.

6.1 Graphics Devices ¶

R ships with several graphics devices, and there is support for third-party packages to provide additional devices—several packages now do. This section describes the device internals from the viewpoint of a would-be writer of a graphics device.

• Device structures
• Device capabilities
• Handling text
• Conventions
• ‘Mode’
• Graphics events
• Specific devices

typedef struct _GEDevDesc GEDevDesc;
struct _GEDevDesc {
    pDevDesc dev;
    Rboolean displayListOn;
    SEXP displayList;
    SEXP DLlastElt;
    SEXP savedSnapshot;
    Rboolean dirty;
    Rboolean recordGraphics;
    GESystemDesc *gesd[MAX_GRAPHICS_SYSTEMS];
    Rboolean ask;
}

    pGEDevDesc gdd;

    R_GE_checkVersionOrDie(R_GE_version);
    R_CheckDeviceAvailable();
    BEGIN_SUSPEND_INTERRUPTS {
        pDevDesc dev;
        /* Allocate and initialize the device driver data */
        if (!(dev = (pDevDesc) calloc(1, sizeof(DevDesc))))
            return 0; /* or error() */
        /* set up device driver or free 'dev' and error() */
        gdd = GEcreateDevDesc(dev);
        GEaddDevice2(gdd, "dev_name");
    } END_SUSPEND_INTERRUPTS;

int R_GE_getVersion(void);
void R_GE_checkVersionOrDie(int version);

void text(double x, double y, const char *str, double rot, double hadj,
          pGgcontext gc, pDevDesc dd);

double strWidth(const char *str, pGEcontext gc, pDevDesc dd);

void metricInfo(int c, pGEcontext gc,
               double* ascent, double* descent, double* width,
               pDevDesc dd);

    Rboolean Unicode = mbcslocale && (gc->fontface != 5);
    if (c < 0) { Unicode = TRUE; c = -c; }
    if(Unicode) UniCharMetric(c, ...); else CharMetric(c, ...);

double xCharOffset;
double yCharOffset;
double yLineBias;
double cra[2];

dd->cra[0] = 0.9 * fnsize;
dd->cra[1] = 1.2 * fnsize;

    /* Character Addressing Offsets */
    /* These offsets should center a single */
    /* plotting character over the plotting point. */
    /* Pure guesswork and eyeballing ... */

    dd->xCharOffset =  0.4900;
    dd->yCharOffset =  0.3333;
    dd->yLineBias = 0.2;

    void glyph(int n, int *glyphs, double *x, double *y, 
               SEXP font, double size,
               int colour, double rot, pDevDesc dd);

     * device_Mode is called whenever the graphics engine
     * starts drawing (mode=1) or stops drawing (mode=0)
     * GMode (in graphics.c) also says that
     * mode = 2 (graphical input on) exists.
     * The device is not required to do anything

6.2 Colours ¶

Devices receive colours as a typedef rcolor (an unsigned int) defined in the header R_ext/GraphicsEngine.h). The 4 bytes are R ,G, B and alpha from least to most significant. So each of RGB has 256 levels of luminosity from 0 to 255. The alpha byte represents opacity, so value 255 is fully opaque and 0 fully transparent: many but not all devices handle semi-transparent colours.

Colors can be created in C via the macro R_RGBA, and a set of macros are defined in R_ext/GraphicsDevice.h to extract the various components.

Colours in the base graphics system were originally adopted from S (and before that the GRZ library from Bell Labs), with the concept of a (variable-sized) palette of colours referenced by numbers ‘1...N’ plus ‘0’ (the background colour of the current device). R introduced the idea of referring to colours by character strings, either in the forms ‘#RRGGBB’ or ‘#RRGGBBAA’ (representing the bytes in hex) as given by function rgb() or via names: the 657 known names are given in the character vector colors and in a table in file colors.c in package grDevices. Note that semi-transparent colours are not ‘premultiplied’, so 50% transparent white is ‘#ffffff80’.

Integer or character NA colours are mapped internally to transparent white, as is the character string "NA".

Negative colour numbers are an error. Colours greater than ‘N’ are wrapped around, so that for example with the default palette of size 8, colour ‘10’ is colour ‘2’ in the palette.

Integer colours have been used more widely than the base graphics sub-system, as they are supported by package grid and hence by lattice and ggplot2. (They are also used by package rgl.) grid did re-define colour ‘0’ to be transparent white, but rgl used col2rgb and hence the background colour of base graphics.

Note that positive integer colours refer to the current palette and colour ‘0’ to the current device (and a device is opened if needs be). These are mapped to type rcolor at the time of use: this matters when re-playing the display list, e.g. when a device is resized or dev.copy is used. The palette should be thought of as per-session: it is stored in package grDevices.

The convention is that devices use the colorspace ‘sRGB’. This is an industry standard: it is used by Web browsers and JPEGs from all but high-end digital cameras. The interpretation is a matter for graphics devices and for code that manipulates colours, but not for the graphics engine or subsystems.

R uses a painting model similar to PostScript and PDF. This means that where shapes (circles, rectangles and polygons) can both be filled and have a stroked border, the fill should be painted first and then the border (or otherwise only half the border will be visible). Where both the fill and the border are semi-transparent there is some room for interpretation of the intention. Most devices first paint the fill and then the border, alpha-blending at each step. However, PDF does some automatic grouping of objects, and when the fill and the border have the same alpha, they are painted onto the same layer and then alpha-blended in one step. (See p. 569 of the PDF Reference Sixth Edition, version 1.7. Unfortunately, although this is what the PDF standard says should happen, it is not correctly implemented by some viewers.)

The mapping from colour numbers to type rcolor is primarily done by function RGBpar3: this is exported from the R binary but linked to code in package grDevices. The first argument is a SEXP pointing to a character, integer or double vector, and the second is the rcolor value for colour 0 (or "0"). C entry point RGBpar is a wrapper that takes 0 to be transparent white: it is often used to set colour defaults for devices. The R-level wrapper is col2rgb.

There is also R_GE_str2col which takes a C string and converts to type rcolor: "0' is converted to transparent white.

There is a R-level conversion of colours to ‘##RRGGBBAA’ by image.default(useRaster = TRUE).

The other color-conversion entry point in the API is name2col which takes a colour name (a C string) and returns a value of type rcolor. This handles "NA", "transparent" and the 657 colours known to the R function colors().

6.3 Base graphics ¶

The base graphics system was migrated to package graphics in R 3.0.0: it was previously implemented in files in src/main.

For historical reasons it is largely implemented in two layers. Files plot.c, plot3d.c and par.c contain the code for the around 30 .External calls that implement the basic graphics operations. This code then calls functions with names starting with G and declared in header Rgraphics.h in file graphics.c, which in turn call the graphics engine (whose functions almost all have names starting with GE).

A large part of the infrastructure of the base graphics subsystem are the graphics parameters (as set/read by par()). These are stored in a GPar structure declared in the private header Graphics.h. This structure has two variables (state and valid) tracking the state of the base subsystem on the device, and many variables recording the graphics parameters and functions of them.

The base system state is contained in baseSystemState structure defined in the private header GraphicsBase.h. This contains three GPar structures and a Boolean variable used to record if plot.new() (or persp) has been used successfully on the device.

The three copies of the GPar structure are used to store the current parameters (accessed via gpptr), the ‘device copy’ (accessed via dpptr) and space for a saved copy of the ‘device copy’ parameters. The current parameters are, clearly, those currently in use and are copied from the ‘device copy’ whenever plot.new() is called (whether or not that advances to the next ‘page’). The saved copy keeps the state when the device was last completely cleared (e.g. when plot.new() was called with par(new=TRUE)), and is used to replay the display list.

The separation is not completely clean: the ‘device copy’ is altered if a plot with log scale(s) is set up via plot.window().

There is yet another copy of most of the graphics parameters in static variables in graphics.c which are used to preserve the current parameters across the processing of inline parameters in high-level graphics calls (handled by ProcessInlinePars).

Snapshots of the base subsystem record the ‘saved device copy’ of the GPar structure.

• Arguments and parameters

6.4 Grid graphics ¶

[At least pointers to documentation.]

7.1 R.app ¶

R.app is a macOS application which provides a console. Its sources are a separate project²¹, and its binaries link to an R installation which it runs as a dynamic library libR.dylib. The standard CRAN distribution of R for macOS bundles the GUI and R itself, but installing the GUI is optional and either component can be updated separately.

R.app relies on libR.dylib being in a specific place, and hence on R having been built and installed as a Mac macOS ‘framework’. Specifically, it uses /Library/Frameworks/R.framework/R. This is a symbolic link, as frameworks can contain multiple versions of R. It eventually resolves to /Library/Frameworks/R.framework/Versions/Current/Resources/lib/libR.dylib, which is (in the CRAN distribution) a ‘fat’ binary containing multiple sub-architectures.

macOS applications are directory trees: each R.app contains a front-end written in Objective-C for one sub-architecture: in the standard distribution there are separate applications for 32- and 64-bit Intel architectures.

Originally the R sources contained quite a lot of code used only by the macOS GUI, but this was migrated to the R.app sources.

R.app starts R as an embedded application with a command-line which includes --gui=aqua (see below). It uses most of the interface pointers defined in the header Rinterface.h, plus a private interface pointer in file src/main/sysutils.c. It adds an environment it names tools:RGUI to the second position in the search path. This contains a number of utility functions used to support the menu items, for example package.manager(), plus functions q() and quit() which mask those in package base—the custom versions save the history in a way specific to R.app.

There is a configure option --with-aqua for R which customizes the way R is built: this is distinct from the --enable-R-framework option which causes make install to install R as the framework needed for use with R.app. (The option --with-aqua is the default on macOS.) It sets the macro HAVE_AQUA in config.h and the make variable BUILD_AQUA_TRUE. These have several consequences:

• The quartz() device is built (other than as a stub) in package grDevices: this needs an Objective-C compiler. Then quartz() can be used with terminal R provided the latter has access to the macOS screen.
• File src/unix/aqua.c is compiled. This now only contains an interface pointer for the quartz() device(s).
• capabilities("aqua") is set to TRUE.
• The default path for a personal library directory is set as ~/Library/R/arch/x.y/library.
• There is support for setting a ‘busy’ indicator whilst waiting for system() to return.
• R_ProcessEvents is inhibited in a forked child from package parallel. The associated callback in R.app does things which should not be done in a child, and forking forks the whole process including the console.
• There is support for starting the embedded R with the option --gui=aqua: when this is done the global C variable useaqua is set to a true value. This has consequences:
  • The R session is asserted to be interactive via R_Interactive.
  • .Platform$GUI is set to "AQUA". That has consequences:
    • The environment variable DISPLAY is set to ‘:0’ if not already set.
    • /usr/local/bin is appended to PATH since that is where gfortran is installed.
    • The default HTML browser is switched to the one in R.app.
    • Various widgets are switched to the versions provided in R.app: these include graphical menus, the data editor (but not the data viewer used by View()) and the workspace browser invoked by browseEnv().
    • The grDevices package when loaded knows that it is being run under R.app and so informs any quartz devices that a Quartz event loop is already running.
  • The use of the OS’s system function (including by system() and system2(), and to launch editors and pagers) is replaced by a version in R.app (which by default just calls the OS’s system with various signal handlers reset).
• If either R was started by --gui=aqua or R is running in a terminal which is not of type ‘dumb’, the standard output to files stdout and stderr is directed through the C function Rstd_WriteConsoleEx. This uses ANSI terminal escapes to render lines sent to stderr as bold on stdout.
• For historical reasons the startup option -psn is allowed but ignored. (It seems that in 2003, ‘r27492’, this was added by Finder.)

12.1 Long vectors ¶

Vectors in R 2.x.y were limited to a length of 2^31 - 1 elements (about 2 billion), as the length is stored in the SEXPREC as a C int, and that type is used extensively to record lengths and element numbers, including in packages.

Note that longer vectors are effectively impossible under 32-bit platforms because of their address limit, so this section applies only on 64-bit platforms. The internals are unchanged on a 32-bit build of R.

A single object with 2^31 or more elements will take up at least 8GB of memory if integer or logical and 16GB if numeric or character, so routine use of such objects is still some way off.

There is now some support for long vectors. This applies to raw, logical, integer, numeric and character vectors, and lists and expression vectors. (Elements of character vectors (CHARSXPs) remain limited to 2^31 - 1 bytes.) Some considerations:

• This has been implemented by recording the length (and true length) as -1 and recording the actual length as a 64-bit field at the beginning of the header. Because a fair amount of code in R uses a signed type for the length, the ‘long length’ is recorded using the signed C99 type ptrdiff_t, which is typedef-ed to R_xlen_t.
• These can in theory have 63-bit lengths, but note that current 64-bit OSes do not even theoretically offer 64-bit address spaces and there is currently a 52-bit limit (which exceeds the theoretical limit of current OSes and ensures that such lengths can be stored exactly in doubles).
• The serialization format has been changed to accommodate longer lengths, but vectors of lengths up to 2^31-1 are stored in the same way as before. Longer vectors have their length field set to -1 and followed by two 32-bit fields giving the upper and lower 32-bits of the actual length. There is currently a sanity check which limits lengths to 2^48 on unserialization.
• The type R_xlen_t is made available to packages in C header Rinternals.h: this should be fine in C code since C99 is required. People do try to use R internals in C++, but C++98 compilers are not required to support these types.
• Indexing can be done via the use of doubles. The internal indexing code used to work with positive integer indices (and negative, logical and matrix indices were all converted to positive integers): it now works with either INTSXP or REALSXP indices.
• The R function length returns a double value if the length exceeds 2^31-1. Code calling as.integer(length(x)) before passing to .C/.Fortran should checks for an NA result.

12.2 64-bit types ¶

There is also some desire to be able to store larger integers in R, although the possibility of storing these as double is often overlooked (and e.g. file pointers as returned by seek are already stored as double).

Different routes have been proposed:

• Add a new type to R and use that for lengths and indices—most likely this would be a 64-bit signed type, say longint. R’s usual implicit coercion rules would ensure that supplying an integer vector for indexing or length<- would work.
• A more radical alternative is to change the existing integer type to be 64-bit on 64-bit platforms (which was the approach taken by S-PLUS for DEC/Compaq Alpha systems). Or even on all platforms.
• Allow either integer or double values for lengths and indices, and return double only when necessary.

The third has the advantages of minimal disruption to existing code and not increasing memory requirements. In the first and third scenarios both R’s own code and user code would have to be adapted for lengths that were not of type integer, and in the third code branches for long vectors would be tested rarely.

Most users of the .C and .Fortran interfaces use as.integer for lengths and element numbers, but a few omit these in the knowledge that these were of type integer. It may be reasonable to assume that these are never intended to be used with long vectors.

The remaining interfaces will need to cope with the changed VECTOR_SEXPREC types. It seems likely that in most cases lengths are accessed by the length and LENGTH functions²⁶ The current approach is to keep these returning 32-bit lengths and introduce ‘long’ versions xlength and XLENGTH which return R_xlen_t values.

See also https://homepage.cs.uiowa.edu/~luke/talks/useR10.pdf.

12.3 Large matrices ¶

Matrices are stored as vectors and so were also limited to 2^31-1 elements. Now longer vectors are allowed on 64-bit platforms, matrices with more elements are supported provided that each of the dimensions is no more than 2^31-1. However, not all applications can be supported.

The main problem is linear algebra done by Fortran code compiled with 32-bit INTEGER. Although not guaranteed, it seems that all the compilers currently used with R on a 64-bit platform allow matrices each of whose dimensions is less than 2^31 but with 2^31 or more elements and index them correctly, and a substantial part of the support software (such as BLAS and LAPACK) also work.

There are exceptions: for example some complex LAPACK auxiliary routines do use a single INTEGER index and hence overflow silently and segfault or give incorrect results. One example seen was svd() on a complex matrix.

Since this is implementation-dependent, it is possible that optimized BLAS and LAPACK may have further restrictions: a segfault have been reported from svd() using ATLAS on ‘x86_64’ Linux.

For matrix algebra on large matrices one almost certainly wants a machine with a lot of RAM (100s of gigabytes), many cores and a multi-threaded BLAS.