System libraries on Windows

class: center, middle, inverse, title-slide

# <code>System libraries on Windows</code>
## <code>DSC 2019</code>
### <code>Jeroen Ooms</code>
### <code>2019/09/19</code>

---

# Outline

In this talk:

- System libraries on Mac/Windows are complicated.

- Rtools 4.0 includes `pacman`: a package manager  (cf. `apt`, `yum`) to build, distribute and install system libraries for building R packages on Windows.

- This allows CRAN and package authors to build and install up-to-date system libs using Rtools compilers on Windows.

- From the perspective of R base / users nothing changes. The compilers from rtools40 are used exactly the same way as previous Rtools.

---

# Background

[![pkgscreen](packages4.png)](https://cran.r-project.org/web/checks/check_results_jeroen_at_berkeley.edu.html)

---

# Background

[![pkgscreen](packages5.png)](https://cran.r-project.org/web/checks/check_results_jeroen_at_berkeley.edu.html)

---
background-image: url(screen3.png)
background-position: 50% 100%
background-size: 110%

# Maintainer of Rtools, Windows Installers

---
class: inverse, center, middle

# PART I: System Libraries in R

---

background-image: url(cransize.png)
background-position: 50% 50%
background-size: 100%

# CRAN is always growing

---

background-image: url(crandeps.png)
background-position: 50% 50%
background-size: 100%

# CRAN system dependencies

---

background-image: url(gdaldeps.png)
background-position: 50% 50%
background-size: 100%

# Some libs depend on many more libs

---

# System libraries

CRAN is growing in size but also complexity:

- Increasing proportion of user runtime is spent in external libs.
- Need properly configured, up-to-date system libs for these R packages to work well.

On Linux we can rely on system libs from `apt` / `yum`.

But on Windows / Mac, the OS does not provide libs. We need to build our own (using Rtools compilers on Windows).

<table>
  <tr>
    <th></th>
    <th>Native Compiler</th>
    <th>Package Manager</th>
    <th>Linking</th>
  </tr>
  <tr>
    <th>Linux</th>
    <td>yes</td>
    <td>yes</td>
    <td>dynamic</td>
  </tr>
  <tr>
    <th>MacOS</th>
    <td>yes</td>
    <td style="color: red;">no</td>
    <td>static</td>
    <td></td>
  </tr>
  <tr">
    <th>Windows</th>
    <td style="color: red;">no</td>
    <td style="color: red;">no</td>
    <td>static</td>
  </tr>
</table>

Additional challenge is that we need to build all of these __statically__ so that we get these __standalone R binary packages__, that do not depend on runtime dll’s from external system libraries that the user does not have.

---
# System libraries on Windows

The problem:

> Currently R has no official method of providing system libs on Windows / Mac.

Some libraries happen to be available on the build server. Package authors have no way of knowing if/which libs are available, and they are rarely updated.

This is especially problematic on Windows because system libraries need to be built with Rtools (or compatible) toolchains.

---
# Unofficial solution

Currently we manually build most libraries needed for CRAN with Rtools compilers and host them on Github for packages to download on-the-fly at install time.

![rwinlib](rwinlib.png)

---

# System libraries on Windows

This is sub-optimal in many ways:

- High maintenance (doesn't scale well)
 - Difficult to debug/reproduce for pkg authors
 - Downloading at install-time may be insecure

For the new Rtools we want to:

- Take advantage of existing building and packaging tools from msys2
 - Automate the build and distribution process for system libraries
 - Allow contributors to provide updates/new packaging scripts 
 - Simplify building and debugging of base R on Windows

---
class: inverse, center, middle

# PART II: Rtools 4.0

---
# Rtools 4.0

MSYS2 is a software distro and building platform for mingw-w64  (Windows). Maintained by some of the same people as mingw with contributors from python, conda, etc.

![msys2-website](msys2-website.png)

---
# Rtools 4.0

What do we gain by using msys2?

- Similar to Rtools, but with a proper package manager, terminal, etc.
 - Large community, most libs and tools are already available.
 - As close to unix build/debugging environment as you will get on Windows
 
--

For rtools 4.0 we use msys2 build system, but with own repository of mingw packages:

- Toolchain/libraries reconfigured for static linking.
 - Configured with features and dependencies that we need in R.
 
Building libraries can both be done locally or automatically on CI servers.

---

# Using Rtools 4.0

Rtools 4.0 beta and a version of R that has been configured for rtools40 is available from CRAN:

[![rtoolscran](rtoolscran.png)](https://cloud.r-project.org/bin/windows/testing/rtools40.html)

---

background-image: url(startmenu.png)
background-position: 0% 100%
background-size: 115%

# Using Rtools 4.0

---

# Using Rtools 4.0

<br />

![rtools40](rtools40.png)

---
# What's included?

Rtools 4.0 is a full unix-like environment based on msys2 (cygwin):

--
 - terminal: (mintty)

- shell (bash)

- make, sed, perl, etc

- mingw-w64 compilers targeting i686 and x86_84
 
--

- __a package manager to build and install anything else: pacman__

---
# Using Pacman

Pacman is a port of the arch package manager. Some examples:

```r
# Sync repository index
pacman -Sy

# Upgrade installed packages
pacman -Su
```

To install new packages (for example libxml2)

```r
# Install both 32 and 64 bit build of libxm2
pacman -S mingw-w64-{i686,x86_64}-libxml2
```

More examples: https://cloud.r-project.org/bin/windows/testing/rtools40.html

---
# Installing Libraries with Pacman

<video id="rtoolsvideo" class="fullscreenvideo" controls>
  <source src="rtools-fast.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>

---
# Pacman Package Formulas for Rtools 4.0

![rtools-packages](rtools-packages.png)

---
# Rtools 4.0 Package Format

![arrow-pkgbuild](arrow-pkgbuild.png)

---
# Building a Package in Rtools 4.0

<video id="rtoolscmake" class="fullscreenvideo" controls>
  <source src="cmake-fast.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>

---
# Automatic Building on AppVeyor

![appveyor-status](appveyor-status.png)

---
# Automatic Building on AppVeyor

![rtoolsav](rtoolsav.png)

---
# Automatic Deployment to Bintray

![appveyor-log](appveyor-log.png)

---
# The Rtools 4.0 Binary Repository

![appveyor-deploy](appveyor-deploy.png)

---
# Package is now live!

Now in Rtools 4.0 when users do:

```r
pacman -Syu
```

It will automatically update to the new Rtools 4.0 build.

RTools will automaticaly find the new libraries when building R package. No need to manually download and copy system libs and pass special compiler / linker flags.

Should feel much more like `apt` / `yum` / `brew`, where external libraries needed for building R packages can be installed from the command line, and it just works.

---
# Recap

In Summary:

- Rtools as proper __build__ environment and package manager for __external libs__.

- Pacman allows for __packaging, distributing and installing__ libs, same as Linux, Homebrew.

- R automatically finds installed libs. No manual downloading / settings headers/libs.

- __Transparent and reproducible__ builds, __automated__ via CI, collaborative maintenance.

This will allow us to provide reliable, up-to-date system libs for Rtools on Windows.

- Improve Windows support for packages interfacing C/C++ libraries
 
 - Also makes it easier to ship debugging tools (gdb, drmingw, etc)
 
 - Simplfy building / debugging base R on Windows

These slides: `https://jeroen.github.io/dsc2019`