Jörg Sonnenberger

Bulk building in the many core era

AsiaBSDCon 2017

Agenda

  1. A short history of pbulk
  2. Modern requirements
  3. Configuration template
  4. Impact of configuration choices
  5. Current work
  6. Future work

A short history of pbulk

  • Project from Google's Summer of Code 2007
  • Goal: Provide binary packages and regression tests
  • Requirements:
    • Parallel builds using sandboxes (chroot, jail, zone)
    • Distributed builds in a LAN
    • Reproducable builds
    • Customizable build
  • Assumptions:
    • Ressource sharing via null mounts, NFS
    • Restartable builds
    • Start-and-forget use

Modern requirements

  • Shared filesystems ↔ explicit file transfers
  • Fast rescanning
  • Master may run on different platform
  • Semi-interactive use for developers
  • Package signing
  • Intelligent scheduling

Configuration template (I): distfiles

  • Problem for shared DISTDIR: partial downloads
  • Fix: Local distfile mirror for all pkgsrc users
  • Reduces impact of race conditions
  • New distfile pushed via HTTP-POST or SCP: 
    MASTER_SITE_OVERRIDE = http://distfiles.example.com/
DISTDIR = ${WRKDIR}/.distfiles
POST_FETCH_HOOK = .../upload-distfile
IGNORE_INTERACTIVE_FETCH = yes

Configuration template (II): log files

  • Sharing via NFS: occasional IO errors
  • Sharing via null mounts: lock contention unclear
  • Push-model for post-build upload available
  • Preference for HTTP-POST or SCP over null mounts?
  • keep_wrkdir=yes very useful for debugging, but can require 10GB+ space
  • ...but tarballs are not uploaded

Configuration template (III): packages

  • Direct read-only access needed up-to-date
  • Upload via HTTP-POST or SCP possible
  • ...allows package signing on master
  • ...key material only on master, HSM possible

Configuration template (III):
scratch directories

  • Throw-away content, recreated every build
  • ${PREFIX} typically <1GB, peak: ~5GB
  • ${WRKOBJDIR} typically <1GB, peak: 10GB+
  • tmpfs is useful with enough RAM: no wear
  • async file systems possible, i.e. over iSCSI

Configuration template (IV):
configuration (I)

  • USE_CWRAPPERS=yes easily 10% faster
  • PKG_COMPRESSION=xz slower, but gzip 39% larger
  • Process limits:
    • time: typically limited to 60min
    • memory: 2GB VA limit
    • processes/user, number of files: double check!

Configuration template (IV):
configuration (II)

  • MAKE_JOBS vs chroots:
    • configuration, install phase serialized
    • balance of memory overhead
    • CPU over commit mostly harmless
    • 1GB * MAKE_JOBS vs. 8GB scratch directories

Impact of configuration choices

  • On-going project
  • How to create individual chroots?
  • What's the cost of a hypervisor?
  • Further suggestions welcome!

Root file system: nullfs vs tmpfs (I)

  • Sets: kernel, base, comp, etc, games, misc, text
  • ...optionally: xbase, xcomp, xetc, xfonts
  • Man pages are pruned
  • 500MB total (with xsrc: 600MB)
  • Option 1: read-only tmpfs
  • Option 2: read-only null mount

Root file system: nullfs vs tmpfs (II)

  • NetBSD current, clang as system compiler
  • 2x Xeon E5-2630 v3: 16 cores + HT
  • 64GB RAM, SSD for local storage


Configuration Scan phase only RSD Full build RSD
tmpfs 106min 5% 2178min 1.2%
nullfs 107min 4% 2797min 1.9%
+28%

Bare metal vs Xen Paravirtualisation

  • Base setup: tmpfs configuration as before
  • Small NetBSD dom0, 60GB RAM for PV domU
  • Disk IO expected to be insignificant


Configuration Scan phase only RSD Full build RSD
Bare metal 106min 5% 2178min 1.2%
Xen (1 DOMU) 186min 1.4% 2893min 3.2%
+76% +33%
Xen (4 DOMUs) 33min 0.8% 1913min 0.1%
-69% -12%

Bare metal vs Full Virtualisation

  • Xen: same as 4 DOMUs case above, only guest type changed
  • SmartOS: manual limit for ARC size, 4 * 14GB KVM zones


Configuration Scan phase only RSD Full build RSD
Bare metal 106min 5% 2178min 1.2%
Xen (4 DOMUs) 33min 0.8% 1913min 0.1%
Xen HVM 38min 1.8% 2852min 1.5%
-65% +31%
SmartOS KVM 42min 7.4% 2741min 0.5%
-61% +26%

Current work

  • Investigate moving up-to-date check into the master
  • Move data storage into SQLite database
  • ...significantly reduces memory footprint
  • ...checksum based up-to-date check
  • Integrate idle process monitor
  • ...hello GNU make 4.2.1, Onyx
  • Figure out how to best compute ressource consumption

Future work

  • Moving to HTTP(S) based communication protocol
  • ...with integrated package/log upload
  • ...and a nice web GUI for progress reports
  • New scheduler based on ressource consumption

Q&A