pbulk

Jörg Sonnenberger

joerg@NetBSD.org

Overview

• Background and Motivation
• Anatomy of a bulk build
• pbulk components
• Future plans

Some background

• Fixing issues in FreeBSD ports since end of 2003
• Moved to pkgsrc in summer 2005
• Regular bulk builds since October 2005
• Rule: A problem fixed in the bulk build is a problem no user can complain about.
• Rule: A problem in a bulk build is easier to reproduce than a problem from a user.
• Binary packages are a by-product

Anatomy of a bulk build

• Setup of machine and bulk build
• Prepare the pkgsrc tree
• Find out what to build
• Build packages
• Summarize results
• Upload results
• Upload binary packages

Specific problems

• Stray directories in /usr/pkg
• Stray dependencies in /usr/pkg (gmake, pkg-config)
• .if exists(${X11BBASE}/include/X11/Xdefs.h) ... .endif
• math/p5-Math-Pari
• Writing to /etc
• Writing to /home
• Scalability

pbulk components (I)

• pbulk-scan: Schedule scanning of pkgsrc parts
• pbulk-resolve: Compute dependency tree
• pbulk-build: Schedule build of packages
• Small glue in mk/pbulk for scan target

pbulk components (II)

• Schedulers in C: network access, memory usage
• Resolver in C: code reuse for Dewey, speed
• Upload logic somewhat overlapped with pbulk-build
• Report logic completely in awk

The scan component

• Take list of PKGPATH or extract subdirectories using make
• Pull/push model for distributed scans
• Call "make pbulk-index" for every PKGPATH
• make is the only way to get reliable results
• Collect and reorder list, drop duplicate entries

The resolver component

• Implement a mix of pkg_add and pkgsrc algorithm to find dependencies
• If in doubt, prefer pkgsrc view, but warn
• {gs,gs-esp}-[0-9]*:../../print/ghostscript requested, gs-7.0 and gs-esp-7.0nb1 exist — what to pick?
• Complain or list what couldn't be found

The build component

• Pull/push model for distributed scans
• Find package with all dependencies done and highest priority
• Sort order: number of packages depending on this

Future plans

• Allow to easier restrict multi-version logic
• Allow using incremental mode + post-filtering when missing dependencies
• Move restricted subset logic for upload into upload itself
• Tag bootstrap packages and don't install them
• Remove package if install/deinstall test fails
• Implement poll loop directly to get rid of libevent
• Split into base part and distinguish official bulk build and simple user case

Q&A

Questions?