Ø
«
»
0 : pbulk
1 : Overview
2 : Some background
3 : Anatomy of a bulk build
4 : Specific problems
5 : pbulk components (I)
6 : pbulk components (II)
7 : The scan component
8 : The resolver component
9 : The build component
10 : Future plans
11 : Q&A
pbulk
Jörg Sonnenberger
joerg@NetBSD.org
Overview
Background and Motivation
Anatomy of a bulk build
pbulk components
Future plans
Some background
Fixing issues in FreeBSD ports since end of 2003
Moved to pkgsrc in summer 2005
Regular bulk builds since October 2005
Rule: A problem fixed in the bulk build is a problem no user can complain about.
Rule: A problem in a bulk build is easier to reproduce than a problem from a user.
Binary packages are a by-product
Anatomy of a bulk build
Setup of machine and bulk build
Prepare the pkgsrc tree
Find out what to build
Build packages
Summarize results
Upload results
Upload binary packages
Specific problems
Stray directories in /usr/pkg
Stray dependencies in /usr/pkg (gmake, pkg-config)
.if exists(${X11BBASE}/include/X11/Xdefs.h) ... .endif
math/p5-Math-Pari
Writing to /etc
Writing to /home
Scalability
pbulk components (I)
pbulk-scan: Schedule scanning of pkgsrc parts
pbulk-resolve: Compute dependency tree
pbulk-build: Schedule build of packages
Small glue in mk/pbulk for scan target
pbulk components (II)
Schedulers in C: network access, memory usage
Resolver in C: code reuse for Dewey, speed
Upload logic somewhat overlapped with pbulk-build
Report logic completely in awk
The scan component
Take list of PKGPATH or extract subdirectories using make
Pull/push model for distributed scans
Call "make pbulk-index" for every PKGPATH
make is the only way to get reliable results
Collect and reorder list, drop duplicate entries
The resolver component
Implement a mix of pkg_add and pkgsrc algorithm to find dependencies
If in doubt, prefer pkgsrc view, but warn
{gs,gs-esp}-[0-9]*:../../print/ghostscript requested, gs-7.0 and gs-esp-7.0nb1 exist — what to pick?
Complain or list what couldn't be found
The build component
Pull/push model for distributed scans
Find package with all dependencies done and highest priority
Sort order: number of packages depending on this
Future plans
Allow to easier restrict multi-version logic
Allow using incremental mode + post-filtering when missing dependencies
Move restricted subset logic for upload into upload itself
Tag bootstrap packages and don't install them
Remove package if install/deinstall test fails
Implement poll loop directly to get rid of libevent
Split into base part and distinguish official bulk build and simple user case
Q&A
Questions?