Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzBuild lfs system with sway lol
BUILD.md | 298 + HANDOFF-history.md | 10550 +++++++++++++++++++ HANDOFF-sysv-session.md | 86 + HANDOFF.md | 1351 +-- Makefile | 118 +- README.md | 150 +- blfs | 2350 ++++- kernel.conf | 114 + lfs | 1813 +++- lfs-helper | 4226 +++++++- lfs-kernel | 789 ++ lfs-phases | 568 + lfs-pkgusr.zip | Bin 0 -> 1031198 bytes lfs-sanity.sh | 71 +- lfs-sysvbook | 641 ++ packagemanager | 5212 ++++++++- stacks/kernel-sway.sh | 30 + stacks/machine.conf | 33 + stacks/mytools.sh | 108 + stacks/mytools/install_mytools | 116 + stacks/network.stack | 42 + stacks/package-fixes.conf | 60 + stacks/services.stack | 63 + stacks/shell-env.sh | 88 + stacks/sway-session.sh | 102 + stacks/sway.stack | 302 + test_lfs_crosschain.sh | 22147 ++++++++++++++++++++++++++++++--------- 27 files changed, 44129 insertions(+), 7299 deletions(-) diff --git a/BUILD.md b/BUILD.md new file mode 100644 index 0000000..0f8dcbf --- /dev/null +++ b/BUILD.md @@ -0,0 +1,298 @@ +# Build a whole system + +From an empty partition to a booted SysV desktop, with every file owned by a +package user. One document, in build order. Every step says what to +**configure first**, because the tools read their settings when a step +starts, not before. + +> **100% vibecode, but tested.** These are the commands that built the +> reference machine. Dry run is the default everywhere; `--run` applies. + +--- + +## The shape of it + +| Where | You run | It does | +|---|---|---| +| host | `lfs run` | books, session, sources, cross toolchain, enters the chroot | +| chroot | `lfs-helper build-all` | chapters 7-9, one package user each | +| host | `lfs build-system install-tools --run` | hands over from `lfs-helper` to `packagemanager` | +| chroot | `packagemanager bootstrap --run` | wget, Python modules, BLFS book, certificates | +| chroot | `packagemanager stack <name> --run` | BLFS packages, in stacks | +| chroot | `lfs-kernel` | kernel, modules, firmware, headers | +| host | `lfs-helper build refind` | a boot entry beside your current one | +| booted | `packagemanager verify` | is everything where it should be | + +`lfs run` and `lfs-helper build-all` both continue from wherever they +stopped. Run them again after a failure, a cancel or a reboot. + +--- + +## 0. Install the tools on the host + +```sh +make install # /usr/bin; make uninstall exists +lfs --version +``` + +Needs Python 3.9+, `beautifulsoup4`, `requests`, bash, tar. + +--- + +## 1. Books + +LFS stopped shipping a SysV edition with 13.0. Make one: + +```sh +lfs fetch 12.4 # donor: last SysV book +lfs fetch 13.0-systemd # target: current release +lfs-sysvbook make # -> 13.0-sysv in the store +lfs-sysvbook check --book 13.0-sysv +lfs build-system set-book 13.0-sysv +``` + +For a plain systemd system, `lfs fetch 13.0-systemd` and `set-book` that. + +`lfs run` also asks for the **BLFS** book and copies it into the tree. Say +yes: the built system cannot fetch it until it has wget, and wget is looked +up in that book. + +--- + +## 2. Session and settings + +**Configure first.** These become the generated scripts; a wrong value only +shows up hours later inside the chroot. + +```sh +lfs build-system session # the interview: partition, prefixes, locale, + # timezone, hostname, login account, strip +lfs config --target --ask # the BUILT system's prefixes and homes +lfs config # everything else, grouped +``` + +Worth setting before `run`: + +| Key | Default | Why | +|---|---|---| +| `lfs_mount`, `lfs_device` | `/mnt/lfs`, – | the partition; the tool refuses a host path | +| `pkgusr_prefix` / `cfguser_prefix` / `collector_prefix` | `p` / `cfg` / `sysgroup` | account names: `p_gcc`, `cfg_bootscripts`, `sysgroup_glibc`. Hard to change later | +| `locale`, `charmap`, `paper_size`, `timezone` | `en_US.UTF-8`, `UTF-8`, `A4`, – | baked into chapter 9 | +| `hostname`, `network` | `lfs`, `dhcp` | `static` writes interface files; `dhcp` expects dhcpcd from BLFS | +| `main_user` | – | your login. `init-accounts` creates it (group `users`) and asks for root's and its password. Blank = root only, no password | +| `makeflags` | `-j$(nproc)` | | +| `remove_tools` | no | delete `/tools` at the end; it is your way back if chapter 8 needs redoing | +| `bootloader`, `esp` | `none`, – | `refind` + `/boot/efi` generates a rEFInd step that only *adds* an entry | +| `collector_import_file` | – | groups export from a previous build; skips every sharing question | +| `strip` | off | strip debug symbols at the end, as each file's owner | + +Change one: `lfs config <key> <value>`. See all with meaning: `lfs config --long`. + +Stack files and `kernel.conf` are copied into the chroot by `lfs run`, so +edit those now too (sections 5 and 6 say what). + +--- + +## 3. Host: up to the chroot + +```sh +lfs run +``` + +On a fresh tree it prints the book and every session setting once and asks +`[Y/n/e(dit)]`. Then, in order: layout, sync-tools, lfs user, BLFS book, +sources (`get-sources` also downloads what `packagemanager bootstrap` will +need), cross toolchain (chapters 5-6), chroot prepare, script generation. +It ends inside the chroot. + +Follow along or fix a step by hand: + +```sh +lfs build-system next # what is done, what is next +lfs build-system list # every step and its state +lfs build-system snapshot save <name> --run +lfs build-system crosschain --list # the editable chapter 5-6 scripts +``` + +--- + +## 4. Chroot: the LFS system + +```sh +lfs-helper next # where the build stands +lfs-helper build-all # chapters 7-9 +``` + +What happens along the way: + +- Book 7.6 creates `/etc/passwd`; before it nothing can be owned. + `init-ownership` runs right after and adopts everything built so far. +- A step that installs into another package's directory asks which + collector group to use. Answers are saved; `lfs-helper export-groups` + writes them out for the next build (`collector_import_file`). +- The first failure stops the run. Fix, then `build-all` again; it continues + from that step. One step: `lfs-helper build <name> [--phase p] [--force]`. +- At the end the install directories become sticky and `/etc/passwd` is + sorted. It then prints the two things left: the kernel and the bootloader. + +Then hand over to the Python tools, from the **host**: + +```sh +lfs build-system install-tools --run --force +``` + +It copies the tools, the book and the scripts in and writes +`/etc/pkgusr/packagemanager.conf` from your session settings. `--force` +because the Python modules `lfs` and `blfs` need are not there yet; the next +section's `bootstrap` installs them as package users. `packagemanager` +itself works now. + +--- + +## 5. Chroot: BLFS + +### Bootstrap + +```sh +lfs build-system chroot resolv # host: network inside the chroot +packagemanager bootstrap # where you are, five stages +packagemanager bootstrap --run # wget, Python modules, BLFS book, make-ca, certs +packagemanager blfs set-default 13.0-systemd +``` + +Stages 1-2 use tarballs `get-sources` already downloaded, so they work +offline. The rest needs the network `resolv` just gave you. + +### Configure first + +| File | What it holds | +|---|---| +| `/etc/pkgusr/stacks/machine.conf` | this machine's build options (`[mesa] gallium-drivers = radeonsi`, `[iptables] configure_args = --enable-nftables`, `[dbus] systemd = disabled`). Merged into generated scripts when they are born, so set it **before** the stack runs | +| `/etc/pkgusr/stacks/package-fixes.conf` | workarounds the tools ship. Replaced on every `lfs run`; put yours in `machine.conf`, which wins | +| `/etc/pkgusr/stacks/*.stack` | which packages, in order; `avoid` lines; pinned git refs | +| `/etc/pkgusr/desktop-user` | the login that owns the desktop. Set once with `LFS_DESKTOP_USER=<name>`; `sway-session` and `mytools` both read the file | + +Files under `/etc/pkgusr/` are yours once edited. `make install` updates a +copy you never touched, otherwise it lands as `.new` +(`packagemanager stack <name> --take-new` accepts it). + +### The stacks, in order + +```sh +usermod -aG audio,video,input <name> # the main_user from the interview +export LFS_DESKTOP_USER=<name> + +packagemanager stack sway --check-refs # every git ref still exists? +packagemanager stack sway --run # desktop: wlroots, sway, foot, webkit, pipewire, mesa +packagemanager stack network --check-refs +packagemanager stack network --run # wpa_supplicant, iptables-nft, net-tools +packagemanager stack services --run # bootscripts, fcron, acpid, mytools, shell-env +``` + +`sway` includes `check kernel-sway`, which reads the running kernel's config +and fails naming the missing option. In the chroot that is the host kernel; +section 6 builds the real one. + +Useful while a stack runs: + +```sh +packagemanager stack sway --status # how far, what is next +packagemanager stack sway --only <entry> --run +packagemanager errors # what failed, with logs +packagemanager verify # every package, with why +``` + +--- + +## 6. Kernel + +**Configure first:** `/etc/pkgusr/kernel.conf`. It is installed, not +generated, so it can be edited before the first build. + +| Key | Decide | +|---|---| +| `VERSION` | exact, or `latest` (asks kernel.org). `lfs-kernel list` shows what is there | +| `ESP`, `KERNEL_NAME` | *which* kernel this is. `vmlinuz_new` lands beside the working one; `vmlinuz` replaces it. Previous file is kept as `_old` | +| `JOBS` | leave a core free | +| `CONFIG_FROM`, `CONFIG_REFRESH` | `running` + `olddefconfig` for unattended; `oldconfig`/`menuconfig` to decide yourself | +| `FIRMWARE_FROM`, `FIRMWARE_ONLY` | `git` and a short list (`amdgpu amd-ucode amd intel`). Empty means 10 GB | +| `HEADERS`, `V4L2LOOPBACK`, `BOOTLOGO` | as you like | + +```sh +lfs-kernel # build, install, record as p_linux +lfs-kernel verify # image, modules, depmod, headers, firmware +``` + +The kernel is a package: `packagemanager remove linux` takes it away. The +tree of the last kernel known to boot is never deleted, and nothing is +recorded as known-good until it has actually booted. + +--- + +## 7. Bootloader + +Nothing so far has written to any ESP, boot sector or partition table. +Either add an entry to your current bootloader by hand, or: + +```sh +lfs config bootloader refind +lfs config esp /boot/efi +lfs-helper build refind --force # adds beside what boots you today +``` + +Then point the entry at `ESP/KERNEL_NAME` from `kernel.conf`, keeping the +old entry as the way back. + +--- + +## 8. Reboot and check + +```sh +uname -r +lfs-kernel verify # last-good should now say "running now" +dmesg | grep -i firmware # the definitive FIRMWARE_ONLY list +packagemanager verify +``` + +Things a first boot commonly still wants: + +- **wifi:** `/etc/sysconfig/ifconfig.wifi0` with `SERVICE=wpa`, and one + file per SSID in `/etc/sysconfig/wpa_supplicant/`. +- **certificates:** `make-ca -g` weekly, as a cron entry in + `/etc/cron.weekly/` (the systemd book used a timer). +- **desktop:** log in on the active tty as the desktop user, run `sway`. + The seat comes from `/usr/bin/seatd-<user>`, a setuid wrapper `mytools` + built for that one name. If it fails, the launcher checks the wrapper, + `/run/udev` and whether this tty is the active console. + +--- + +## 9. Living with it + +```sh +packagemanager install <pkg> --run # a BLFS package, deps first +packagemanager update <pkg> --run # what the book has newer +packagemanager which-package /usr/bin/foo +packagemanager reclaim <pkg> --run # give files back another package took +packagemanager user create firefox --shared --share-dir --launcher # an app account +lfs-kernel latest # try a newer kernel beside this one +``` + +A new BLFS release: `blfs fetch`, `packagemanager blfs set-default`, +`packagemanager script-status`, then `update`. + +A new LFS release: `lfs-sysvbook make --target 13.1-systemd`, `check`, and +start again from section 2 with `collector_import_file` set. + +--- + +## When something is wrong + +| Symptom | Look at | +|---|---| +| `lfs run` stops | `lfs build-system next` names the step and the command | +| a chroot step fails | `/usr/src/lfs-pkgusr/logs/<step>.log`; `lfs-helper check-toolchain` | +| a stack entry fails | `packagemanager errors`; `--only <entry> --run` to retry one | +| files owned by nobody | `lfs-helper fix-orphans --run`, `packagemanager files_with_broken_id` | +| ownership drifted | `packagemanager verify --fix`; `lfs-helper fix-ownership --run` | +| tools and scripts disagree | `lfs build-system gen-chroot-scripts --run --overwrite` (host) | diff --git a/HANDOFF-history.md b/HANDOFF-history.md new file mode 100644 index 0000000..ac2880f --- /dev/null +++ b/HANDOFF-history.md @@ -0,0 +1,10550 @@ +# lfs-pkgusr — context for a new chat + +Four tools that build and maintain an LFS/BLFS system where **every package +is owned by its own unprivileged user**. + +Attach: the four tools (`lfs`, `lfs-helper`, `packagemanager`, `blfs` -- +`packagemanager_install` was retired in 1.12.0), plus +`test_lfs_crosschain.sh`, `lfs-sanity.sh`, `stacks/` and this file. + +**New here? "Start here" below says what to read and in what order.** + +**ALWAYS TREAT THE CAUSE.** The user's standing instruction, and the rule +this whole project is built on. Shielding a path, special-casing a package +or silencing a message is acceptable only as a stopgap that is written down +HERE as unfinished, with the real cause named. A symptom fixed and called +done is how one bug becomes a species. (Said after 1.12.11 shielded five +paths without fixing why manifests over-claimed -- 1.12.13 fixed the cause.) + + +## Current state + +FOUR tools at **1.14.144** (packagemanager_install is retired), plus +**lfs-phases 1.14.144** (the phase runner every generated script sources, new +in 1.14.0) and **lfs-sysvbook 1.0.0** (its own version line — it is a book +converter, not part of the build engine). Build ids — check these match what +is installed: + + lfs a28626f + lfs-helper b266328 + packagemanager f23fb4c + blfs b4c59f0 + lfs-phases 3eb3e81 + lfs-sysvbook 62ebe64 (1.0.0) + +687 regression tests pass in a bare checkout with the books beside the tool; +the one failure (rc.site) needs the SysV book. **1.14.0 has not built a +system yet** -- the script format changed (lfs v18, blfs v17) and the user is +starting a fresh build on it. + +644 regression tests (counted results), 640 pass in a bare checkout; the 4 +failures need the book HTML findable (beside the tool or in +`$LFS_STORE/books`). **The suite has run CLEAN on the built system**: 607 pass, 0 fail, one +honest SKIP (bootstrap already done), on 1.11.29 installed at builds +209ac78/b0213c4/c3eea26. Only the slice-noise open item below still prints +to stderr; every counted result passes. + +**The bootstrap chain works end to end.** A real system: wget installed offline +from a staged tarball, make-ca fetched over the temporary unverified +connection, 172 certificates in /etc/ssl/certs, the wgetrc weakening removed, +and `wget https://ftp.gnu.org/` verifying. + +**1.11.6 built a complete system**, chapters 1-10, and the reported problem was +a false alarm in the snapshot's own toolchain check. Work has moved to the +booted system: `bootstrap`. The five failures are +environment-only: four need the book HTML beside the tool, one needs +`skel-u_xdg/.bash_profile`. + +**A full 104/104 build has completed on 1.10.1** and verified clean: 66265 +paths owned by the right package, every account prefixed, every home owned by +its user, install directories sealed, `check-toolchain` usable. + +**A full build on 1.9.3 verified clean: 66265 paths owned by the right +package, nothing else outstanding.** The one failure is `the XDG profile is not +shipped`, which looks for `skel-u_xdg/.bash_profile` next to the tool — +environment-only, it passes where the repo is laid out properly. + +**A full 104/104 build has completed** on the 1.8.2 code. Everything since is +either a fix for something that build revealed, or the 1.9.0 cleanup. + +(History: the 1.9.0 rename required a fresh build — account names, homes, +scratch and state all moved. That build happened; nothing above applies to a +tree at the current version.) + +Kernel and bootloader are DONE — the system boots standalone. + +## The user's system, in numbers + +The real machine this history happened on (so a new session can check claims +against it): collector_prefix `nimgnu`, pkgusr_prefix `p`, cfguser_prefix +`cfg`, main_user `n76310`, pkgusr_home `/usr/src/pkgusr`, appuser_home +`/home/shared_user`, SysV init, 93 build accounts, 66230 paths verified owned, +7 collector groups at 90000+ (plus the malformed one below). The user's +systemctl-removal edit lives in +`/usr/src/pkgusr/p_make-ca/install_make-ca-1.16.1` — `--regenerate` would +discard it. + +## Pending on the real system (as of 1.14.94, 2026-09-09) + +The previous version of this section dated from 1.11.x and every item in it +had long been closed -- including the one that matters most for orientation: +**the system already boots standalone; kernel and bootloader exist.** What is +pending now is a Wayland desktop and the services around it, all of which are +BUILT (319 packages, `verify` clean) and none of which has been run. + +Things to check or do, in order of likely payoff: + +1. **Kernel options for what was added since it last booted.** The kernel + that boots today predates sway, iptables-nft and pipewire. Confirm, in its + config: `CONFIG_DRM_<your gpu>` with KMS, `CONFIG_INPUT_EVDEV`, + `CONFIG_NF_TABLES` + `NFT_COMPAT` (iptables-nft is useless without them), + `CONFIG_SND_*` for the sound card. Rebuild only if something is missing. + +2. **The desktop user and the seat.** Create the user, then run the services + stack -- `cfg mytools` builds everything of yours under ONE package user + (p_mytools): /usr/bin/seatd-<user> (-rwsr-x--- root:<user>), sway_start, + wifi, and the `sway` alias: + useradd -m -G audio,video,input,wheel <name> && passwd <name> + LFS_DESKTOP_USER=<name> packagemanager stack services --yes --run + No seatd daemon and no `seat` group: the wrapper is setuid and built for + one user, so the seat exists only while that user asks for it. (elogind + cannot provide one here -- it needs pam_elogind at login, and Linux-PAM + is avoided.) + +3. **First launch, on a tty**: log in as that user and run `sway-session`. + This is where a missing RUNTIME dependency shows; a build cannot catch it. + +4. **iptables really is nft**: `iptables -V` must say `(nf_tables)`. If it + says `(legacy)`, libnftnl did not land before iptables, or the machine.conf + [iptables] section was not present when it built. + +5. **wifi**: the user's script needs `/etc/sysconfig/ifconfig.wifi0` (with + SERVICE=wpa, which `install-service-wpa` provided) and per-SSID files + under `/etc/sysconfig/wpa_supplicant/`. The script's `wpaUp` calls + `ifup $IFNAME wifi0` with IFNAME unset -- `INTERFACE` is defined and + never used; one of them should go. + +6. **make-ca's weekly update** -- the systemd book's page enables a timer, + which the shim skipped. Now that fcron exists, add the 12.4 SysV book's + cron job by hand: + cat > /etc/cron.weekly/update-pki.sh << "EOF" + #!/bin/bash + /usr/sbin/make-ca -g + EOF + chmod 754 /etc/cron.weekly/update-pki.sh + +7. **dbus**: machine.conf carries `[dbus] systemd = disabled`, added AFTER + dbus was built. Either rebuild dbus (`packagemanager install dbus --run + --regenerate`) or accept the systemd-flavoured build; it works under + elogind either way, the difference is a few files nothing reads. + +8. **Stray `.service` in /usr/lib/systemd/user** from at-spi2-core: harmless. + +Tool work deferred, none blocking: + +- Whether the CONFIGURE phase should run as root (1.14.79). The book says + root for every "Configuring" section. Decide at the start of a session. +- `needs=` for the remaining git entries (swaybg, wlroots, seatd, ...): their + dependencies are only discoverable by building; add a line per discovery. +- `--check-refs` cannot see tar URLs; a `have=`-style check for tarballs + (HEAD request, or a sha256 beside the entry) would close the gap libnftnl + exposed. +- The "busy tree" test is flaky (races a `sleep`); use a private marker. +- The rc.site test needs a SysV LFS book to pass; it should say so instead + of failing. +- The whole-book invariant "no empty install phase" should be extended to + build phases, with the known-empty pages listed by name. + +## Start here + +If you are picking this up cold, read in this order: + +1. **The core idea** — one paragraph, and everything else follows from it. +2. **Where everything lives** — the layout changed in 1.9.0. +3. **The ownership epoch** — the single moment the scheme turns on. +4. **One bug, twelve times** — every failure this project has had was the same + shape. Knowing the shape is worth more than knowing the twelve. +5. **Open items** — item 1 is the next job. + +## The core idea + +Each package gets a user that owns the files it installs. + +- Every file traces to the package that installed it. +- Removing a package is removing that user's files. +- A broken build cannot overwrite another package's files. + +Directories several packages install into are shared via **collector groups** +(`<prefix>_<owner>`, e.g. `nimgnu_python`). The install group (`install`, gid +9999) means "anyone may install here" and is only for genuinely shared dirs — +never a package's own tree, and never `/root` or `/home`. + +## The five scripts + +| Script | Runs | Purpose | +|---|---|---| +| `lfs` | host, Python | Drives the build: partition, sources, chapters 5-6, chroot, snapshots | +| `lfs-helper` | inside the chroot, **bash** | Builds each package as its user. Bash because there is no Python in the chroot yet | +| `packagemanager` | built system, Python | Installs/updates packages, application users, config | +| `packagemanager_install` | built system, bash | The install engine `packagemanager` calls | +| `blfs` | built system, Python | Parses the BLFS book, generates install scripts, resolves dependencies | +| `lfs-phases` | everywhere, **bash** | The runner every generated install script sources: fetch, unpack, phases, dispatcher | + + +## Where everything lives + +Two trees under `/usr/src`: the accounts, and what the build knows about them. + +``` +/usr/src/pkgusr/p_gcc a package's account +/usr/src/pkgusr/p_gcc/src where gcc unpacks and builds +/usr/src/cfg/cfg_bootscripts a config step's account +/usr/src/u_firefox an application user +/usr/src/lfs-pkgusr/ what the build knows about all of them +``` + +`SRCROOT` (`/usr/src`) is the parent and stays whole-tree, **deliberately**: +every ownership pass excludes it as one path (`-not -path "$SRCROOT/*"`). +Splitting that exclusion would silently un-exclude half the accounts. + +That is also why the state directory moved here in 1.9.0. It was +`/.lfs-pkgusr` — a hidden directory at the root of the filesystem, sitting +beside `/boot` and `/etc` as if it were part of the system being built. It is +not; it is this toolchain's working notes *about* that system. Under +`/usr/src` the scans exclude one path instead of two. + +The state directory is sorted, so opening it tells you what is in it: + +| | | +|---|---| +| `scripts/` | one shell script per build step, generated from the book | +| `manifests/` | what each package installed (`.files`, `.dirs`) | +| `logs/` | one log per step, plus `verify.log` | +| `progress/` | `steps-built`, `steporder`, `adopted.list`, tree snapshots, the ownership marker | +| `groups/` | collector groups and the grants that created them | +| `config/` | `installdirs.lst`, `env` — settings you may edit | +| `stage/` `steps/` `tmp/` | transient, only during a build | + +Both tools name it once: `STATE` in `lfs-helper`, `PKGUSR_DIR` + +`pkgusr_state()` in `lfs`. They read each other's manifests, so a disagreement +here is a silent one. + +**Definition order matters.** `SRCROOT` must be set before anything derived +from it. Under `set -u` a forward reference is not a subtle bug — the tool +will not print its own version. There is a test on the ordering. + +## The build scratch is the package's + +A package unpacks and builds in `<its home>/src` — not in a shared `/build` +that every package writes into at once. + +The shared scratch was a permanent ownership fight. Every build's touched-file +scan swept it in, so each manifest claimed every other package's sources and +chowned them; the next package chowned them back. A real `verify` reported +3363 wrong-owner and 3348 of them were `/build`: + + /build/bash-5.3.tar.gz is p_shadow want p_acl + /build/bash-5.3.tar.gz is p_zstd want p_acl (after "repairing" it) + +Under the package user's home it is settled by construction: the user creates +the tree so the user owns it, `$SRCROOT` is already excluded from every scan so +it can never reach a manifest, and removing a package removes its build tree. +Steps with no account keep the shared `/build` — they run as root, which owns +everything there anyway. + +`~/src`, not `~/build`: `~/build` is already the hint's build helper script, +symlinked into every package user's home. `pkgusr_skel_links` names the +skeleton once and a test checks the build subdir is not one of them. + +## Names and places + +Three kinds of account share one passwd file, so each carries a prefix. +Without one, a package called `man` or `news` collides with a real system +account and nothing says which accounts belong to the build. + +| Thing | Looks like | Lives in | +|---|---|---| +| package user | `p_gcc` | `/usr/src/pkgusr/p_gcc` | +| config-step user | `cfg_bootscripts` | `/usr/src/cfg/cfg_bootscripts` | +| application user | `u_firefox` | `/usr/src/u_firefox` | +| collector group | `<prefix>_python` | — | + +**One account, one prefix — its own.** + +| Range | Used for | +|---|---| +| 9998 | `lfs` build user *(inside the chroot; the host's is unpinned)* | +| 9999 | `install` group | +| 10000+ | Package users, in build order (binutils first) | +| 90000+ | Collector groups | + +**Prefixing is idempotent, and that is load-bearing.** It is applied to names +a person typed (`gcc`), to directory names read back from disk (`p_gcc`), and +to values that already went through it. Because a second pass cannot produce +`p_p_gcc`, every call site can apply it without first knowing which kind of +name it holds — which is what made 24 call sites a mechanical change instead +of 24 judgement calls. + +**One chokepoint per tool.** Never build a home by hand: + +- `lfs-helper` — `pkgusr_kind()`, `pkgusr_prefix_for()`, `pkgusr_root_for()`, + `pkgusr_home_for()`, `pkgusr_roots()`, `unprefix_pkg_user()`, + `step_stage_dir()` +- `packagemanager` — `pkgusr_kind()`, `pkgusr_name()`, `unprefix_pkgusr()`, + `pkgusr_root()`, `pkgusr_roots()`, `pkgusr_home()` +- `lfs` — `pkgusr_prefix()`, `cfguser_prefix()`, `LAYOUT_DEFAULTS`, `layout()` + +`pkgusr_kind()` is the top of that chain: it decides the kind once and the +prefix and the root are both answered from it, so a name cannot be given one +kind's prefix and placed in another kind's root. + +`lfs-helper pkgusr-home <name>` exposes the rule to shell scripts, so nothing +else re-derives it. `packagemanager_install` and `lfs-completion.bash` go +through a chokepoint now; a test fails if either builds `/usr/src/$something` +by hand. + +**The chokepoint must resolve to the ACCOUNT home, not just the root.** +Callers pass a mixture and always will: `build_pkg` has the step name +(`man-pages`, `util-linux-tmp`), the staging code has the owner +(`p_man-pages`). `pkg_owner_name` maps one to the other — prefix, lowercase, +fold `-tmp`/`-passN` onto the real package — and it is idempotent. Routing by +kind without that mapping shipped once and broke the build at step 9: + + # created package user p_man-pages ... + line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory + +Chapter 7 did not catch it — root steps `mkdir -p` their own directory first, +so it surfaced at the first real package user. + +**A step that is not a package has no home.** `pkgusr_home_for` answers for +accounts; `step_stage_dir` answers for steps, and routes `init-*`, `last-step`, +`refind` and prose `cfg_*` sections to `$STATE/steps/<name>` because they never +get an account. Asking `pkgusr_home_for` for their home is what left +`/usr/src/pkgusr/p_init-dirs` behind. + +Note `SRCROOT` in `lfs-helper` is the **parent** (`/usr/src`), deliberately. +Tree scans exclude it as one path (`-not -path "$SRCROOT/*"`); splitting it +there would silently un-exclude half the accounts from every ownership pass. + + +## Two directories, two jobs + +| Directory | Job | Permissions | +|---|---|---| +| `$LFS/sources` | downloaded tarballs, never written by a build | `root:root 1777` | +| `<package home>/src` | where that package unpacks and builds | owned by its user | +| `$LFS/build` | scratch for steps with no account | `root:root 1777` | + +Book 3.1 settles the ownership — `chmod a+wt`, then +`chown root:root $LFS/sources/*` — and gives the reason: a host uid means +nothing in the built system. Here it is sharper than the book's case. The host +`lfs` account is created by book 4.3's own `useradd`, which pins no uid, so it +lands wherever the host had a gap — 10753 on a Debian host. Package users start +at 10000, so it does not merely show as a number, it **collides with a real +package user** and tarballs report as owned by something unrelated. + +`/sources` and `/build` are in `never_adopt_list` and NOT in +`install_dirs_list`. Being in both made `/sources` `root:install 775`, +dropping the `o+w` that lets the unprivileged build user unpack at all. + +**The symlink farm.** `unpack_pkg` links every downloaded file into +`BUILD_ROOT`, because the book reaches sibling tarballs through `..`: + + cd .. + tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 + +`cd ..` leaves the build subdirectory, so `..` is then `BUILD_ROOT`. Linking +everything rather than a known list means any future book works without a +second thing to keep in sync. + + +## The ownership epoch + +**Before book 7.6 there is no `/etc/passwd`, so ownership is not wrong — it is +meaningless.** No name resolves, not even root's; the prompt in the chroot +reads `I have no name!`. Every ownership question asked before that point gets +a confident wrong answer: + + chown root:install fails on every directory -> "# 0 install directories" + find -nouser matches every file in the tree -> 11802 "orphans", incl. / and /proc + +There is one gate: + +| | | +|---|---| +| `ownership_established()` | is it done? (`progress/ownership-established`) | +| `ownership_possible()` | can it be? (passwd + chown/chgrp/chmod) | +| `establish_ownership()` | do it | +| `_ownership_checkpoint()` | the one call site, after every step | + +`init-ownership` is a **step you can see**, inserted straight after +`init-files` (book 7.6). `lfs-helper` performs it itself — there is no book +script — and marks it done, so `list` and `next` agree with reality: + + 3. init-ownership built + +It does everything in order: install group and install directories, the build +user's leftovers handed back to root, package users for everything already +built, adoption, our own tools, then orphans. Idempotent, and the marker is +durable, so a resumed or restored build crosses it once. +`lfs-helper establish-ownership` runs it by hand. + +Everything that needs a user database refuses without one: `_vfy_orphans`, +`_has_orphaned_files`, `_vfy_build_user_leftovers`, `_vfy_install_dirs`. + +Install directories are set and read back **by number** (`0:$INSTALL_GID`) — +`root` need not be a name. + +## The two chowns that can destroy a tree + +Book 4.3 gives the tree to the build user; book 7.2 gives it back. Both are +recursive, and each is catastrophic on the wrong side of chapter 7. + +- **4.3 on a built tree** hands the finished system to the host's build + account. `_chown_tree_to_lfs` refuses when `_handover_is_safe` is false. + + There were **four doors to that chown** and only one had the guard: + `_chown_tree_to_lfs` (book 4.3), `session --run` re-applying it by hand, + the pre-build ownership check, and `fix-ownership --run`. All four now ask. + A test walks every `chown -R lfs` in the file and fails on any whose + enclosing function does not consult `_handover_is_safe` — the property, not + the four paths, because a fifth door would otherwise be silent too. +- **7.2 on a built tree** strips every package user in one pass. Refused for + the same reason. + +Refusing both left a tree with no way forward, so there is a third, precise +form: `_handover_build_user_only` selects on the **build user's uid**, taking +back exactly what 4.3 took and leaving every package user's files alone. Safe +at any stage. `chroot prepare` runs it instead of refusing, and `chroot enter` +runs it before deciding to stop. + +`chown -h root`, user only — 4.3 chowns the user, so the inverse does too; +taking `root:root` would strip `install` off every shared directory. And `-h`, +because `/bin`, `/lib` and `/sbin` are symlinks. + +`_handover_is_safe` asks the **tree's own passwd** which accounts are package +users (`_pkg_users_in_tree`), never a uid range. Book 4.3's `useradd` pins no +uid, so the host's `lfs` came out **10753** on one machine — inside the package +range — while the chroot's own `lfs` is 9998. Across that boundary only the +number means anything. + +## Saying things + +Four colours, one meaning each, identical in both tools — they print into the +same terminal one after the other, and a log where the same colour means two +things is worse than no colour at all. + +| | | +|---|---| +| green | something finished, and finished correctly | +| yellow | something needs your attention | +| red | something failed | +| dim | detail you can skip: provenance, and what to do next | + +Prose gets none. Colour on every other line stops carrying information. + +`lfs-helper`: `say` (prose) `ok` `warn` (stderr) `note` (stdout) `detail` +`hint` `fail` `die`. `lfs`: `warn` `note` `ok` `fail` `hint`. + +**Use the helper, never the variable.** A colour opened in one `echo` and +closed in another leaves the terminal painted if anything between them writes — +the failure banner was 22 lines inside a single open/close. The only direct use +left is the status column in `list`, where the colour *is* the information. + +## One bug, twelve times + +Every failure this project has had was the same shape: **one decision held in +more than one place, free to disagree.** They are worth reading as one list, +because the next one will look like them. + +| What disagreed | How it showed up | +|---|---| +| Package prefix applied regardless of kind | `p_cfg_bootscripts` — two prefixes, in the config root, claiming to be a package | +| Two wrapper directories | `make_wrappers` wrote `$STATE/wrappers`; the profile said `/usr/lib/pkgusr`, which nothing created | +| Two copy routines for the tools | `install_helper` claimed ownership, `_install_pkgusr_helpers` did not — eight files owned by nobody | +| `chgrp` vs `chown root:install` | `init-pkgusr` set the group, `verify` demanded owner+group; `/usr/share` sat at `lfs:install` | +| Install-dir rule in three places | `cmd_verify` carried inline copies without the guards the helpers had grown | +| **`cmd_verify` defined twice** | bash keeps the last; the tidy one had never run | +| Scratch scanned by two rules | the snapshot scans excluded `/build`, the build's own scan did not — 3348 false findings | +| Build tree named `build` | collided with `~/build`, the hint's helper script | +| Wrapper `chgrp` shadowing root's | reported 46 successes, changed nothing, and `verify --fix` could not repair it either | +| Handover guarded by one inode | `_owner_is_root` checks `$LFS/usr`; the pass it guards covers eight trees | +| uid range used to identify the build user | host `lfs` was 10753, inside the package range | +| Adoption reached from two directions | an up-front pass and a mid-loop "late" pass with its own flag | +| Ownership scans kept their own exclusion lists | build trees moved into the package homes in 1.7.6, the snapshot scans followed and the ownership scans did not — `verify` ran for minutes walking every unpacked source tree | +| Generated scripts had no version stamp | the step order lives in the tree, the tools outside it — a tree generated by an older version went on running `last-step` under a lfs-helper that had removed it | +| Collector export written and read in two places | 1.9.0 sorted the state dir; the writer stayed at the top level, the reader moved into `groups/` | +| `pkgusr-home` was exposed, `owner-name` was not | a script could ask where a package lives but not what its user is called, so `last_build_step.sh` assumed — `chown: invalid user: 'wget:wget'` at step 104 of 105 | +| `add-user` named the account raw, the home prefixed | `add-user wget` gave account `wget` in `/usr/src/pkgusr/p_wget` — every caller but `last_build_step.sh` passed an already-prefixed name, so it never showed | +| Tree snapshot written and read in two places | same move, same shape: `lfs` kept `.snapshot`, `lfs-helper` read `progress/tree-snapshot` — the chapter 6→7 handover snapshot was lost | + +**Half a chokepoint is not a chokepoint.** `pkgusr_home_for` was reachable from +outside as `lfs-helper pkgusr-home`; `pkg_owner_name` was not. So a shell script +could ask where a package lives but had to *guess* what its user is called — +right by accident until `add-user` started applying the prefix, then wrong at +step 104 of a 105-step build. `lfs-helper owner-name <package>` is the other +half, and a test checks the two agree. + +The fix is always the same: **name the decision once, and make everything ask +that one thing.** `pkgusr_kind()`, `build_root_for()`, `never_claim_list()`, +`ownership_possible()`, `pkgusr_skel_links()`, `_real_tool()` all exist for +this reason. + +Two rules follow from it, and both have tests: + +- **No function may be defined twice.** A duplicate is invisible — both parse, + one silently wins. +- **Every `cmd_*` a tool calls must be defined in that tool.** `verify --fix` + called `cmd_fix_orphans`, which never existed, and died mid-repair after + chowning 3473 paths. + +## The collector export + +A previous build's group decisions, so a rebuild does not ask +"which group should share this directory?" for every package again. + + lfs-helper export-groups # write one out + collector_import_file = <path> # session setting on the host + +`lfs` copies it into `groups/collector-groups.import`; `lfs-helper` reads it +from there in `choose_collector_group`, **before** asking. Those two paths are +the whole mechanism, and nothing else connects them — when the state directory +was sorted in 1.9.0 the writer stayed at the top level while the reader moved, +and a rebuild with a perfectly good export asked about `/usr/lib/bfd-plugins` +anyway. Silent, because a question that gets asked looks exactly like a +question that was never answered. There is a test that the two paths match. + +It is re-copied on **every chroot entry**, so editing it on the host reaches +the next build without regenerating the scripts. It only prints when the +contents changed. + +`session` reports what is in the file (`N directory decision(s), M group(s)`) +rather than echoing a path — a file that was configured, present and never read +looked identical to one that was working. + +## One rule for /usr/src + +Everything under `/usr/src` is `root:install 1775`: `/usr/src` itself, the +account roots, and the state directory. + +The account roots were `root:root 0755` — defensible, since only root creates +an account home, but it made one subtree with two rules and a special case to +remember. **The sticky bit is what makes group-writable safe and is not +optional:** without it any member of `install` could delete or rename another +package's home. With it, only the owner can, so group write grants only the +ability to *create*, which nothing unprivileged does (`add-user` is root-only). + +Sharing the mode is not sharing the meaning. The account roots and the state +directory are still in `never_adopt_list`, so no package may ever take one as +its own, and `install_dirs_list` does not contain them. + +**The state directory is in `never_claim_list`.** It lives under `/usr/src`, +which IS an install directory, so `verify` walked its own manifests, scripts +and logs and reported each as a root-owned file no package claims. A clean +104/104 build came back with `448 root-owned file(s) that NO manifest claims` +and all 448 were this directory — including the two temp files the scan had +just written for itself. + +## Reading a sanity report + +Three of its findings used to be about the report, not the tree. All fixed, but +worth knowing the shape: + +- **The sticky bit means opposite things on either side of the finish line.** + During the build it is a bug — package users cannot replace the temporary + system's files. After `seal-install-dirs` it is the point, and its *absence* + is the bug. The report works out `BUILD_FINISHED` from `progress/steporder` + vs `progress/steps-built` and judges accordingly. +- **An empty manifest is not an orphan.** A prose config step — `cfg_clock`, + `cfg_hosts`, `cfg_locale` — writes into files another package owns, so it + records nothing and legitimately has no account. Fourteen of those were + reported on a perfect tree. +- **A step can install under a name that is not its own.** `libstdcpp`'s files + belong to `p_gcc`, because the book builds it out of the gcc tree. When the + name lookup fails, the report now asks the filesystem who owns the files + before calling it a finding. + +`packagemanager_install` has `--version` now, so section 0 can read all five +tools. A tool you cannot ask "which build are you?" is the one that quietly +goes stale. + +## comm needs one collation + +The snapshot diff is `comm -13 <(before) <(after)`. **comm compares line by +line and trusts both sides to be sorted the same way. It cannot detect +otherwise; it returns the wrong answer.** Two things broke that: + +- the two snapshots may have been written under different locales — a package + user's profile sets `LC_ALL=POSIX`, root's shell may not; +- `load_snapshot` rewrites only the lines carrying the host mount prefix, so a + file holding both forms comes back sorted under neither. + +`snapshot`, `snapshot_dirs` and `load_snapshot` all pin `LC_ALL=C sort`, and +`load_snapshot` re-sorts after the rewrite. The visible symptom is a package +installing forty files and being told it installed none. + +**A re-run over its own work is not a silent failure.** The zero-files check +counts what *this run* wrote, which is the wrong question the second time a +package is built: many installers skip a file that is already up to date -- +perl's ExtUtils::Install prints "Installing <path>" for every target and copies +only the stale ones. So the run writes nothing, the snapshot diff sees nothing +new, no mtime beats the stamp, and the count is 0 for the best possible reason. + +`xml-parser` hit exactly that: forty files on disk, every one owned by +`p_xml-parser`, and the step failed as a no-op. `_package_already_owns_files` +asks the tree instead of the counters. A package owning nothing is still a +failure. + +**"installed NO files" now shows its working.** There are exactly three ways to +reach zero and they need different fixes: nothing was written, it was written +somewhere excluded from tracking, or it was written over files another package +owns and dropped by the ownership filter. The message reports the snapshot-diff +count, the touched count, and which of the three it is. + +## A bootloader is opt-in + +`bootloader` defaults to **none**. It used to default to `refind`, on the +reasoning that rEFInd only ever adds files to an already-mounted ESP — true, +and still the safe choice, but it stopped a finished build to ask about a +bootloader on a machine that already had one. The machine booted in order to +build LFS at all. A bootloader is a decision about the whole machine, not a +package. + + lfs config bootloader refind + lfs config esp /boot/efi + lfs build-system gen-chroot-scripts --run --overwrite + +Skipping it is not silent — `gen-chroot-scripts` says so and prints those three +lines, because someone who *does* need a bootloader would otherwise reach a +finished, unbootable system without ever having been offered one. + +GRUB is unchanged and still withheld unless explicitly chosen: its book +instructions run `grub-install /dev/sda`, which writes a disk's boot sector. + +## The group belongs to the tree, and a name is a question (1.12.43) + +1.12.42 created the directories correctly and then put the wrong group on +them: + /usr/share/vim/vimfiles p_cmake nimgnu_cmake + +grant_dir_access names the group after the DIRECTORY'S owner -- which, for a +directory cmake had just been given, is cmake. The group belongs to the +TREE: another package installing into vim's directories will look for +nimgnu_vim, and nimgnu_cmake tells it nothing. + +share_new_dir_with_tree_owner replaces that call: + * the tree owner's group exists -> reuse it, silently + * it does not exist, terminal -> ASK, offering the tree's name first, + the creator's second, or a typed name + * no terminal -> the TREE's name, never the creator's +--yes deliberately does NOT skip this: it means "do not ask me to confirm", +not "pick a name for me" (the user's words, and they are right -- a name is +a decision, and a wrong one is permanent). + +## The portal tail (1.13.1) + +With the status finally honest it read 255, and the list said why: xfce4-*, +lxqt-*, gnome-keyring, ModemManager, ruby, aspell, openssh -- and Systemd, on +a SysV system. None are avoided by name; they are the TAILS of things that +are, and since the automatic prune was reverted (1.12.83, it dropped +graphite2) tails have to be named too. + +They arrive through one edge: the book makes xdg-desktop-portal REQUIRE all +four backends (gtk, wlr, gnome, lxqt), so a single gtk4 recommendation drags +in the whole of xfce and lxqt. Named now, with the rest of the +not-this-desktop list. + + 273 -> 226 packages, 43 named exclusions + webkitgtk, gtk4, mpv all still in. + +REMAINING, and the honest limit: roughly thirty packages in that 226 exist +only to serve something avoided (Qt's double-conversion/jasper/libmng, the +cups tail). Pruning them needs EVERY edge, and `blfs order --anchors` emits +one parent per package. The real fix is to make blfs emit the full edge list +so the plan can prune by reachability properly -- the 1.12.82 attempt failed +precisely because it used the one-parent output. Worth doing; not worth +guessing at again. + +## Stopping the bug is not undoing it (1.13.25) + +mesa failed again with `Unknown option: "needs"` after 1.13.24 stopped adding +-Dneeds= -- because the flag was already IN the script on disk, and the merge +is idempotent, so nothing would ever have taken it out. A fix that prevents +a bad write must also repair what the bad write left behind. + +apply_machine_opts now strips -Dneeds= from an existing meson line and says +so. + +Also: the stale-script box (1.13.21) said + packagemanager install p_mesa --run --yes --regenerate +-- the ACCOUNT, which names no package. unprefix_pkg_user uses +PKGUSR_PREFIX, empty on that system, so the p_ survived; the tools' own +default prefix is stripped as well now (same fix as 1.13.5 for group names). +That is the third place this year where an empty PKGUSR_PREFIX let an +account name leak into somewhere expecting a package name -- worth one sweep +for `unprefix_pkg_user` calls that do not also strip p_. + +## The book already knew the flag (1.13.44) + + vala: configure: error: Package requirements (libgvc >= 2.16) were not met + +libgvc is Graphviz, which this desktop avoids. Rather than guess a flag -- +which is exactly how the libassuan --disable-doc mistake cost three releases +-- the book's own page was read first, and it says: + + "--disable-valadoc: This option is required if Graphviz is not installed." + + [vala] + configure_args = --disable-valadoc + +A useful habit confirmed: when a package fails for a missing OPTIONAL +dependency, the book's Command Explanations usually name the switch that +turns it off. Reading them costs one grep; guessing costs releases. + +WORTH BUILDING NEXT: the failure box could do this grep itself. A +"Package requirements (X) were not met" error plus the book's page for the +failing package is enough to find the sentence with the switch in it. + +## A refusal has to say who is refusing (1.14.5) + +The 1.14.1 chroot fix was reported as still broken: + + root ~ # lfs run + You are inside the LFS chroot, ... + root ~ # packagemanager --version + packagemanager 1.11.26 (build 2daedaf, plan cache v1) + +The detection was fine. The binary being run was three minor versions old -- +`make install` had been landing somewhere other than the copy on $PATH -- and +nothing on screen said which `lfs` was talking or what convinced it. So a +stale install presented as a bug in the current one, and the debugging went +to the wrong place. + +The refusal now ends with its own path, version, build id, and the signal it +used (LFS_IN_CHROOT / the kernel / the fallback). Cheap, and it turns "this +is still broken" into "that is not the binary I installed" at a glance. + +Also in this release: `make install` no longer aborts when +lfs-completion.bash is absent from a checkout -- it skipped nothing and +failed the whole target after the tools were already in place, which is the +same failure shape as above (a run that mostly worked, reported as a failure). + +## A health check must lead with what is wrong (1.14.34) + +A real verify on a working tree: + + broken: 4 not installed: 12 unvalidated: 226 + +The 226 are normal -- "unvalidated" only means the page named no file to +check. The twelve that need action were buried inside them, and three of the +four "broken" were nimgnu_* COLLECTOR GROUPS: groups, not packages, reported +as "has a user but no home dir" because `list_all_users` lists every directory +under the account roots. A false alarm in a health check is worse than no +check -- it teaches people to skip the output. + +Collector groups are filtered out (by the CONFIGURED prefix, not a hardcoded +`nimgnu`), and verify ends with the packages that actually need attention, +their reasons, and the command to rebuild them -- or says nothing does. + +Also worth recording, since it cost the user a wasted pass: the filter +suggested in chat was `grep -vE "installed "`, which also matches "not +installed " and hid exactly the lines that mattered. The tool should not need +a filter to be readable, which is what this change is for. + +## Your packages, under your name (1.14.106) + +Three things from one run. + +**The account.** "the user should be called p_n76310 -- that's a fixed name +for those packages as they come from me." Right, and better than `mytools`: +p_<your user> is where anyone would look. Both the script and the step take +the name from the desktop user (MYTOOLS_NAME overrides). + +**The build line.** It died with `error: 'n76310' undeclared`. The SOURCES +element was DOUBLE-quoted, so the escaped quotes collapsed before eval and gcc +received a bare identifier. Single-quoted now, and the command is the +author's own, verbatim -- the C program stringifies the macro itself, so +copying what works by hand beats reasoning about which form "should" be +right. + +**--only.** Declared in the parser long ago, documented as "repeatable or +comma-separated ... rebuilt even if the cache says done", and NEVER +IMPLEMENTED. It works now: named entries are exempt from the done-cache, +everything else is skipped, and an unknown name is refused with the list of +what the stack has. A stack is long and its entries fail one at a time; +re-running all six to retry one is not an answer. + +An option that exists in --help and does nothing is worse than a missing one: +the help is a promise, and this one had been lying for as long as it existed. + +## Only a kernel that has booted is known to boot (1.14.144) + +The kernel step is complete: 32 checks green, the manifest down from 268815 +to 36539 (modules plus ~12k Documentation files -- no source tree), 199 +Intel wireless blobs in place, v4l2loopback loaded into modules.dep. + +One line in that report was false: + + note last kernel known to boot: 7.2.4 + +7.2.4 has never booted. 7.2.0 is carrying the machine. The old fallback +wrote the version just built into last-good whenever the file was empty, so +the file that exists to name SOMETHING YOU CAN GO BACK TO named the one +thing that had never been tried. A record holding a guess is worse than an +empty one. + +Three cases now, and only the first two write anything: the built version IS +the running one (it demonstrably boots); the running version has a tree here +(that is the fallback); or nothing has been shown to boot, in which case the +tool says so and records nothing. verify explains an empty record rather +than omitting the line, and says when the known-good kernel is the live one. + +The pattern is 1.14.64's again, one level up: a default that fills a gap with +something plausible is indistinguishable from knowledge, and this file is +only useful if its contents were EARNED. + +## A config from one kernel is never complete for another (1.14.143) + +"do we do an 'make old_config' when installing a new version?" It did -- +`make olddefconfig`, one line, undocumented and not selectable. Which is the +right default and the wrong silence: every kernel version adds symbols the +old .config has no answer for, and olddefconfig answers all of them with +UPSTREAM'S defaults without saying so. On a config taken from a +distribution kernel that can quietly turn things on or off that the person +had chosen deliberately. + +CONFIG_REFRESH now selects it, documented with what each choice costs: + + olddefconfig upstream's default for each new symbol, silently + oldconfig ask about each one (needs someone at the terminal) + menuconfig decide everything at once, in the editor + none leave the .config alone and let `make` deal with it + +And it reports the size of what happened -- "47 config line(s) differ from +the config you started from" -- because a number is the difference between +knowing a refresh occurred and knowing it mattered. + +## Three ways to be wrong about a path (1.14.142) + +**The headers.** Copying them AS p_linux-headers cannot work: the source is +inside p_linux's home, which is 0750, so that account cannot even stat it -- +"cp: cannot stat .../usr/include: Permission denied". Opening one account's +home to another would undo the point of having accounts. Root copies (it +reads both sides) and then chowns to p_linux-headers; ownership afterwards is +what makes the files that package's. Same shape as the ESP copies and the +setuid bit. + +**The manifest, still 268815.** The filter was there and did nothing, +because `index($0,h)==1` assumes every line is an absolute path beginning +exactly at the home. Matching the home ANYWHERE in the line cannot be fooled +by a leading ./ or a relative listing, and no installed file has a package +home in its path. Verify now FAILS on an implausible count and PRINTS the +offending lines -- two releases went into reasoning about a filter without +once looking at what it failed to match, which is the actual lesson here. + +**The iwlwifi question.** The firmware was installed all along -- under +intel/iwlwifi/ -- and verify never mentioned it, because its firmware checks +came from CONFIG_EXTRA_FIRMWARE, which lists only what the IMAGE embeds. +Run-time firmware was covered by nothing. verify now checks every +FIRMWARE_ONLY entry and counts the Intel wireless blobs by name. A check +that covers one of two sources looks complete and is not. + +## Say it before it takes a while (1.14.141) + +"after that output nothing happens and i do not know whats going on,.. +please always provide output before we get a long loading process." Fair, +and it applies to four steps: the Documentation copy (~10k small files), the +compile, modules_install, and the manifest scan that walks the filesystem. +Each now announces itself BEFORE it starts, and a test asserts the +announcement comes before the command rather than after it -- an "installing +X" printed afterwards is a log, not a progress message. + +Second, the headers. /usr/include/asm/*.h is owned by **p_linux-headers** +-- the LFS step that installed them -- and /usr/include is sticky, so p_linux +could not replace a single file. Installing a newer kernel's headers is an +UPDATE OF THAT PACKAGE, so the copy runs as that account and its manifest is +refreshed. HEADERS=no leaves /usr/include alone, which is right when the +running system's headers are deliberately older. This is the same boundary +as the firmware directory and the setuid bit: when another package owns the +destination, either become that package or do not write there. + +AND A NEAR MISS WORTH RECORDING: two edits to lfs-kernel in this release +collided and deleted `make $JOBS` outright -- the tool would have installed +modules from a tree it never compiled. Nothing in the suite noticed, because +no test asserted that the build script builds. It does now, including that +the compile precedes modules_install. A test suite for a build tool should +start from "does it build". + +## ...and not through a symlink into it (1.14.140) + + note p_linux records 266623 file(s) + +and the "includes its own home" check added one release earlier said NOTHING +about it -- because the paths are not under the home. modules_install leaves +**/lib/modules/<ver>/build as a symlink to the source tree**, the scan +follows it, and the whole tree comes back recorded as +/lib/modules/7.2.4/build/... . Outside the home, so the home filter never +saw a single one. + +The filter drops anything under /lib/modules/*/build/ or /source/ and KEEPS +the symlink itself, which is a genuine installed file. verify reports that +case separately, in its own words -- "the manifest was scanned THROUGH +/lib/modules/7.2.4/build" -- because "includes its own home" would have been +the wrong explanation for the right symptom. + +Two releases to fix one number, and the reason is worth keeping: 1.14.138 +found A path into the sources and assumed it was THE path. A filter is only +as good as the enumeration behind it, and following a symlink is a second +door into the same room. + +## Verify must check what it installs (1.14.139) + +THE KERNEL STEP IS DONE. 26 checks green on the real system: image, +System.map, config, /lib/modules/7.2.4 with a full depmod index, headers, all +15 embedded firmware blobs, both manifests. + +Which exposed the last gap in verify: it checked everything the tool installs +EXCEPT the out-of-tree module -- the one piece whose build had failed three +times. A report that omits the flakiest thing it covers is worth less than +its length suggests. When V4L2LOOPBACK is on, verify now checks for +v4l2loopback.ko under /lib/modules/<version> and that modules.dep knows it. + +It also reports the manifest's size and FAILS if the account's own home has +reappeared in it -- the 266k-entry bug from 1.14.138 would have been caught by +its own verify, which is the standard the rest of the checks are held to. + +WHAT REMAINS, and none of it is software: point rEFInd at vmlinuz_new (or +rename it over vmlinuz), reboot into 7.2.4, then run sway_start as the +desktop user with /run/udev present. The tools have nothing left to install. + +## The home is not installed files (1.14.138) + + # recorded 266621 file(s) for p_linux + +That is the kernel SOURCE TREE. `list_package` lists everything an account +owns and its home is included -- sources, build trees, the install script. A +manifest is what a package put ON THE SYSTEM: it is what `verify` checks and +what `remove` DELETES. 266k source paths in it make the first meaningless +and the second dangerous. + +write_pkg_list filters the home (and the mail spool) out of the listing. awk +rather than grep, because a filter that removes every line must still exit 0 +-- with grep, a package whose files were all excluded would fail the `&& mv` +and silently keep its old manifest. This affects every package, not just the +kernel; the build's own snapshot tracking always excluded the home, and only +the rescan did not. + +Second bug, same run: `git pull` ran as ROOT with the chown afterwards, so +the next command -- as the package user -- met a repository owned by somebody +else, and git refused with "detected dubious ownership". The directory is +created owned by the account and every git command runs as it, which removes +the class rather than adding safe.directory to work around it. + +A note on the test for this: two assertions failed against correct code +because `grep` over the source matched the COMMENT explaining the fix. Any +check that greps for an anti-pattern has to exclude comment lines, or the +explanation of why something is wrong reads as the wrong thing itself. + +## A report should not build anything (1.14.137) + +`lfs-kernel verify` on the real system: twenty-five checks, one failure. + + FAIL p_linux records its files + +The kernel is completely installed -- image, System.map, config, modules, +depmod, headers, and every one of the 15 embedded firmware blobs. The +manifest was missing because reload-pkg-list ran at the very END, after +v4l2loopback, and that step had failed. So a perfectly installed kernel +reported itself unrecorded. The manifest is written as soon as depmod +finishes -- the files exist at that point -- and again after the headers. + +The other half is what verify printed before the report: it resolved +"latest" over the network, announced the chroot paths it was never going to +write, and would have CREATED the package account if it were missing. A +report that fetches, announces and creates is not a report. Four guards, and +a test that verify shares as little of the build path as possible. + +Worth stating as a rule, because it took the user asking for a validation +command to notice any of this: A CHECK MUST BE CHEAPER AND QUIETER THAN THE +THING IT CHECKS. If it needs the network, the accounts, or the config to be +right, it cannot be trusted to tell you whether those things are right. + +## uname -r is not the version being built (1.14.136) + +THE KERNEL BUILT AND INSTALLED. bzImage, System.map and config are on the +ESP, /lib/modules/7.2.4 is populated. Everything after that failed for one +reason: + + depmod: ERROR: could not open directory /lib/modules/7.2.0 + make[1]: *** /lib/modules/7.2.0/build: No such file or directory + +7.2.0 is the RUNNING kernel -- what this chroot booted. `depmod -a` with no +argument means `uname -r`, and an out-of-tree module's Makefile defaults to +/lib/modules/`uname -r`/build. Both are the wrong tree whenever the version +being built is not the version running, which in a chroot is always. + +depmod takes "$VERSION" now, and v4l2loopback is driven from the tree +directly -- `make -C "$tree" M="$d" clean|modules|modules_install` -- which +sidesteps every such default and works for any out-of-tree module rather than +just this one. A test asserts that uname -r appears only in REPORTING and in +last-good tracking, never in a path. + +The user's original script had this right for its own situation: on a host +building the kernel it is already running, uname -r and $VERSION are the same +string. Every difference between "the machine I am on" and "the machine I am +building for" has cost a release this session. + +## lfs-kernel verify (1.14.135) + +"is the kernel part done? if yes do we have a validation command so we know +everything is installed as expected?" It was not done -- three more embedded +blobs (amd-ucode/*, amd/*) stopped it, and those are in the shipped +FIRMWARE_ONLY now: `amdgpu amd-ucode amd intel`. + +And there was no validation command, which is why every answer to "is it +done" so far has been me reading a transcript. `lfs-kernel verify [version]` +looks at the SYSTEM: + + ok /boot/efi/EFI/NimGnu/vmlinuz_new (what the firmware loads) + ok .../vmlinuz_new-7.2.4 + ok the two are the same image + ok .../System.map-7.2.4 .../config-7.2.4 + ok /lib/modules/7.2.4 modules.dep (depmod has run) + ok headers in /usr/include/linux + ok firmware /lib/firmware/amd-ucode/microcode_amd.bin + ok p_linux records its files p_linux-firmware records its files + note last kernel known to boot: 7.2 + +With no version it takes the most recently installed one from the ESP, not +whatever kernel.org offers today. Two checks are worth their place on their +own: `test -s` rejects a zero-byte image (a truncated copy to a full ESP), +and `cmp` catches $KERNEL_NAME differing from $KERNEL_NAME-$VERSION, which is +what happens when a later build fails halfway through the copies. The +embedded-firmware list is read from the INSTALLED config, so it verifies what +that image actually needs rather than what the current config says. + +## intel/iwlwifi (1.14.134) + + git ls-files | grep -i iwlwifi + LICENSES/LICENCE.iwlwifi_firmware + intel/iwlwifi/iwlwifi-100-5.ucode + ... + +There it is. Current linux-firmware nests the wireless blobs INSIDE intel/, +so `intel` -- which was already in FIRMWARE_ONLY and already installed -- +covers the AX200. The separate `iwlwifi` entry asked for a top-level +directory that does not exist, and that single wrong word cost four releases. + +FIRMWARE_ONLY is `amdgpu intel`, and the config now says where the wireless +firmware lives so the next reader does not have to find out: + + intel EVERYTHING Intel, in one directory: + intel/iwlwifi/ wireless (AX200 and the rest) + intel/ibt-* Bluetooth + +What actually resolved this was 1.14.133's index search -- one command, run +by the user, printing the real paths. Everything before it was me reasoning +about a layout I could not see. When a tool needs a fact about someone +else's data, the first version should go and read it. + +## Ask the index, not the working tree (1.14.133) + +Asked where the iwlwifi firmware might be, with a link and a paste that +arrived empty. The user's own output answered it better: the checkout is +SPARSE, and a sparse checkout REMOVES every path it did not select. So +`iwlwifi` was reported missing, and the near-match search I added to explain +it searched the same pruned directory -- it could only ever find amdgpu and +intel, the two things already installed. + +git's index lists every path the repository has, selected or not. +`git ls-files | grep -i <stem>` is the question that was actually being +asked, and the fixture proves the difference: after +`sparse-checkout set /amdgpu/ /intel/`, `ls` shows two directories and +`ls-files` still shows iwlwifi/iwlwifi-cc-a0-77.ucode. The find() search +stays as a fallback for a plain directory that is not a repository. + +Four releases on this one line, and the mistake was the same every time: +answering from what I believed about linux-firmware's layout instead of +making the tool ask. The tool can now answer it in one run, which is what +should have been built first. + +STILL UNANSWERED, and it needs the machine: whether this linux-firmware calls +the directory `iwlwifi`. The next run prints the real paths. + +## Firmware trouble should not cost you the kernel (1.14.132) + +"are we sure we successfully installed the kernel or is it the step +afterwards?" No -- and that the question had to be asked is the bug. The +firmware install runs BEFORE the build (CONFIG_EXTRA_FIRMWARE needs it) and +exited on failure, so three runs in a row built no kernel at all, and nothing +said so. + +Now the firmware failure only stops the build when the config actually embeds +firmware. Otherwise it warns, builds the kernel, and repeats at the end: +"NOTE: firmware was NOT installed". A missing blob is a run-time problem for +the driver that wants it; it is not a reason to have no kernel. + +And `iwlwifi` does not exist in this linux-firmware either -- the near-match +search from 1.14.131 only ran for PATTERNS, so a plain name got the same +useless "does not exist" the pattern had. Both cases search now, and when +the stem matches nothing at all it lists the top level, because a sparse +checkout that fetched two directories looks identical to a repository that +does not have the file. + +Three releases on one line of config. The pattern in all three: I guessed at +someone else's directory layout instead of making the tool report it. The +first version of a message like this should print what it found, not what it +expected. + +## A pattern that matches nothing should say what is there (1.14.131) + + # installing firmware: amdgpu + # installing firmware: intel + !! nothing matches 'iwlwifi-cc-*' -- check FIRMWARE_ONLY + +amdgpu and intel resolved as directories, so linux-firmware keeps the Intel +wireless blobs in an `iwlwifi/` directory too -- the top-level +iwlwifi-cc-a0-*.ucode layout I assumed in 1.14.128 is an older release's. +The shipped default is `amdgpu intel iwlwifi`; the pattern form stays for a +release that puts them at the top level. + +The change that matters is the failure: a pattern matching nothing now +searches for its stem two levels deep and prints what it found. "nothing +matches iwlwifi-cc-*" sends the reader to go and look; +"iwlwifi/iwlwifi-cc-a0-77.ucode" IS the answer. linux-firmware moves files +between the top level and subdirectories between releases, so this will +happen again to somebody. + +Also: the sparse-checkout on the update path was `|| true`. A failure there +leaves the previous selection in place, and the missing firmware then gets +blamed on the pattern -- which is exactly the wrong place to look. It is +checked now. + +## A config is shell, and a directory has an owner (1.14.130) + + /etc/pkgusr/kernel.conf: line 74: intel: command not found + # installing ALL of linux-firmware (FIRMWARE_ONLY is empty) + cp: cannot create directory '/lib/firmware/3com': Permission denied + ...two hundred more lines + +Two bugs in one release of mine. + +kernel.conf is SOURCED BY THE SHELL, and I shipped +`FIRMWARE_ONLY=amdgpu intel iwlwifi-cc-*` unquoted. The shell ran `intel` as +a command, assigned nothing -- and an empty value meant "install +everything", ten gigabytes of it. Quoted now, and an empty value is refused +with the likely cause named: FIRMWARE_ALL=yes is required to really mean all +of it. A default that turns a typo into ten gigabytes is a bad default. + +And /lib/firmware already existed, owned by root, so every copy by the +package user failed. Firmware has exactly one owner, so root hands the +directory over -- the same boundary as the setuid bit (1.14.109) and the ESP +copies: a package user cannot chown, so anything about IDENTITY is root's, +done explicitly, once. + +The lesson that generalises: a config file that is `.`-sourced is code. Any +value with a space, a wildcard or a quote in it must be quoted, and every +shipped example is the thing people copy. + +## -j14 by default (1.14.129) + +"(by default we just do -j14)". kernel.conf shipped -j16; it ships -j14 now, +with the reason beside it -- a core or two left free keeps the machine usable +while a kernel builds, and the link step is memory-hungry. + +Note for the next session: the attachment with this message came through +empty, so whatever the kernel build printed was not visible. The job count +is the only change here. If `make headers` or another late stage was +failing, that is still open and wants the log. + +## Not all firmware lives in a subdirectory (1.14.128) + +"do we get the required iwl driver for wifi and so?" With +FIRMWARE_ONLY=amdgpu, no. The lsusb shows an Intel AX200, which needs +`iwlwifi-cc-a0-*.ucode` -- and those sit at the TOP LEVEL of linux-firmware, +not in a directory. Its Bluetooth half needs intel/ibt-20-1-3.*. + +So FIRMWARE_ONLY takes file PATTERNS as well as directory names, and the +shipped default is `amdgpu intel iwlwifi-cc-*`. That forced a real change in +the clone: git's sparse-checkout uses CONE mode by default, which matches +directories only, so a pattern would have been silently ignored and the +wireless firmware never fetched. When any entry contains a wildcard the +checkout switches to --no-cone and bare directory names become /dir/, which +is what that mode requires. Installing follows the same split: a glob for +patterns, a name for directories, and a pattern matching nothing is an error. + +Firmware without a driver is useless, so the config check grew the drivers +that need it: CONFIG_IWLWIFI, CONFIG_IWLMVM, CONFIG_BT_HCIBTUSB, +CONFIG_DRM_AMDGPU -- each reported with what it is for rather than as a bare +symbol. + +Where the firmware settings ARE, since the question was asked: in +/etc/pkgusr/kernel.conf. The user's installed copy predates them, so it has +no FIRMWARE_* lines at all -- kernel.conf.new beside it has. + +## Firmware is part of the job (1.14.127) + +"we shouldnt jsut scip the firmware part,- its relevant." Right, and +1.14.126's advice -- clear EXTRA_FIRMWARE and move on -- was the lazy half of +the answer. amdgpu will not bring up a display without its blobs, whether +they are built in or loaded at run time. + +lfs-kernel installs firmware now, before the check that needs it, as ITS OWN +PACKAGE (p_linux-firmware): a separate upstream project with its own release +pace should be one row in `verify`, updatable without touching the kernel. + + FIRMWARE_FROM=git # kernel.org's linux-firmware + FIRMWARE_FROM=copy:/lib/firmware # a tree you already have + FIRMWARE_FROM=none # leave it alone + FIRMWARE_ONLY=amdgpu # which subdirectories; empty = all + +The size problem is why FIRMWARE_ONLY exists: linux-firmware is over ten +gigabytes. With a subset named, the clone is `--filter=blob:none --sparse` +plus `sparse-checkout set`, which fetches those directories and nothing else. +Without it, the tool says how large the download is before starting. + +And the message for a missing built-in blob now names the fix that exists +rather than telling the user to go and install something: the subdirectory of +the missing file is what to add to FIRMWARE_ONLY. + +## Firmware built into the kernel must be on disk (1.14.126) + + make[6]: *** No rule to make target '/lib/firmware/amdgpu/renoir_asd.bin', + needed by '.../builtin/amdgpu/renoir_asd.bin.gen.o'. Stop. + +Not the cancelled build -- a real missing file. CONFIG_EXTRA_FIRMWARE embeds +blobs IN the image, read at build time from CONFIG_EXTRA_FIRMWARE_DIR +(/lib/firmware). The config came from the running distribution kernel +(CONFIG_FROM=running), which lists a dozen; an LFS tree has no +linux-firmware. So the build compiled amdgpu for ten minutes and then died +in drivers/base, naming ONE file and no reason. + +lfs-kernel now reads CONFIG_EXTRA_FIRMWARE straight after olddefconfig, +checks every blob against the configured directory, and lists all of the +missing ones with both ways out: install linux-firmware, or + + ./scripts/config --set-str EXTRA_FIRMWARE '' + +...after which the drivers load the same firmware at RUN time instead, which +is what a normal kernel does anyway. + +The pattern, and it is the one this whole session keeps returning to: the +check belongs where the DECISION is made, not where the consequence appears. +The decision is in the config; the consequence is a link error in a +subdirectory of drivers ten minutes later. + +## Input devices come from udev (1.14.125) + + [wlr] libinput initialization failed, no input devices + [sway] Failed to start backend + +The DRM error is GONE -- wlroots opens the card now, so 1.14.115 was the +right fix -- and libinput found nothing to listen to. "maybe sway would +start if i wouldnt be in a chroot": yes. libinput enumerates through +libudev, which reads /run/udev; a chroot without the host's /run has no +device database at all, and the message names libinput rather than the +chroot. + +sway-session checks /run/udev first and says which of the two causes it is (a +chroot, or udev not running), how to make a chroot work (bind-mount /run and +/dev), and how to prove the rest of the stack works without any input +devices at all (WLR_LIBINPUT_NO_DEVICES=1). It also refuses when started +from a tty that is not the active console, which opens nothing. + +Two things found while writing it. The seat check looked for +/bin/seatd-<user> while mytools installs to /usr/bin -- correct only where +/bin is a symlink, and reporting a missing file that exists where it is not. +It uses `command -v` now. And the sway-session assertions from 1.14.100 had +been LOST when 1.14.106 rewrote that test block: the file had one line of +coverage left. Rewriting a test block is not the same as updating it, and +the difference is invisible until something breaks unnoticed. + +## A config from an older release (1.14.124) + +"i still see that in the config: ... SRC=/usr/src/linux -- maybe with a +reason, idk." No reason: SRC was removed in 1.14.123, and what the user was +reading is the INSTALLED copy from before it, kept because kernel.conf is +never overwritten once edited. The protection is right; the gap was that +nothing noticed a config describing settings that no longer exist. + +lfs-kernel checks now: a leftover SRC is named and ignored, and a config with +no PKG is refused with the two ways out (take the .new, or add PKG=linux). +A file whose MEANING has changed is not the same problem as a file that is +out of date, and the tool has to say which. + +Shipped values are the author's: ESP=/boot/efi/EFI/NimGnu, +KERNEL_NAME=vmlinuz_new -- a new kernel lands beside the working one and +rEFInd still boots vmlinuz until they swap it. + +Note on the shipped config as documentation: the example above the setting +used to read "ESP=/boot/efi/EFI/NimGnu ... # the real one", which is now the +line below it as well -- an example that duplicates the actual value teaches +nothing. It shows the two KERNEL_NAMEs instead, which is the choice being +made. + +## The kernel is a package (1.14.123) + +"in the config i dont realy get the 'source' part. the linux will get +installed like a normal package, so over the pkguser and in its +corresponding home." Correct, and SRC was a location I invented outside the +model -- exactly what the model exists to avoid. + +PKG=linux names the account. Its sources live in +/usr/src/pkgusr/p_linux/src, like every other package's; there is nothing to +configure. The account is created if missing, fix-home runs first (1.14.111), +and reload-pkg-list records the manifest at the end, so `verify`, pkg.lst and +`remove` all work on the kernel. + +The division of labour, which is the interesting part: + * AS THE PACKAGE USER: olddefconfig, make, modules_prepare, + modules_install, headers, the doc copy, v4l2loopback. So + /lib/modules/<version>, /usr/include and /usr/share/doc/linux-<version> + are owned and recorded like any package's files. + * AS ROOT: the tarball fetch, and the copies onto the EFI partition. vfat + carries no ownership and the mount is root's -- the same boundary as the + setuid bit in 1.14.109. + +A detail worth stating because it bit twice: EVERY command that touches the +tree must run as the package user. The first version left the .config write +and olddefconfig as root, which puts root-owned files in the account's home +and breaks the next build -- a test now asserts that no build stage runs at +top level. + +## The refresh list came from a literal (1.14.122) + +"lfs run commmand is not copying that new script i think" It was not. +TOOLCHAIN_SCRIPTS was a hardcoded tuple of five names, lfs-kernel went into +the Makefile and not into it, and `lfs run` refreshed five of six tools -- +the missing one being the tool written minutes earlier. + +Two lists of the same thing is the fault. The Makefile's TOOLS line decides +what `make install` places, so it now decides what the refresh carries; the +literal survives only as a fallback for a tree with no Makefile beside the +tools, and a test asserts the two agree. + +kernel.conf travels with it, from beside the tool or /etc/pkgusr, keeping an +edited copy in the chroot -- refreshing lfs-kernel without its config would +put a command in the chroot that refuses to start. + +Fourth time this session that adding something meant editing a second place +that nobody remembered (the Makefile's stacks loop, the chroot stack refresh, +the .shipped record, this). Anything enumerating the tools or the files +should DERIVE the list. HANDOFF's standing note now reads: when adding a +file to the project, grep for the name of a sibling file first -- every place +it appears is a place the new one belongs. + +## lfs-kernel: its own tool, its own config (1.14.121) + +"it should not get provide by the packagemanager command,- its an extra file +we install id say, so we can easily configure it beforehand." Right: a +config written as a side effect of trying to build is a config you cannot +edit first. + + * **stacks/kernel.sh is now /usr/bin/lfs-kernel**, installed with the other + tools, out of services.stack entirely. `lfs-kernel`, `lfs-kernel list`, + `lfs-kernel 6.12.40`, `lfs-kernel latest`, `lfs-kernel detect-esp`. + Arguments are handled BEFORE the config is read, so --help and + detect-esp work on a system that has no config yet. + * **kernel.conf ships** and `make install` places it, keeping an edited one + and leaving the new one as .new -- the same rule as the stack files. A + missing config is reported, never written silently. + * **ESP + KERNEL_NAME** are the pair that decides which kernel this is, + with the test/real example spelled out in the config: + ESP=/boot/efi/EFI/NimGnu KERNEL_NAME=vmlinuz # the real one + ESP=/boot/efi/EFI/test KERNEL_NAME=vmlinuz-test # a trial + The old file is kept as ${KERNEL_NAME}_old and a copy as + ${KERNEL_NAME}-${VERSION}, so versions sit side by side. + * **THE LAST WORKING SOURCE IS NEVER DELETED.** A kernel that boots is the + only way back from one that does not, and its TREE is what you need to + rebuild or bisect -- not just the bzImage. The running version is + recorded in /etc/pkgusr/kernel.last-good before the new one is installed, + and nothing removes that tree. + +## A kernel step you can read (1.14.120) + +"the kernel.sh script is a bit to trashy to simply configure it,... also id +like to be able to easily get the newest release of kernel.org ... also the +ESP var is not set, idk what it is and how to set it." All three fair. + +Rewritten around them: + + * **The commands are one block**, after a line saying THE COMMANDS. + Everything above it picks a version and checks paths; everything below is + the build, and is meant to be read and edited. + * **VERSION=latest** asks kernel.org's releases.json for the newest + stable. KERNEL_VERSION=7.3 overrides it for one run, KERNEL_LIST=1 + prints what is on offer (mainline/stable/longterm), and the tarball is + fetched from cdn.kernel.org if it is not already in $SRC. Picking a + different kernel when one fails is now one variable. + * **ESP detects itself**: the vfat mount that already contains a vmlinuz, + written into the config with a comment saying what it is and how to check + it (findmnt -t vfat). It was previously an unexplained variable with no + default, which is why it was empty. + +A parsing note worth keeping: releases.json lists "version" BEFORE "moniker", +and pairing the two by POSITION -- `grep | paste - -` -- silently swapped the +columns, so `$1=="stable"` matched nothing and "latest" resolved to the empty +string. The keys are paired by NAME now. Key order in someone else's JSON +is not a thing to depend on. + +## Check the paths, not the chroot (1.14.119) + +"i dont think its a problem to run the commands in chroot,- we still should +be able to get running config and to install modules and kernel file at its +places". Correct, and 1.14.118's refusal was wrong. The chroot usually IS +the tree that boots: /proc is shared so /proc/config.gz is the running +config, /lib/modules is the right tree, and the ESP is right if the EFI +partition is mounted inside. + +So the step no longer judges WHERE it runs; it checks the things that can +actually be wrong. In a chroot it lists the three paths to verify -- $ESP, +/lib/modules, /proc/config.gz -- and continues. + +The real check, which 1.14.118 half-had: the ESP must be ON the EFI +partition. `stat -f` must report vfat/msdos/exfat; a directory of the right +name on the root filesystem is refused, and an unidentifiable filesystem is a +warning. That is the failure worth catching, because every copy SUCCEEDS and +the firmware finds nothing at the next boot -- an error separated from its +cause by a reboot. + +Two releases to arrive at: do not police the environment, verify the specific +thing that would break. The first version confused "unusual" with "wrong". + +## The kernel is built on the booted system (1.14.118) + +Asked how to run the kernel step. The answer needed a guard first, because +the obvious thing -- running it in the chroot like everything else -- is +wrong in a way that looks right: a chroot SHARES /proc, so /proc/config.gz +reads the host's config and the step would happily proceed, while +modules_install writes the chroot's /lib/modules, depmod indexes the wrong +tree, and /boot/efi is not the real ESP. A kernel built for a system that is +not running it. + +kernel.sh compares / with /proc/1/root and refuses in a chroot, naming the +command to run outside. It refuses only on POSITIVE evidence: where +/proc/1/root cannot be read the comparison says nothing, and refusing on "I +could not tell" would block the step on ordinary systems. Both directions +are asserted. + +Second guard: an absent ESP is refused rather than created. `install -d` on +an unmounted mount point makes a directory on the root filesystem, the copy +succeeds, and the firmware never sees the kernel -- a failure that appears +only at the next boot. + + # on the BOOTED system, not in the chroot + packagemanager stack services --run --only kernel # writes the conf, stops + $EDITOR /etc/pkgusr/kernel.conf + packagemanager stack services --run --only kernel # builds + +## Giving the files back (1.14.117) + +The 1.14.116 diagnosis fired exactly as written -- "rounds run: 3", naming +wrap_mode=nodownload -- and the wrap_mode itself never took effect, because +the section lives in the SHIPPED machine.conf and the user's own copy is +protected. meson's summary still said `wrap_mode: nofallback`. Same .new +mechanism as ever; the fix is one section pasted in, or machine.conf.new +taken. + +The question worth answering properly was the user's: "we will sometimes get +that problem that we have to overwrite files,... idk how to handle that". +THE POLICY, in three cases: + + 1. LEGITIMATE OVERLAP -- two packages genuinely install the same path + (icon caches, .desktop dirs). Ownership follows the installer; the + handover is right and automatic. + 2. A BUNDLED COPY -- the package builds another package inside itself. + Stop the bundling (wrap_mode, --force-fallback-for, a configure flag). + Never let the handover "win" here: it silently transfers a whole + package to the wrong account. + 3. A DELIBERATE REPLACEMENT -- the new package is meant to supersede the + old. Remove the old one; two owners for one file is the thing the + model exists to prevent. + +And case 2 needs a way back, which did not exist: `packagemanager reclaim +<package>` reads the victim's manifest, finds every recorded path now owned +by someone else, and returns it. Dry run by default, root required, symlinks +not followed. After it, the aggressor's own manifest still lists those paths +-- `reload-pkg-list <aggressor>` cleans that up. + +## One file per round is a loop, not progress (1.14.116) + +libdisplay-info -- added one release earlier to fix the DRM backend -- went +TWELVE rounds, each re-cloning a subproject and handing over one more file: + + .../locale/fr/.../v4l-utils.mo belongs to p_v4l-utils -- handing it over + ... round 7 ... pt_BR/v4l-utils.mo ... round 8 ... libv4lconvert.so ... + +It builds **v4l-utils as a meson subproject** for its edid-decode comparison +tests, and installs that package's real files over p_v4l-utils's. meson +stops at the FIRST file it may not replace, so its log names only that one, +and each round learns exactly one more: the handover can never catch up. + +Two fixes. The book passes --wrap-mode=nofallback, which blocks fallbacks +for dependency() lookups only -- an explicit subproject() still downloads -- +so machine.conf ships `[libdisplay-info] wrap_mode = nodownload`. And the +retry loop counts single-file handovers from the same package: three in a row +means a bundled copy, it says so, names wrap_mode=nodownload, and stops. + +The general rule, which the ownership machinery had been missing: HANDING +OVER FILES IS FOR PACKAGES THAT LEGITIMATELY OVERLAP. A package installing +another's entire output is not overlapping, it is duplicating, and no amount +of correct ownership transfer makes that right. + +## The DRM backend was never built (1.14.115) + + [wlr] Cannot create DRM backend: disabled at compile-time + [sway] Unable to create backend + +Not the kernel, and not seatd: the log shows seatd opening seat0 and the +client attaching cleanly. wlroots 0.19's DRM backend requires +**libdisplay-info**, which is in the BOOK and was not in the stack. meson +disables drm when it is missing, notes it in a summary line, and builds +happily -- so a complete desktop was installed that cannot open a screen. +libdisplay-info is in the stack now, ahead of wlroots, and the backend list +is EXPLICIT (-Dbackends=drm,libinput,x11): an auto-detected list is a silent +dependency, and this was the silently missing one. + +Also from the user's own commands: + * sway -Dman-pages=enabled -Dswaybar=true -Dswaynag=true + -Dbash-completions=true + * seatd -Dserver=enabled -Dlibseat-seatd=enabled. Which of the two option + sets to use was the user's open question: seatd-user execs + /usr/bin/seatd, so the SERVER must be built even though no daemon runs at + boot, and libseat needs its seatd backend because LIBSEAT_BACKEND=seatd. + * the seatd boot service is confirmed gone from the stack (1.14.100). + +stacks/kernel.sh: the author's kernel procedure as a step, configured by +/etc/pkgusr/kernel.conf (version, ESP, jobs, boot logo URL, v4l2loopback +on/off, where .config comes from). It writes the config and stops on the +first run. It also WARNS about CONFIG_DRM, DRM_FBDEV_EMULATION, INPUT_EVDEV +and SYSVIPC, because a monolithic kernel missing those produces exactly the +error above, one layer further away. + +And add-user's report: `add-user n76310` created p_n76310 and then described +"n76310" -- the human login of the same name, its uid, its /home, its +groups. Every line true of the wrong account. + +STILL OPEN, needs the machine: root cannot log in after boot ("no password +entry"?), and `su` in the chroot says "must be run from a terminal". The +second is normal in a chroot without a controlling tty. The first wants +`ls -l /etc/shadow` (0600 root:root), the root line from `getent shadow root`, +and whether /bin/bash is in /etc/shells. + +## One base, two readings (1.14.114) + +1.14.113 made the symptom legible: after fix-home the passwd entry was right, +and running the step put it straight back. Both were working correctly on +different answers to the same question. + +`lfs` exports LFS_PKGUSR_ROOT from the config key `pkgusr_home`. Its default +is /usr/src/pkgusr; its help text said "the base the homes hang under +(/usr/src)"; and a SECOND key, `pkgusr_subdir`, is "folder under the base for +package homes (pkgusr)". Read one way the base already contains the folder, +read the other it does not -- and packagemanager appends the subdir while +`lfs` exported the bare base. So on a tree configured the second way, every +account's home is /usr/src/p_<name> to `lfs` and /usr/src/pkgusr/p_<name> to +everything else. add_package_user made the husk there; fix-home moved it; +the step moved it back. + +`lfs` composes the value now, appending the subdir unless the base already +ends in it -- the same rule packagemanager has had all along -- and the help +text no longer invites the other reading. + +The root cause is not the code, it is TWO CONFIG KEYS THAT CAN BOTH ANSWER +ONE QUESTION. Any setting whose meaning depends on how another setting was +read will eventually be read both ways by two different programs. Worth a +sweep: pkgusr_home/pkgusr_subdir and cfguser_home/cfguser_subdir are the pair +here, and one of each ought to go. + +## Check that it took (1.14.113) + + lfs-helper fix-home p_n76310 --run + # p_n76310: home -> /usr/src/pkgusr/p_n76310 + ...and the next command: "p_n76310's home is /usr/src/p_n76310" + +Three runs, three reports of success, no change. `usermod -d` returned 0 +while /etc/passwd kept the old home, and the message was printed on the +strength of an exit code that meant nothing. (Why usermod does that here is +still unknown -- a chroot without a working shadow database is the likely +answer -- and it no longer matters: the tool stops believing it.) + +_set_recorded_home reads the home back after writing. If usermod's claim is +false it falls through to the file rewrite, and the file rewrite is verified +too -- `cat` exiting 0 says nothing about whether the line was matched. An +account that is not in passwd now fails instead of succeeding. + +Fifth instance this session of the same rule, and the clearest: 1.14.64 (a +zero exit over a failed log), .87 (validate against what was declared), .101 +(a step that installed nothing), .111 (a manifest in the wrong place), and +now a write nobody read back. THE TOOLS SHOULD CONFIRM THEIR OWN EFFECTS, +always, because every layer below them will report success it has not earned. + +## The skeleton is not data (1.14.112) + + /usr/src/p_n76310 kept: these files differ ... compare and remove by hand + differs: .project, build.conf, .bash_profile, .bashrc, build + # p_n76310: home -> /usr/src/pkgusr/p_n76310 + ...and the next run: "p_n76310's home is /usr/src/p_n76310" + +Those five files are the SKELETON, written by whoever creates an account. A +husk and a real home therefore differ in exactly them, every time, so the +merge always refused, the husk never went away, and fix-home reported "home +-> ..." while the thing that mattered stayed put. The repair could not +succeed on any system, ever; it just took a package user with two homes to +notice. + +_merge_dir_into treats .bashrc, .bash_profile, .project, build, build.conf, +.wget-hsts and .bash_history as disposable in the SOURCE: the destination's +copies are the ones in use. A file that is neither identical nor skeleton is +still kept, and the real home's copy is never overwritten -- both asserted. + +The tell, in hindsight: an error message that lists the same five filenames +every time is not reporting a conflict, it is reporting a rule. + +## Installed, and invisible (1.14.111) + +add-user now creates p_n76310 at /usr/src/pkgusr/p_n76310 -- and pm-install +still worked in /usr/src/p_n76310, the husk add_package_user leaves behind. +Three files installed, "n76310: done", and verify said "not installed": the +manifest was written to a directory nothing reads. + +`lfs-helper fix-home` is the existing repair (1.14.27's helper knows this +exact case, calling it "the husk"). The mytools step runs it BEFORE anything +touches the account, and refuses if the recorded home still is not canonical. +And it checks the outcome: if pkg.lst is missing or empty after a successful +install, it says so and offers reload-pkg-list rather than reporting done. + +Three consecutive releases on the same package (109, 110, 111), each a +different way for "the account" and "where its files live" to disagree. The +package-user model has three separate notions of a home -- the passwd entry, +the canonical path the tools compute, and the husk the hint's helper makes -- +and every command that touches an account has to reconcile them. ONE function +should own that; there are at least four call sites doing it differently. + +## A package name is not an account name (again) (1.14.110) + +The wrapper installed and got its 4750 root:n76310 -- and then: + + cat: /usr/src/pkgusr/p_n76310/pkg.lst: No such file or directory + +The account's home is /usr/src/**p_n76310**, without the pkgusr/ component, +so verify -- which scans /usr/src/pkgusr -- never saw the package at all. + +`packagemanager add-user n76310` asked whether "n76310" exists. It does: it +is the human's LOGIN account. So add-user said "already exists", created +nothing, and the install runner then made p_n76310 itself with the package- +users hint's default home. Exactly 1.14.85's fault in a different command: +there, reload-pkg-list passed a package name where an account name was +wanted; here, add-user tested one. Both now go through pkgusr_name(). + +Second bug in the same run: the seatd wrapper was still installed three +times, because pm-install prefers the script already in the package home +(1.14.21) and that copy was the old one. That preference is right for a book +page someone tweaked and wrong for a script shipped by releases, so the +mytools step refreshes the home copy when it differs. + +TO REPAIR AN EXISTING MISPLACED ACCOUNT: + packagemanager remove n76310 --purge --run + usermod -d /usr/src/pkgusr/p_n76310 p_n76310 # or purge and re-create +then run the step again. + +## A setuid file cannot be replaced by the user who made it (1.14.109) + +wifi and sway_start installed; then + + install: cannot remove '/usr/bin/seatd-n76310': Operation not permitted + /usr/bin/seatd-n76310 -rwsr-x--- root:n76310 + +The wrapper from the previous run is root-owned and setuid -- step 4 makes it +so on purpose -- and /usr/bin is sticky, so only its owner may replace it. +The package user built a new one and could not put it down. A package that +hands a file to root can never update that file again. + +Root is running the step, so root clears the way: if seatd-<user> exists and +is not owned by the package account, remove it before the build, and put the +setuid bit back after. The test asserts the ORDER -- remove, build, chmod -- +because any other arrangement fails on the second run rather than the first. + +Also fixed there: install_pkg installed both the entry's artifact AND the +seatd wrapper on every pass, so the wrapper was written once per source line. +Three writes, and the third is the one that hit the file it no longer owned. +Each entry installs its own artifact now, and a missing build product is an +error rather than a silent skip. + +## It may be in the file waiting beside this one (1.14.108) + + not in this stack: mytools + entries: acpid, blfs-bootscripts, fcron + +True, and useless: `cfg mytools` was in services.stack.new, three lines +above in the same output, and the message did not connect the two. --only +now looks in the .new file for any name it cannot find and prints the exact +command, with the flags the user already typed: + + mytools IS in the newer file beside this one: + /etc/pkgusr/stacks/services.stack.new + Take it and run the same command: + packagemanager stack services --take-new --only mytools + +A name in neither file is still just an error. + +Seven releases now on the same protection, and the shape of every one has +been identical: the tools knew the answer and made the person work it out. +The .new mechanism should be reconsidered wholesale next session -- probably +by having `make install` and the chroot refresh agree on a single source of +truth, so a second copy never appears. + +## A .new with fewer entries is a downgrade (1.14.107) + + took the shipped stack file; yours is .../services.stack.bak + 3 entries now. + NameError: name 'only' is not defined + +Two faults, and the first is worse than it looks. + +`--take-new` took a .new that was SMALLER: 6 entries replaced by 3. The +chroot refresh writes <stack>.new from the HOST's copy when the chroot's +differs (1.14.105), and the host is the stale one here -- `make install` kept +its file because it looked edited. So the loop was: take the good file in +the chroot, refresh brings the host's old one in as .new, take it again, and +the stack shrinks back. --take-new now refuses a shrink, says the host is +probably the stale side, and gives the one-line fix; --force overrides. + +The NameError was mine, from consolidating the two --only implementations in +1.14.106: I removed the `only` variable and left a reference to it, and every +stack run crashed. A duplicate implementation is bad; removing half of one +without reading the rest is worse. The suite could not catch it because it +greps sources rather than running `stack` -- a test that RUNS the command, +even against a two-line fixture, is now in the block beside it. + +## Their name, their macro, one entry at a time (1.14.106) + +Four things from one run of `cfg mytools`. + +**The account is theirs.** "the user should be called p_n76310 (thats a fixed +name for thoose packages as they come from me)". The package is named after +the person, so verify, pkg.lst and `packagemanager remove` all say their name. +MYTOOLS_NAME carries it from the step into the script, so install_last and the +account agree. + +**The macro.** `gcc seatd-user.c -DSEATD_USER="$u"` reaches gcc as +-DSEATD_USER=n76310 -- the rewritten program stringifies it itself. The +script had passed \"n76310\", a different macro, which is what produced +"error: 'n76310' undeclared". The SOURCES entries are single-quoted now: +inside a double-quoted array element the escapes collapse before eval sees +them, which is how the wrong form got there. + +**--yes must reach what a cfg step calls.** `lfs-helper pm-install` asks +"Press enter to continue" unless PM_YES is set, so an unattended stack run +stopped at a prompt with nobody there. cmd_stack exports PM_YES and +LFS_ASSUME_YES when --yes is given. + +**--only.** It had been DECLARED long ago and never implemented -- the help +promised "repeatable or comma-separated" and the flag did nothing. Now it +exempts the named entries from the done-cache AND skips the rest, so +`--only mytools` re-runs one line of a 56-entry stack. I added a second +declaration before noticing the first; argparse refused the duplicate, which +is the only reason I looked. + +## Not everything in that directory is a stack (1.14.105) + + # refreshed in the chroot: .shipped/desktop-user.sh, .shipped/kernel- + sway.sh, ..., services.stack, services.stack.new + +Two faults, both introduced by 1.14.104's directory copy. + +.shipped/ is make install's record of what it last shipped (1.14.98); .new +and .bak are merge leftovers. Bookkeeping does not travel: the refresh now +skips dot-names and those suffixes. + +Worse, services.stack itself was copied over the chroot's. The "never go +backwards" guard compares VERSIONS, and stack files carry none, so the host's +copy always won -- and the user had just taken the 6-entry file inside the +chroot with --take-new. The refresh put the 3-entry host copy back over it, +undoing the previous release from the outside. A stack file that exists and +differs is now the chroot's own, and the incoming one waits as .new, the same +rule `make install` follows. + +Three releases in a row where a fix in one copy of a walk was undone by +another copy elsewhere (Makefile -> lfs refresh -> this). Everything that +moves stack files should share one function; there are currently three, and +the next session should make it one. + +## The same walk, twice (1.14.104) + + ### stack (1/6): cfg mytools + !! missing /etc/pkgusr/stacks/mytools/install_mytools + +1.14.102 fixed exactly this in the Makefile: the loop over stacks/* took +FILES only, so the mytools directory never installed. The chroot refresh in +`lfs` is a second copy of that walk, and it had the same fault -- so the +directory reached the host and stopped there. + +Both handle directories now, and the test asserts both, because the fault was +never in the logic; it was in there being two of it. When a fix is written, +the next question is where else the same walk lives. + +(A test note: the assertion's sed range ended at /continue/, which matched +the first `continue` INSIDE the inner loop, before the chmod it meant to +check -- so a correct implementation failed. A range delimiter has to be a +line that can only mean the end.) + +## Take it, rather than describe it again (1.14.103) + + a newer version of this stack ships beside it: + services.stack.new (6 entries, this file has 3) + ### stack: services.stack (3 entries) + +1.14.99's note worked perfectly and changed nothing: three runs in a row the +user read it, ran the stack, and got the old three entries -- because the +note asked THEM to do work that was mine to do. `--take-new` takes the +shipped file, keeps theirs as .bak, removes the .new, and runs the result: + + took the shipped stack file; yours is .../services.stack.bak + 6 entries now. + +Five releases around one protection (1.14.58, .92, .98, .99, .103). The +sequence is worth remembering as a whole: protect the file, teach verify past +it, teach install to retire it, tell the user it exists, and finally DO the +thing the telling was asking for. Only the last one actually moved the +system forward. A message that asks the reader to run a command they did not +choose to look up is a design that has not finished. + +## One package user for everything of your own (1.14.102) + +"make a setup script for such things so i can bundle them cleanly ... maybe +even under one packageuser so i can easily find all additional +scripts/programs from my side -- we could also add swas, swov, swbr, +browser." + +The right shape is not a shell script that scatters files, which is what +1.14.100's desktop-user step was: it is an ORDINARY PACKAGE-USER INSTALL +SCRIPT. stacks/mytools/install_mytools declares a SOURCES list -- kind, name, +url, destination, build command -- and everything in it becomes files owned +by p_mytools, recorded in its pkg.lst, listed by `verify`, and removable with +`packagemanager remove mytools`. Adding a program is one line. The four sway +helpers are there, commented, ready to move off their own accounts. + +stacks/mytools.sh does only the two things a package user may not: create the +account, and set the setuid bit on the seatd wrapper afterwards. The build +itself goes through `lfs-helper pm-install`, the same runner as every other +package, so tracking, grants and the failure box all apply unchanged. + +The lesson worth carrying: when a user asks for "a setup script", the answer +in a package-user system is usually a PACKAGE, not a script. The machinery +for owning, listing and removing files already exists; a script that installs +things outside it gives up all of that and gains nothing. + +## A step that installed nothing is not done (1.14.101) + +"which step exactly is compiling the seatd-n76310 program" `cfg +desktop-user` -- and it had already run, reported "done (stack cache)", and +installed nothing. + +If the chroot has no network its three fetches all fail; the script printed +"could not fetch" and exited 0; the stack recorded it done; and the cache +makes a recorded entry disappear from every later run. A false "done" is +therefore permanent. Exactly 1.14.64's fault -- a zero exit over a log that +says otherwise -- one layer up, in a config step rather than a build. + +Two fixes. The step checks its own work: /bin/seatd-<user>, ~/bin/sway_start +and /bin/wifi must exist, and the wrapper must be mode 4750 root:<user>, or +it exits 1 naming what is missing and how to install from local copies +(SEATD_USER_URL=file:///...). And the stack runner refuses to record a +git/tar entry whose prog=/have= is still absent after a build that claimed +success -- it says so and leaves the entry to be retried. + +The rule this is the third instance of: a step's exit status is a claim, and +a claim is worth recording only when the artifact it claims is on disk. The +tools already knew how to check -- prog=, have=, the manifest -- and simply +were not asking at the moment the result was written down. + +## The seat comes from a per-user wrapper (1.14.100) + +1.14.94 built a seatd boot service and a `seat` group, reasoning from what the +system had. The user has something better: seatd-user, a setuid program +compiled for ONE username (-DSEATD_USER=\"$u\"), installed -rwsr-x--- root:$u +as /bin/seatd-<user>. No daemon at boot, no group to join, and the seat +exists only while that user asks for it. The boot script is deleted, not +kept "just in case" -- two ways to get a seat is worse than one. + +`cfg desktop-user` compiles and installs the wrapper, puts their sway_start +in ~/bin with the `sway` alias and $HOME/bin on PATH, and installs their wifi +script in /bin. It refuses rather than guesses: LFS_DESKTOP_USER names the +user, /etc/pkgusr/desktop-user remembers it, and an ambiguous /home is an +error listing the candidates. Anything already installed is left alone, and +a failed download says so instead of installing nothing quietly. + +NOT VERIFIED FROM HERE: the three URLs are on git.christianimmanuel.de, which +this sandbox cannot reach. The commands are the user's own, verbatim; the +fetch either works on their machine or says which URL failed. + +sway-session's checks changed with it -- it looked for the `seat` group and +/run/seatd.sock, both of which are now wrong; it looks for +/bin/seatd-$(id -un). + +## A newer version is sitting beside it (1.14.99) + +Twice in a row, after two separate fixes: + + ### stack: services.stack (3 entries) + +while the shipped file had six. 1.14.98 was right and still did not help on +this run: the .shipped record did not exist yet, so the very first install +under the new scheme had to take the KEPT path and leave services.stack.new +-- correct, and completely invisible. The user's "not sure if it did +anything" was, again, exactly right. + +The moment to mention a pending merge is the moment someone RUNS the file. +`packagemanager stack` now says so, with the counts that make it obvious: + + a newer version of this stack ships beside it: + /etc/pkgusr/stacks/services.stack.new (6 entries, this file has 3) + diff /etc/pkgusr/stacks/services.stack /etc/pkgusr/stacks/services.stack.new + ...and take what you want; this run uses YOUR file. + +A note, not a refusal. Fourth repair around one protection, and the last one +needed: 1.14.58 created the .new file, 1.14.92 taught verify to look past it, +1.14.98 taught install to retire it, and this tells the person it exists. +Three of those four were needed only because the first shipped without a way +for anyone to LEARN that the .new file was there. + +## Keep only what the user really edited (1.14.98) + + packagemanager stack services --yes --run + ### stack: services.stack (3 entries) + +Three, while the shipped file had six: seatd-init, bash-completion and +shell-env sat in services.stack.new waiting for a manual merge that nothing +announced. "not sure if it did anything" is the correct reaction. + +1.14.58 was right that a user's edits must survive `make install`, and +1.14.92 taught verify to see past the same protection -- but the INSTALL kept +every file forever, edited or not, so a shipped fix could never arrive on its +own. make install now records what it shipped, in +/etc/pkgusr/stacks/.shipped/: if the installed copy still matches that +record, the user never touched it and it is ours to update; if it differs, it +is theirs and the new one lands as .new, as before. + + services.stack (updated -- you had not edited it) + sway.stack -> KEPT YOURS; new version saved as ... + +That is the third repair to the same protection (1.14.58 made it, 1.14.92 +made verify see past it, this makes install see past it). A rule of the form +"never touch X" needs, from the start, an answer to "then how does a fix to X +ever reach anyone?" + +## The interactive shell is a config step (1.14.97) + +"we missed ls colors and bash autocompletion, and tab-completing a user after +su -- at which step do we integrate that?" + +At a `cfg` step, and the reason is the same one that hid seatd: BLFS's "The +Bash Shell Startup Files" page is CONFIG, not a package. It writes +/etc/profile and /etc/bashrc, and /etc/bashrc is what evals dircolors and +aliases ls --color=auto. No package stack ever runs a config page, so on a +system built entirely from package stacks, ls is never coloured. +bash-completion is not a BLFS package at all -- it comes from upstream, so it +is a git entry (2.18.0, confirmed with git ls-remote and by --check-refs). + +stacks/shell-env.sh writes /etc/dircolors and a BLFS-shaped /etc/bashrc: +coloured ls and grep, the red/green prompt, the bash-completion loader, and + + complete -A user su sudo passwd chage groups id + +which is what makes `su p_<tab>` cycle the package users -- a daily action on +this kind of system, and worth having even without bash-completion installed. +An existing /etc/bashrc is never overwritten (it lands as .new), and +/etc/pkgusr/bashrc gets a line so package users source it too. + +SUITE RUNTIME: the regression suite now exceeds a single execution slot here; +the last full runs reached the final checks with only the known rc.site +environment failure but were killed before printing totals. The three blocks +added in 1.14.94--97 were run in isolation and pass. The suite should be made +splittable (--from N, or per-tool) before it grows further. + +## Snapshots are not the system (1.14.96) + +The find-based scan walked into /snapshots -- btrfs or rsync copies of the +whole tree, every install directory again, once per snapshot. Fixing those +is pointless (a snapshot is read-only, or a record of the past) and +multiplies the scan by the snapshot count. /snapshots and /.snapshots are +pruned by default; `--exclude DIR` (repeatable) prunes anything else; +`--no-default-excludes` puts them back for whoever really wants that. + +## Every directory with group install (1.14.95) + +The user, now on the MAIN system -- older, in daily use, no manifests -- found +the sticky bits stripped from the install directories. `fix-install +--sticky` fixed twenty-two, and the user rightly did not believe that was all: +"wherever the install group is set the dir needs the +w +t". + +The scan asked forall_direntries_from, the C helper from the package-users +hint. On a system that has been running for years that is an older build, +and an older build may stop at mount points or list by user rather than by +group. The question is one find(1) invocation, so fix-install asks find -- +`-type d -group install`, with /proc, /sys, /dev, /run and /tmp pruned -- and +says how many directories carry the group before saying how many are wrong. +Roots can be given to limit the scan. + +Worth noting for the next session: the main system and the new system are +different generations of the same scheme, and the tools must work on both. +Anything that depends on a helper binary from the hint should fall back to +coreutils, because the helper's version is not under our control. + +## The seat has to come from somewhere (1.14.94) + +Asked what to add or change for "a user in the right groups", I read what the +session script actually assumed: "elogind provides the seat". Against the +stack's avoid list that assumption is false. elogind registers a login +session only through pam_elogind, and Linux-PAM is avoided -- so a tty login +creates no logind session, libseat's logind backend finds no seat, and sway +exits before drawing anything. seatd was installed, but blfs-bootscripts has +no script for it and no group existed for its socket. The first launch would +have failed on the first line. + +stacks/seatd-init.sh (a `cfg` step in services.stack) creates group `seat` +and installs an LFS-shaped boot script for seatd at S31, after dbus. +sway-session exports LIBSEAT_BACKEND=seatd so libseat does not try logind +first and fail with a puzzling message, and it refuses early -- naming the +usermod or the init script to run -- if the user is not in `seat` or seatd is +not up. A launcher that dies on "Unable to create backend" tells the user +nothing; one that says "you are not in group seat" tells them everything. + +The general point, and it is the same one as 1.14.75's four shapes of the +systemd mismatch: a comment that says how something works is a claim, and a +claim written before the avoid list was finalised can be quietly false. The +session script's one-line assumption was never tested because nothing had +ever run it. + +## Read every copy; prefer the one whose proof exists (1.14.92) + +"what am i doing wrong?" Nothing. + +The user's stack file is never overwritten (1.14.58), so a corrected +prog=/have= shipped in a newer release sits in sway.stack.new beside it -- and +verify's first-found rule meant the stale copy always won. For three releases +I asked the user to hand-merge a metadata fix so that a REPORT would come out +right. That is friction the tools created, and 1.14.91's "declared by ..." +suffix only described it better. + +verify now reads every copy -- the user's, the shipped one, and any .new +beside them -- and takes the declaration whose proof is actually on disk. +Only when none is satisfied does it report the first, naming its file. The +user's file is still never touched; it simply no longer has to be edited for +the tools to know what a package installed. + +The lesson is about protections: "never overwrite the user's file" was right, +and it had a cost nobody paid attention to -- every fix to that file now +needs a human. A protection that adds a manual step must come with a way for +the tools to see past it, or the step lands on the user every time. + +## Say where the expectation came from (1.14.91) + +"are we caching something? isnt it strange that we're missing that wlroots +dir?" Neither. verify reads /etc/pkgusr/stacks FIRST, and the user's +sway.stack still said have=/usr/include/wlr; the 1.14.90 fix was sitting in +sway.stack.new beside it, because a user's stack file is never overwritten +(1.14.58) -- a protection that, by design, makes a shipped fix wait for a +human to take it. + +"missing: dir /usr/include/wlr" sent the user to look for a directory. The +useful fact was WHICH FILE said to expect one. The line now ends +"(declared by /etc/pkgusr/stacks/sway.stack)", which points at the thing to +edit rather than the thing to search for. + +Two protections interacted here -- never overwrite the user's stack, and read +the user's stack first -- and the combination is right; it only needed the +report to name its source. When a check reads configuration from more than +one place, its complaint should say which place it believed. + +## The two that were left were real (1.14.90) + +After 1.14.89: 317 installed, 2 unvalidated -- and both were findings, which +is what the previous release promised. + + wlroots missing: dir /usr/include/wlr + minibrowser its script names no verifiable target + +wlroots 0.19 puts its headers under /usr/include/wlroots-0.19/, so the stack's +have= had been wrong for the version it pins since the day it was written; the +.pc file is the stable proof. And from the user: "minibrowser is not called +like that -- the git is called Browser, the projects are browser-mini and +browser-big". prog= now names browser-mini. The ENTRY keeps its old name on +purpose: p_minibrowser is the account already on disk, and renaming the entry +would orphan it. + +So the session's last verify is the first one whose two remaining lines were +both true, both specific, and both fixable in one line each. That is what a +verification report is for. + +STATE AT SESSION END: 319 packages, all validated once the user takes the two +stack lines. sway, foot, seatd, elogind, dbus, fcron, acpid, iptables-nft, +wpa_supplicant all built and their boot scripts in rc3.d. What remains is +not software: a kernel, fstab, a bootloader entry, a root password, and the +first login. + +## A stack entry's own declaration wins over a stale record (1.14.89) + +The user asked, fairly, why verify had taken four rounds. Three stacked bugs +in one column, fixed one at a time as the output revealed each -- and the +last residue was twelve packages whose install_last predates the fix that +carries have= into the script. That file is not rewritten, so they would have +stayed "unvalidated" until each was next rebuilt. + +The stack file itself still says what proves each entry installed. verify +now reads every stack (installed and shipped, once per run) and, for an entry +it recognises, uses the stack's prog=/have= in place of whatever the old +record said. A self-named program in a stale record is dropped in favour of +it; a package no stack knows is untouched. No history rewritten, and git-entry +validation is permanently stronger: libsrtp is checked against its .pc file, +wlroots against /usr/include/wlr, dejavu-fonts against its font directory. + +The 1.14.88 test used `libsrtp` as its fixture name and immediately failed, +because the real stack now knows libsrtp and correctly asked for a .pc file +the sandbox does not have. Fixture names must not collide with real entries +when the code under test reads real configuration -- renamed to +`libstaledemo`. + +VERIFY'S METHOD, stated once for the record, since it was questioned: + 1. account and home exist + 2. every path in pkg.lst is on disk <- the strong check + 3. installed_* targets, from the book page, the script, or the stack + 4. validate_cmd, if the page has one +"installed" means 2 passed and 3/4 either passed or had nothing to say; +"unvalidated" means 2 passed and 3 named something absent -- which after this +release should be rare and real. + +## A target that was never a real target (1.14.88) + +"may it be that its just cached?" No, and the question was a fair one: the +count did not move after 1.14.87. + +install_last is the record of the script AS IT LAST RAN. Those twelve entries +were built before the fix, so their records still say +installed_program=<the entry's own name>, and they will until each is next +built. Rewriting that file would be a lie about history -- it is the one file +that states what actually ran -- so verify has to cope instead: when the ONLY +thing missing is a program named exactly like the package, the script named no +real check, and it says so rather than claiming a perfectly installed library +is missing a binary that never existed. + +A different absent program is still reported. Two missing things are still +listed. Both counter-cases are in the suite. + +Also, for the record: the command I suggested for regenerating those scripts +-- `packagemanager install --regenerate-scripts` -- is not an option, and the +`2>/dev/null` I put after it hid argparse saying so. 1.14.80 added a test for +exactly this class one release earlier; it covers what the TOOLS print, not +what I type in a message, and there is no test for the latter. The only +defence is to run a command before recommending it. + +## Validate against what the entry says it installs (1.14.87) + +1.14.86 took the count from 299 to 307 installed, and twelve stayed +unvalidated: the libraries and the data packages. `_stack_git_script` +defaulted `prog=` to the ENTRY'S OWN NAME, so verify hunted for programs +called `libsrtp`, `tllist`, `dejavu-fonts`. None of those is a binary. + +But every one of them carries `have=` -- the path the stack already uses to +decide whether the entry is done. That is precisely the proof verify wants, +and it was being thrown away when the script was generated. `prog=` no longer +defaults to the name, `have=` becomes the validation target, and the directory +check uses os.path.exists rather than isdir because a .pc file or a header is +the usual proof. + +A test asserts that every git and tar entry in every shipped stack declares +one or the other: an entry that declares neither can never be confirmed, and +would sit in the unvalidated column forever without anyone knowing why. + +Three releases (86, 87, and 85 before them) to make `verify` tell the truth +about twenty packages. Each was small; what they had in common is that the +wrong output was ALREADY THERE, in every run, and had been read past for +weeks. + +## Greedy, and the line has a comment on it (1.14.86) + +Every git and tar package, in every `verify` run, all session: + + sway unvalidated missing: program ), program #, program e.g., + program ('gst-launch-1.0 ... + +The stack's generated scripts document their own metadata: + + installed_programs=() # e.g. ('gst-launch-1.0' 'gst-inspect-1.0') + installed_program="sway" # e.g. "ffmpeg" (checked on PATH) + +and `_parse_bash_vars` matched `\((.*)\)` -- greedy, from the first "(" to +the LAST ")", which is inside the comment. The quoted scalars did the same. +Twenty packages could not be validated because of two greedy regexes, and the +nonsense they produced was visible in every run I read this session without my +chasing it. + +Non-greedy, with a trailing comment allowed and ignored. sway now validates +by checking `sway` on PATH, which is what its metadata said all along. + +Worth naming: this was in plain sight for eighty releases. It looked like +noise -- an odd line in a column of odd lines -- and noise gets read past. A +report with a persistently ugly corner trains its reader to skip that corner. + +## The account name, not the package name (1.14.85) + + $ packagemanager reload-pkg-list mpv + mpv: pkg.lst regeneration started (disowned background task). + $ pgrep packagemanager # nothing + -rw-r--r-- 1 root root 0 pkg.lst.new + +"mpv" went straight through. `pkgusr_home()` accepts either name so the home +resolved -- but `_can_su("mpv")` did not, because the account is p_mpv, so it +took the ROOT branch and ran `list_package mpv`, which found no such user, +produced nothing, and left an empty root-owned pkg.lst.new. `write_pkg_list` +normalises with pkgusr_name() now, as every other entry point does. + +The second half is worse than the first: the command printed "regeneration +started" for a task that had already died. It is asked for by a person, so it +runs in the FOREGROUND by default and reports the result -- "1234 file(s) +recorded", or "no files found -- it owns nothing on disk", which is the answer +the user was actually looking for. --background keeps the old behaviour. + +A message that reports an INTENTION rather than an OUTCOME cannot be wrong, +which is exactly what makes it useless. This is the same shape as 1.14.64's +zero exit over a failed log: both said the encouraging thing without checking. + +## Highlight the words the answer depends on (1.14.84) + +The collector-group prompt is eight lines of prose containing two names, and +the names are the only thing the reader chooses between. C_B -- bold, added +long ago with the comment "for the words in a question that the ANSWER depends +on" -- was never applied to them. It is now, and C_B is cleared for a +non-terminal like every other colour, which it had also been missing. + +Small, and worth doing because this prompt appears many times in one stack +run: services.stack alone asked it for /lib/services, and every package that +installs beside another asks it again. + +## Fix them together, not one round each (1.14.83) + +fcron reports ONE file at a time. /etc/fcron.conf must be owned by +root:fcron; fixed, re-run, and it said /etc/fcron.allow: Permission denied; +/etc/fcron.deny would have been the third round. Each round is a full +configure phase and a fresh reading of the same failure. + +The package creates a system group for itself -- it says so in its own script +-- and every config file it installed under /etc wants root:<that group>. The +box takes the file list from the MANIFEST rather than from the log, skips the +ones already correct, and prints one command covering the rest: + + chown root:fcron /etc/fcron.allow /etc/fcron.conf /etc/fcron.deny + chmod 640 ... + It reports ONE file at a time, so fix them together or you will be back + here for the next one. + +The general lesson, and it applies well beyond fcron: when a program validates +its environment one item at a time, a diagnosis that only repeats what the +program said inherits its pace. The tools know the whole set -- that is what +the manifest is for -- and should say all of it at once. + +Also visible in this run: 1.14.64's swallowed-failure check fired for real. +The phase exited 0 while its log carried "Permission denied", and the run was +correctly turned into a failure instead of reporting fcron as configured. + +## Write down what was refused (1.14.82) + +fcron's configure, running as root, failed with + + ERROR could not change egid to 22: Operation not permitted + +As ROOT. That makes no sense until you know `fcrontab` is installed setgid +`fcron`, and the chmod wrapper refused that bit at install time -- correctly, +a package user must not set setuid/setgid -- and then never mentioned it +again. + +The refusal is recorded now, in the package's home, and reported after the +phase by root, who is the only one who can act on it: + + This package asked for setuid/setgid bits, which a package user cannot + set... Apply as root, if you want the package to work as upstream intends: + chmod 2755 /usr/bin/fcrontab + +The file is cleared once reported, so later builds do not repeat requests that +have been dealt with. + +The principle, which is the third form of the same thing this evening +(1.14.78 accounts, 1.14.81 ownership, now set-id bits): a package-user system +refuses certain operations by design, and every refusal it makes silently +becomes a mystery hours later in a different phase. Refusing is right; +refusing WITHOUT A RECORD is what cost the time. + +## A package that checks its own file ownership (1.14.81) + +fcron's configure, now running as root, got further and stopped on + + ERROR Conf file (/etc/fcron.conf) must be owned by root:fcron + ERROR Could not chdir to /var/spool/fcron: Permission denied + +A package-user install cannot satisfy that. Everything it writes belongs to +p_fcron, and the chown wrapper skips the package's own chown deliberately -- +that skipping IS the model. For a daemon that validates ownership before it +will start, the last step belongs to root. + +Not a bug in the model, and not something to work around: replaying the +skipped chowns wholesale would hand half of /usr/share/doc to root and break +the per-user tracking that the whole design rests on. What was missing is the +telling. The box reads the package's own words and prints the two commands +they imply: + + chown root:fcron /etc/fcron.conf + chown -R fcron:fcron /var/spool/fcron + +A worthwhile distinction for the next session: a package-user system can own +FILES but not IDENTITY. Anything a daemon checks about itself -- who owns its +config, which account its spool belongs to -- is outside what the model can +provide, and the honest answer is a named root command, not a cleverer +wrapper. + +## The commands we print must be commands (1.14.80) + + $ lfs-helper build fcron --phase configure --force --as-root + !! unknown option: --as-root + +1.14.79 printed that advice one release earlier, confidently, having checked +that `--as-root` existed -- on cmd_run_script, one function away. cmd_build +never took it. And once it did, `local as_root=0` further down cmd_build +shadowed the flag, so it parsed cleanly and did nothing: two ways to be wrong +about a single word. + +Both fixed. But the fix that matters is the test: every long option the +failure report prints after "lfs-helper build" must appear in cmd_build's +option parser. It reads both out of the source and compares them, so the next +suggested flag that does not exist fails the suite instead of the user's +evening. + +This is the sixth time this session that a message named something that was +not there -- a path, a package, a directory, a command, a flag. The pattern +is always the same: the message was written from what the author believed, and +nothing checked it against the code. Where the thing named is machine- +checkable -- and a flag, a path and a package name all are -- the check is +cheap and belongs in the suite. + +## A configuration step that needs root (1.14.79) + +1.14.78 worked: fcron's system account was created, the build finished, 74 +files installed. Then the configure phase: + + must be privileged to use -u + +from `fcrontab -u systab`. BLFS's "Configuring" sections are written for the +root user -- they are system configuration, not part of installing a package +-- and no directory grant can help, because the command refuses to run +unprivileged whatever it can write. + +`--as-root` has existed all along for exactly this. The box now recognises a +privilege error IN THE CONFIGURE PHASE and prints + + lfs-helper build fcron --phase configure --force --as-root + +It does not fire for a build failure with the same words; that counter-case is +in the suite. + +Left open deliberately: whether the configure phase should simply RUN as root. +The book says root for every "Configuring" section, so it is arguably correct +-- but files it creates would then be root-owned rather than the package's, +and the tracking assumes otherwise. That is a design change to make +deliberately at the start of a session, not at the end of one. + +## A service account is root's job (1.14.78) + +fcron compiled completely and then died in its install: + + groupadd: cannot lock /etc/group; try again later + Group "fcron" does not exist : please create it + +Its page's own commands are `groupadd -g 22 fcron && useradd ... fcron`, and +the install phase runs as p_fcron, which cannot write /etc/group. Every page +needing a system account -- fcron, avahi, sshd, postfix -- is in this +position, and the package-user model has no answer to it: the account is +system state, not a file the package owns. + +cmd_build now reads those lines out of the script and runs them as ROOT before +the phase, skipping any account that already exists. The book's commands stay +in the script, where a reader expects them, and `groupadd`/`useradd` wrappers +turn the package user's own attempt into a no-op when the account is there -- +and, when it is not, say plainly that a package user cannot create one. + +Parsing note: the account NAME is the LAST argument. Stripping "-x value" +pairs by hand tripped over the quoted comment in +`useradd -c "Fcron User"` and produced the name `User" fcron`. + +## Boot scripts are a package too (1.14.77) + +"why dont i have an fcron startup script -- did we miss it during the lfs +install?" No: /etc/init.d holds LFS's own scripts and nothing else, and every +BLFS service script lives in blfs-bootscripts, a separate tarball that is not +a BLFS package. That is precisely why a package page's `make +install-<service>` is commented out by the generator (1.14.20) -- the target +is in that tarball, not in the package being built. + +stacks/services.stack: acpid and fcron (the cron daemon make-ca's weekly +update needs), then blfs-bootscripts 20250225 -- the FINAL SysV release -- +installing only the scripts for services this system has: dbus, alsa, +bluetooth, acpid, fcron, iptables, random, service-wpa. The packages come +first; `have=/etc/rc.d/init.d/dbus` keeps it idempotent. + +And the parser change that made it readable: a stack entry may now CONTINUE +across lines with a trailing backslash. Every entry had to be one line, which +is why sway.stack's are 300 characters wide and nobody can see the build= +command inside them. A dangling backslash at end of file is an error, not a +silent drop, and the line number reported for a bad entry is still the line it +started on. + +Note what the user found by reading `ls /etc/init.d`: dbus has no boot script +either. On a system whose desktop, seat manager and half its packages talk +over dbus, that would have been a puzzling first boot. + +## The pattern has to be a real path (1.14.76) + +The network stack finished -- six of six -- and signed off with + + results: packagemanager errors logs: /usr/src/p_<pkg>/log + +The hint was assembled by hand, `os.path.join(BASE_DIR, +pkgusr_name('<pkg>'), 'log')`, and was wrong twice over: the placeholder is +never filled in, and the path omits the `pkgusr/` component every package home +has. Anyone following it found nothing -- and until 1.14.71 the logs were not +in the right directory either, so the message was doubly useless. + +`_pkg_log_dir` is the function that knows. Both summary lines use it now, and +a test asserts the placeholder form differs from a real package's path only in +the name. + +Third message this session that named a location which did not exist (1.14.71 +the logs, 1.14.43 the package, here the directory). When a message contains a +path, it should come from the code that builds that path, never from a format +string that happens to look like it. + +## A systemd unit on a system with no systemd (1.14.75) + +wpa_supplicant installed its binaries and its man pages and then died on + + install: target '/usr/lib/systemd/system/': No such file or directory + +The systemd book's page ends by installing a unit file, and on a SysV system +that directory does not exist. Exactly the situation the systemctl shim was +written for -- the book assumes an init this machine does not run -- but a +unit file arrives as a file copy, not a systemctl call, so the shim never saw +it. + +The install WRAPPER is the right place: a destination under /usr/lib/systemd +or /lib/systemd, on a machine with neither the directory nor systemctl, is a +visible skip and a success. Creating the directory instead would have worked +and left a unit nothing will ever read. + +A host that does run systemd still installs its units -- that counter-case is +in the suite beside the SysV one. + +The pattern to remember for the next session: the systemd/SysV mismatch shows +up in four different shapes now -- systemctl calls (shimmed), configure flags +(1.14.51's survey), unit-directory installs (here), and journald headers +(1.14.60). Each needed its own catch, in the layer where it appears. + +## An --enable must beat the book's --disable (1.14.74) + +The user pasted netfilter.org's own project page, whose news column lists +**libnftnl 1.3.2** (2026-08-31) as current. That is the project stating its +version, not a guess, so network.stack now carries the tar entry -- placed +above iptables, with a line saying where the version came from and to check +the .sha256 before trusting it. + +Then the flag that makes it matter did nothing. The book configures +`--disable-nftables`; `configure_args` PREPENDED `--enable-nftables`: + + ./configure --enable-nftables --prefix=/usr \ + --disable-nftables ... + +and autoconf takes the LAST occurrence. Same class as 1.14.60 -- an option +that reaches the command and is then ignored is worse than one that never +arrives, because the file says it is set. + +configure_args now removes the opposite form of each flag it adds +(enable/disable, with/without). Line by line: deleting just the token left + + ./configure --enable-nftables --prefix=/usr \ \ + +a backslash-escaped space, which is an argument. An unrelated +`--disable-static` is untouched, and the result is checked with `bash -n`. + +The shipped stacks/machine.conf carries the [iptables] section; a user's own +copy is never overwritten (1.14.58), so it arrives as machine.conf.new. + +## The nft path, written down but not guessed (1.14.73) + +The user wants iptables-nft (legacy is on its way out) and does not want iw. +iw is gone from network.stack; the nft path is documented rather than shipped, +because libnftnl is not a BLFS package and no mirror of it is reachable from +here to check a version against. + +The template in the stack file gives the exact `tar` entry shape, says it must +go ABOVE iptables (order in the file is build order), gives the machine.conf +flag, and says plainly what `--check-refs` does NOT cover: it asks git +remotes, so a wrong tarball version fails at fetch time rather than before the +build. + +Writing down what cannot be verified, marked as unverified, is the honest +middle between guessing a version and leaving the user with nothing. Three +bad pins this session (usrsctp, foot, needs=pcre2) all came from filling in a +value that looked right. + +Still missing for a bootable system, for the next session: a kernel with the +right DRM driver and evdev, a bootloader entry, /etc/fstab, a root password, +and the make-ca cron job that 1.14.51's survey turned up. + +## A network stack, and every entry checked (1.14.72) + +"is it possible that we miss wpa_supplicant?" Yes. The sway stack builds a +desktop; nothing in it brings up a network, and the user's own wifi script +needs wpa_cli, wpa_supplicant and ifup -- none of which existed. + +stacks/network.stack: libnl, wpa_supplicant, iw, wireless_tools, libmnl, +iptables, net-tools. Every one verified against the book before shipping. + +What it deliberately does NOT ship: an iptables-nft path. The book's iptables +page configures `--disable-nftables`, and the nft backend needs libnftnl, +which BLFS does not carry. No mirror of it was reachable from here, so +rather than guess a version -- the exact mistake of `needs=pcre2` (1.14.66) +and the two rotted refs -- the stack carries a comment saying what to add and +warning not to take a version on trust. + +And the check that generalises it: EVERY book entry in EVERY shipped stack +must name a package the book has, `needs=` included. That test immediately +found `elogind` in sway.stack -- which turned out to be legitimate, a `book` +entry carrying url= as a git fallback, because the systemd book has no +elogind page. So the rule is: an entry with no fallback must resolve. A new +check earning its keep by finding something, and then being narrowed by what +it found, is the good case. + +## The log belongs beside the package (1.14.71) + +"shouldnt we log everything within the pkgusers home log dir?" + +Every failure box already claims we do -- it ends with "logs: +/usr/src/pkgusr/p_<name>/log" -- and for a modern script that directory held +one empty .diff. The phase log went to the central $LOGS/<name>-<phase>.log +and nowhere else, and that copy is overwritten by the next run of the same +package. The history someone wants three days later was never kept where they +were told to look. (The OLD-style path did write into the home; the +dispatcher path, which is everything now, did not.) + +run_phase_as copies the log into the owner's home, timestamped so runs +accumulate, owned by the package user, keeping the last ten -- a webkitgtk log +is megabytes and a package that fails twenty times should not fill the disk. + +A message that names a location is a promise about that location. This one +had been wrong since the 1.14.0 refactor moved every install onto the +dispatcher path. + +## Retry the phase that failed, not the whole build (1.14.70) + +"is it possible that if we fail the install part we recompile everything?" +Yes, and it had been doing so all along. + +foot needed seven rounds to collect its directory grants -- /etc/xdg, then +/usr/share/zsh/site-functions, then /usr/share/fish/vendor_completions.d, +because meson reports one unwritable path at a time -- and EVERY round +re-cloned the repository, re-ran meson and recompiled all 150 targets, to redo +an install that had failed on one completion file. The grant-and-retry loop +passed `$phase`, which was `all`. + +Granting a directory cannot invalidate a compile. When an `all` run fails in +install or configure, the retry runs THAT phase; the sources and build tree +are untouched, which is precisely why a grant was all that was missing. An +early-phase failure still retries everything, and a retried install is still +followed by the configure step that `all` owes. + +The first version grepped the log inside cmd_build for the failed phase -- +and an existing test refused it: "cmd_build still infers finished phases from +the log itself", from the release that moved phase bookkeeping into +run_phase_as. The rule was right and the fix was better for obeying it: +run_phase_as already computes that value, and now publishes it as +LAST_FAILED_PHASE. A test that pushes back on a new change for an old reason +is doing its job. + +## An entry with no ref still has a url (1.14.69) + +The stack came back clean -- every pinned ref verified. And four entries had +not been contacted at all: the ones tracking a default branch, reported as +"no ref= (tracks the default branch)" and skipped. + +"No ref to check" is not "nothing to check". A repository that has been +renamed, made private or moved fails exactly like a bad ref -- at clone time, +after everything before it has built. Those entries are asked for their +branches now: reachable, unreachable (counted, so the summary cannot claim an +all-clear), or present-but-empty, which is what a moved repository looks like. + +Also: the four local sway tools in sway.stack pointed at +github.com/nimbin2/..., which is not where they live. They now point at +git.christianimmanuel.de, the URLs their author gave. A stack that ships with +someone else's URL for your own software is a pin that was never right, not +one that rotted. + +## Sort tags as versions (1.14.68) + +--check-refs did its job on the real stack: twelve pins verified, one bad. +And then suggested the wrong replacement: + + foot 1.24.1 DOES NOT EXIST + remote has: 1.9.2, 1.9.1, 1.9.0, 1.9, 1.8.2 + +foot's newest tag is 1.28.0. Those five are what a reverse STRING sort +produces, because "9" > "2" one character at a time. A suggestion that names +an ancient release as the newest is worse than no suggestion: it invites a +downgrade, and the person has no reason to doubt it. + +Tags are compared component by component now, numbers as numbers, with a +leading "v" ignored and non-numeric parts (release-3.2.4) sorted after. The +summary also reads "1 entry pins" rather than "1 entry pin". + +Both faults were in output I had just written and read twice. The test that +catches it uses foot's actual tag list -- a fixture taken from the failure is +worth more than an invented one, because it encodes what really happened. + +## Check every pinned ref at once (1.14.67) + +Second rotted pin in the same stack. usrsctp was 1.14.44; now foot: + + error: pathspec '1.24.1' did not match any file(s) known to git + +whose tags are 1.24.0 and 1.28.0 -- 1.24.1 never existed. The 1.14.44 +diagnosis worked perfectly and listed the real tags, which is the right answer +to the wrong question: each rotted pin still costs a clone, a build attempt +and a round trip to discover. + +1.14.44's note said `--check-refs` would turn this into a five-second report. +`packagemanager stack <name> --check-refs` asks every git remote whether its +ref still exists and builds nothing. Commit ids are reported as unchecked +(cheap verification is not possible), an entry with no ref= as tracking the +default branch. + +The detail that took a second pass: an unreachable remote must not be counted +as verified. "every pinned ref exists" over seven unreachable remotes is a +false all-clear, and the ones that could not be asked are exactly the ones +that might be wrong. It now says "every ref that COULD be checked exists" and +counts the rest. + +sway.stack's foot pin is corrected to 1.28.0. + +## A git entry can name its dependencies (1.14.66) + +sway is built from git, so nothing in the book orders anything before it: + + ERROR: Dependency "json-c" not found, tried pkgconfig and cmake + +json-c IS a BLFS package (JSON-C-0.18). It had simply never been built, +because a git stack entry had no way to say it needed one: book entries get +their dependencies from the book, git entries got nothing at all. Every git +package in the stack is in this position, and the failures arrive one +dependency at a time, at configure, after the clone. + +`needs=` on a git or tar entry names book packages to install first -- through +the same door as everything else (1.14.13), skipped when already installed, +and stopping the stack if one cannot be built. sway.stack now declares +`needs=json-c,cairo,pango`. + +A test asserts that every name in every `needs=` in the shipped stack is a +package the book actually has. The first draft said `pcre2`, which is LFS and +not BLFS, so the install it triggered could never have worked -- a declaration +that names something unbuildable is worse than none, because it fails later +and further from the cause. + +Remaining for a future session: the other git entries (swaybg, wlroots, +libnice, ...) have no needs= yet, and their dependencies are only discoverable +by building them. Each failure of this kind should end with a line added to +the stack, not just a package installed by hand. + +## A warning that is only fatal because of -Werror (1.14.65) + +wlroots 0.19 against libinput 1.31: + + error: enumeration value 'LIBINPUT_SWITCH_KEYPAD_SLIDE' not handled in + switch [-Werror=switch] + cc1: all warnings being treated as errors + +Nothing is wrong with wlroots. A dependency grew a new enum value since that +version was written, and -Werror makes the warning fatal. The box called it +"a COMPILE error in the package's own source ... the tools cannot repair it" +and offered `c_args = -DSOMETHING`, which is no help. + +Version skew like this is the NORMAL state of a rolling book against pinned +git sources, and the cure is always the same flag. The box names it, and +carries the warning class through so the narrow flag is possible too: + + [wlroots] + werror = false # meson + c_args = -Wno-switch # cmake or autotools + +An ordinary compile error keeps the old advice and nothing more; that +counter-case is in the suite. + +Flaky test noted: "a busy tree hangs the build instead of warning" spawns a +`sleep` and greps for it, and fails perhaps one run in four. It should use a +marker only it can see rather than racing the process table. + +## A zero exit is not proof that nothing failed (1.14.64) + +Rebuilding alsa-lib to fix the missing libasound, the install ended with + + install: cannot create regular file '/usr/include/sys/asoundlib.h': + Permission denied + make[4]: *** [Makefile:786: install-data-hook] Error 1 + make: *** [Makefile:408: install-recursive] Error 1 + +and the run reported "# alsa-lib: done. 1 package(s) installed." + +THIS IS THE ORIGIN OF THE WHOLE CLASS. Every "recorded installed but is not" +in this session -- pathspec, cargo-c, Whois, the pip modules, alsa-lib's own +missing library -- is a package whose install did not finish while something +said it had. The runner itself is correct (a phase stops at the first failing +command; asserted), so the status was swallowed somewhere else: an older +script format, a trailing command that succeeded, a wrapper that returned 0. + +Rather than chase which, the LOG is now checked whenever a phase exits 0. A +line matching `make: *** ... Error N`, `ninja: *** build stopped`, `install: +cannot create`, or a trailing `Permission denied` turns the phase into a +failure, with the offending line quoted. A package that could not write its +files is not installed, whatever the exit status said. + +The false-positive cases are in the suite too ("checking for Permission denied +handling... yes", "Error handling tests: 12 passed"), because a check this +blunt earns its place only by not firing on prose. + +Note the shape: the permission failure would have been REPAIRED by the +grant-and-retry if the phase had failed properly. A swallowed status does not +just mislead, it disables the machinery built to fix exactly that fault. + +## Ask the manifests about a library too (1.14.63) + +The three-way diagnosis landed on the user's real case -- "Nothing here +provides libasound at all" -- and stopped one step short. p_alsa-lib had a +1711-byte pkg.lst while /usr/lib had no libasound of any kind: SOMEBODY +recorded installing it, and that package is the one to rebuild. + +The library branch now does what 1.14.33 does for a missing program: greps the +manifests, names the package whose record claims the file, and offers + + packagemanager install alsa-lib --run --reinstall + +Two diagnoses written a fortnight of versions apart, for the same underlying +fault -- a record that outlived the files. The manifests are the one place +that knows, and every "X is missing" message should ask them. + +Also worth recording, because it cost ten minutes of hunting: three suite +checks failed with no visible FAIL line, and the cause was the packaging step +(`rm -f *.html`) having removed the book symlinks before the run. The checks +that need a book should say so rather than failing silently -- the same "an +environment failure must look like one" problem as the rc.site check that has +sat at the bottom of every run this session. + +## A library that will not link (1.14.62) + +alsa-utils found alsa-lib's headers and then: + + checking for snd_ctl_open in -lasound... no + configure: error: No linkable libasound was found. + +Headers present but the library unlinkable is a different fault from a +missing package, and none of the existing diagnoses covered it -- the box said +nothing at all. Three ways it happens, told apart by what is on disk: + + libfoo.so AND libfoo.so.N -> it should link; stale ldconfig cache, or + the real error is further up in config.log + libfoo.so.N only -> the DEVELOPMENT symlink is missing: the + library exists at run time and not at build + time, so its owner installed incompletely + neither -> nothing provides it + +The middle case is the one worth naming: a package-user install that missed +the .so symlink leaves a system where everything RUNS and nothing BUILDS, and +the error blames the package that is merely the first to notice. + +Both wordings are read -- configure's "No linkable libX was found" and the +linker's own "cannot find -lX". + +## A skipped variant does not keep its install step (1.14.61) + +"is it possible that the install of webkitgtk is repeating itself at the end? +i think im building it the second time!" Its install phase held + + ninja -j8 install + ## OPTIONAL, skipped -- ... the GTK-4 version ... + # <the whole GTK-4 build, commented> + ninja -j8 install <- the GTK-4 variant's install, still live + +1.14.53 enables one of a page's alternative BUILD blocks; each variant also +has its own install step, and the skipped one's stayed. The repeat is a no-op +rather than a second build -- but on a package that takes hours, nobody +watching can tell that, and the question is the right one to ask. + +The first attempt at the rule commented ANY repeat after a skipped block, and +dropped GLib's last `ninja install` -- the one that installs the reconfigured +build with introspection enabled, several live commands later. The rule is +now exactly "X ; <skipped variant> ; X": a repeat is only redundant when +nothing happened in between. One page in the book matches (WebKitGTK), and +GLib's counter-case is in the suite beside it. + +Whole-book invariants held throughout: 954 scripts, 0 syntax problems, 0 empty +install phases. Those two numbers have caught or confirmed every filter +change in this session; keeping them cheap has been worth more than any +individual fix. + +## Put the option on the command, not in the prose (1.14.60) + +WebKitGTK died on + + fatal error: systemd/sd-journal.h: No such file or directory + +with `ENABLE_JOURNALD_LOG = OFF` sitting in machine.conf. The generated +script showed why: + + ## NOTE: ... add the -D CMAKE_CXX_FLAGS_RELEASE=... option to the cmake + -Denable_journald_log=OFF -Duse_gstreamer_webrtc=ON to disable some ... + +Two bugs in that one line. + +The options were inserted at the first line CONTAINING the word "cmake" -- +which on that page is a NOTE, prose, three paragraphs before the command -- +so nothing reached the build. Insertion now skips comment lines. + +And they were lower-cased: configparser folds keys, and CMAKE VARIABLES ARE +CASE-SENSITIVE. Even correctly placed, `-Denable_journald_log=OFF` sets a +variable nothing reads, and cmake accepts unknown -D silently. `optionxform += str` keeps the case. meson's options are lower-case anyway, which is why +[mesa] worked for months and hid both. + +Same shape as 1.14.23, 1.14.41, 1.14.51, 1.14.56: text that mentions a command +is not that command. Four filters and now an INJECTOR have made it. Anything +in these tools that scans generated script text should skip comments unless it +has a specific reason not to -- that rule is cheap and would have prevented +all five. + +## ninja does not read MAKEFLAGS (1.14.59) + +WebKitGTK's page says it outright: some of its source files need more than +4 GiB of RAM each, so pass `-j<N>` to ninja with N = RAM / 4 GiB, or the OOM +killer takes the build. + +MAKEFLAGS cannot express that -- ninja ignores it and uses nproc+2 -- so +following the book meant editing the generated script by hand, and the failure +it prevents arrives hours in as a killed compiler. machine.conf takes a +`jobs` key now, which rewrites bare `ninja` invocations to `ninja -jN` (a line +that already has -j is left alone, and re-applying does not stack flags). + +Also confirmed while answering the user's question about GTK-3 vs GTK-4: the +page's two build variants "differ only in the setting of -D USE_GTK4" (its +own words), and `apply_machine_opts` REPLACES an option the book sets rather +than appending -- so `USE_GTK4 = ON` in machine.conf turns the enabled GTK-3 +block into the GTK-4 build, without touching which block 1.14.53 enabled. +That is the mechanism working as intended across three separate features. + +## A stack file is the user's (1.14.58) + +Found while answering a question about WebKitGTK's feature list: `make +install` copies stacks/* over /etc/pkgusr/stacks/* UNCONDITIONALLY. + +The user had just removed `avoid ruby` from sway.stack to get webkitgtk to +build. Every fix in this session begins with `make install`. The next one +would have put `avoid ruby` back, the build would have failed the same way, +and nothing would have said why. machine.conf lives in the same directory -- +that is where the per-package build options go, and it was equally at risk. + +Missing file: installed. Identical: reported as unchanged. Different: the +user's is KEPT and the shipped one saved as `.new` beside it. The skel +handling a few lines further down has done this since it was written; the +stacks loop simply never did. + +A file the tool tells people to edit cannot be a file the tool overwrites. +Worth a sweep for others: anything under /etc that `make install` writes. + +## One place to set the job count (1.14.57) + +Asked where -j is set. Four places: + + lfs config makeflags -> <tree>/usr/src/lfs-pkgusr/config/env, + exported by every chroot shell + /etc/pkgusr/build.env -> the per-machine file, via build_env_pairs + lfs-helper build --jobs N -> one build + lfs-phases -> -j$(nproc), only if nothing else set it + +And they did not agree. build.env is added to the command FIRST and the +caller's MAKEFLAGS after it; the last assignment on an `env` line wins. So an +ambient -j16, inherited from the chroot's config env, silently beat a -j13 put +in the file that exists for exactly this. + +Precedence is now --jobs, then build.env, then the ambient value: the caller's +MAKEFLAGS is only passed when nothing has already set one. + +For the record, the fix a user wants is one line in /etc/pkgusr/build.env +(`packagemanager env --set MAKEFLAGS=-j13`), which now actually takes effect; +`lfs config makeflags` remains the setting for the chapters 5-8 build, where +there are no package users and no build.env yet. + +## A demonstration is not an installation step (1.14.56) + +enchant installed perfectly -- 17 files tracked, enchant-2.pc in place -- and +then the configure phase failed: + + No dictionary available for 'en_GB' + +Its page ends with "You can test your installation and configuration by +creating a test file and running the following commands", and the generator +turned that demonstration into a step. It needs an aspell dictionary the +system may not have, and running it is nobody's idea of installing enchant. +Blocks introduced by "you can test / verify / try" are commented, with the +book's text kept. Two pages in the book: enchant and autofs. + +The failure box got it wrong twice over. It reported + + 'not' is not installed -- the build needs it + find what provides it: packagemanager search not + +from a line in enchant's own configure: + + ./configure: line 26491: not: command not found + +which is a broken autoconf probe, harmless, and fifty lines above the real +failure. Shell keywords and a few English words are skipped now, and the next +candidate is taken -- so a log with both `not` and `cmake` reports cmake, and +a log with only noise reports nothing. + +Both halves of this are the same mistake in different places: text that LOOKS +like a command (a book's example, a shell diagnostic) treated as one. The +tools read prose all day; every reader of it needs to know what it is reading. + +## cmake takes -D options too (1.14.55) + +WebKitGTK, next configure error: "Enchant is needed for ENABLE_SPELLCHECK". +enchant is in the user's avoid list and webkitgtk only RECOMMENDS it, so +1.14.54's check correctly stayed quiet -- a recommended package being absent +is what an avoid list is FOR. The natural fix is to turn the feature off, and +that had nowhere to live: + + machine.conf has [webkitgtk] but the script has no `meson setup` line + -- options NOT applied + +webkitgtk is a cmake package. `apply_machine_opts` only knew `meson setup`, +so for every cmake page in the book the file was inert -- which also explains +the [vulkan-loader] and [libass] warnings the user had been seeing for days +and reasonably ignoring. + +Both build systems take -Dkey=value; only the command word differs, and +autotools already had configure_args. An option the BOOK sets is replaced +rather than duplicated (cmake takes the last, so a duplicate would have been a +silent no-op), and re-applying changes nothing. + +The diagnosis chain that got here is worth noting as a whole: the box named +the module (enchant-2), traced it to the package (enchant), said it was not +installed, and offered the install -- 1.14.43's work, doing exactly its job on +a failure nobody had in mind when it was written. + +## An ignored package that something REQUIRES (1.14.54) + +WebKitGTK configured for an hour and stopped at + + CMake Error: Ruby 2.5 or higher is required. + +webkitgtk REQUIRES Ruby-4.0.1. The stack's avoid list contains `ruby`. So +the plan omitted a hard requirement of a package the user had asked for, and +said nothing about it. + +Not walking into an avoided branch is right and stays. What was missing is +the report: `walk_deps` records (parent, ignored) whenever the edge it skips +is REQUIRED, `blfs order --anchors` emits `#ignored-required`, and the plan +warns before building: + + 1 package(s) in this plan REQUIRE something the ignore list excludes: + webkitgtk requires ruby, which is ignored + Either drop it from the ignore list, or install it: + packagemanager install ruby --run --recursive + +Only REQUIRED edges are reported; ignoring a merely recommended package is the +normal use of an avoid list. + +A note on the test for this: the first version asserted that ignoring +bubblewrap (which webkitgtk only recommends) reports nothing -- and it failed, +because glycin in the same tree REQUIRES bubblewrap. The code was right and +the test's premise was wrong. When a new test fails, the first question is +which of the two is mistaken. + +## A page that offers a choice must still build (1.14.53) + +WebKitGTK unpacked, and then + + ninja: error: loading 'build.ninja': No such file or directory + +Its page gives TWO complete builds -- "If you want to install the GTK-3 +version of WebKitGTK, run the following commands", and the same for GTK-4. +Each is introduced by a conditional, so `_conditional_guard` commented BOTH, +the build phase came out as `:`, and the install phase ran `ninja install` +against a directory with nothing in it. + +Guarding a conditional block is right; guarding EVERY block is a package that +cannot be built. When a build phase has no live command left but a skipped +block exists, the first is restored, with a line saying which variant was +chosen and that the others are still there to switch to. Five pages in the +book are in this shape: WebKitGTK, okular, Vim, docbook-dsssl, Py3c -- all of +which previously built nothing. 954 scripts still parse. + +Third failure this session from the same root (gegl 1.14.39, Whois 1.14.41): +a filter that comments commands was right about the line and wrong about the +phase. The invariant that catches all of them is already in the suite -- no +generated script may have an empty install phase -- and it should be extended +to the BUILD phase too, with the handful of legitimately-empty pages listed +by name rather than counted. + +## A fallback must not take you back a release (1.14.52) + +The user is building 13.0. They fetched the 12.4 book as a REFERENCE for +SysV commands -- and the next build produced **BlueZ-5.83**, which is 12.4's +version. Fetching a book to read from silently changed the book they build +from. + +The configured default is `stable-sysv`, which no longer exists, so +`_best_cached_for` scored the cached books: init flavour is worth 4, an exact +selector match 8, version nothing at all. The 12.4 sysv book matched the +flavour and won, and the 13.0 book it passed over was never mentioned. + +Flavour still decides -- that is what the selector asks for -- but among books +of the same flavour the newest now wins, and the note names what it passed +over: + + note: no book matched 'stable-sysv'; using the cached BLFS-BOOK-12.4... + a NEWER book is also cached: BLFS-BOOK-13.0-systemd... (13.0) + building from 12.4. To pin the other one: + blfs set-default 13.0-systemd + +Also here: `make install-<name>` in a CONFIGURATION section is the SysV books' +boot-script idiom, and the target lives in blfs-bootscripts, not in the +package. BlueZ installed everything correctly and then died on + + make: *** No rule to make target 'install-bluetooth'. Stop. + +It is commented with the real instructions now. automake's own +install-strip/-data/-exec/-man are left alone. + +The lesson for the session: on 13.x the SysV book is a REFERENCE, never the +build source. Build from 13.0-systemd and carry the init differences as +per-package overrides -- which is what diff-init exists to find. + +## Judge the difference on lines that run (1.14.51) + +The narrowed survey named eight pages on the user's system. Four were +nothing: gcc differed only in a NOTE whose URL contains "13.0-systemd", +libxml2 in a NOTE saying "systemctl stop httpd.service", rust in a NOTE about +the "systemd journal". Prose, all of it -- 1.14.23's lesson in a different +filter. Comments are stripped before the difference is judged now, and the +three real ones (bluez, dbus, make-ca) still come through. + +What the eight actually were, for the record -- this is the SysV work a 13.x +system needs, and it is small: + + bluez --disable-systemd is gone from 13.0's configure REAL + dbus -D systemd=disabled is gone from 13.0's meson REAL + make-ca 12.4 writes /etc/cron.weekly/update-pki.sh; + 13.0 does `systemctl enable update-pki.timer` REAL + at-spi2-core 12.4 redirects systemd_user_dir to /tmp and + deletes the .service afterwards cosmetic + gcc, libxml2, rust prose + llvm 13.0 runs its TESTS via systemd-run --user; the + systemd shims turned that into a visible skip handled + +That last one is worth noting: the shims added for exactly this (a systemd +book on a SysV system) did their job silently and the build succeeded without +its test suite. The design held. + +## What differs AND concerns you (1.14.50) + +The survey worked: 47 of 929 shared pages differ in their init handling. And +that is still the wrong list to hand someone -- most of the 47 are GNOME, +samba, qemu, postgresql, software this machine does not have and never will. + +`--installed` intersects the survey with the package users on disk: a package +with an account has been built or is about to be. On the user's tree that +turns 47 into the handful they are actually going to build. An empty +package-user root says so rather than comparing nothing. + +The general shape: a report is only as useful as it is short, and the filter +that shortens it is usually already on the machine. Same move as verify's +"16 need attention" out of 242 (1.14.34), and the plan's "127 of 233 already +installed" (1.14.15). + +## There is no 13.0 SysV book (1.14.49) + +The fact this whole thread has been working around, established by the user's +own `blfs books`: BLFS stopped publishing the SysV edition after **12.4**. +The 13.x books are systemd-only. So "just fetch the sysv book" -- advice +given repeatedly in this session, including by me -- was never possible for a +13.0 system. + +Three things hid it. + +`blfs fetch stable-sysv` printed the routine "no book matched" note and then +"cached." A fetch that fetched nothing reported success. It now checks the +published list, fails, and names the newest SysV edition that does exist. + +`blfs diff-init 12.4` answered "0 of 947 shared pages differ" -- true and +useless: the default selector has no book, so the fallback picked the newest +cached one, which was 12.4, and it compared that book with itself. Both sides +are printed now, and comparing a file with itself is refused. + +And comparing ACROSS versions (12.4-sysv against 13.0-systemd) is the only +comparison available, where most pages differ for reasons that have nothing to +do with init. The survey filters to differences that MENTION init +(systemd, systemctl, bootscript, init.d, tmpfiles, unit-dir, service), with +--all for everything. + +What this leaves, for the next session: on 13.x, SysV pages must come from +12.4 plus judgement, or from per-package `configure_args`. lfs-sysvbook +already grafts SysV onto a systemd LFS book; the same treatment for BLFS -- +take 12.4's init-related command differences and apply them to the 13.x pages +-- is the obvious next tool, and `diff-init` is the survey it would be built +on. + +## See the difference, do not guess it (1.14.48) + +BlueZ stopped with + + configure: error: systemd system unit directory is required + +on a SysV system. Nothing was broken: the SYSTEMD book's page was used, and +its configure line carries no --disable-systemd because on a systemd machine +there is nothing to disable. Every run had already printed "no book matched +'stable-sysv'"; this is what that note costs when it is ignored. + +Two additions. + +The failure box now recognises a systemd-only configure failure on a machine +with no systemctl, and points at the BOOK -- `blfs fetch stable-sysv`, which +fixes every later page too -- with the per-package `configure_args = +--disable-systemd` as the offline fallback. + +And `blfs diff-init <book> [package]` answers the question the user actually +asked ("let me hold the sysv in a vimdiff"): with no package it lists every +page whose commands differ between the two books; with one it prints the +unified diff of that page's commands. Measured against a doctored copy: 1 of +954 shared pages, and the diff shows exactly the --disable-systemd line. + +That is the shape worth keeping. A whole-book survey turns "which of my 954 +pages will bite me on SysV?" from a question answered one failed build at a +time into a list printed in a minute, before anything is built. When the two +sources of truth disagree, show the disagreement -- do not wait for a package +to discover it. + +## One typo must not disable a whole file (1.14.47) + +The recipe was followed twice, so machine.conf held + + [webrtc-audio-processing] + cpp_args = -include cstdint + [webrtc-audio-processing] + cpp_args = -include cstdint + +configparser raises on a repeated section, and `_read_conf_section` discarded +the ENTIRE file on any error. So the new section did nothing -- and neither +did [mesa]: its driver selection quietly reverted to "build everything" +because of a duplicate heading further down. That is how you get a mesa build +that wants libdrm_intel on an AMD machine, from an edit made somewhere else +entirely. + +`strict=False` merges repeated sections, later keys winning, which is what +someone appending a correction means. A file that is genuinely malformed +still reports -- and now says how far the damage reaches: "NO options from +that file were applied, not for this package and not for any other." + +`cat >>` is how every recipe in this session told the user to edit that file, +so a repeated section was not an edge case, it was the expected outcome of +following instructions twice. A config parser that fails closed on the whole +file, over a duplicate heading, punishes the ordinary way of using it. + +## Advice that nothing reads (1.14.46) + +1.14.45's box told the user to put + + [webrtc-audio-processing] + cpp_args = -include cstdint + +in machine.conf. webrtc-audio-processing is a GIT STACK ENTRY, and +`apply_machine_opts` was called only from `resolve_script` -- the book path. +Nothing would have read that section. + +Correct and inert is worse than wrong: it looks like it worked, the build is +retried, the same error comes back, and the person has no way to tell whether +the advice was bad or their edit was. The stack's git/tar branch applies the +options now; a script is a script, and the options belong to the PACKAGE, not +to where its commands came from. + +Three consecutive releases (1.14.44, .45, .46) have been "the message was +right, the machinery behind it was not". Whenever a diagnostic prints a +recipe, the next question is which code path consumes it -- and whether the +path that just failed is one of them. + +## One flag, and the same one every time (1.14.45) + +webrtc-audio-processing stopped with + + trace_event.h:202:18: error: 'uint8_t' does not name a type + +That is not a broken package: GCC 13 stopped pulling <cstdint> in through +other headers, so anything written before that no longer compiles. Half the +older C++ in BLFS hits it. The box said "if it needs a build flag, put it in +machine.conf" -- true, and no help at all when the flag is always the same +one. + +It now names the cause and the cure, in the right language for the file that +failed: + + cpp_args = -include cstdint (C++) + c_args = -include stdint.h (C) + +...and an ordinary compile error still gets the generic advice, so this does +not become "every failure is cstdint". + +While writing that: the flag would NOT have worked. `apply_machine_opts` +pasted values into the shell line unquoted, so `cpp_args = -include cstdint` +became `-Dcpp_args=-include cstdint` -- two shell words, one of them a stray +`cstdint`. Every option that takes a flag LIST has this shape; the +single-word examples in the help text (`c_args = -DSOMETHING`) hid it for as +long as the feature has existed. Values with whitespace are quoted now, and +re-applying does not nest the quotes. + +Recommending a fix and then mangling it is the worst of both: the person +follows the advice, it fails differently, and the advice is what they stop +trusting. + +## A ref that does not exist is a typo, not a build error (1.14.44) + +sway.stack pinned usrsctp to `ref=1.0.0`. That tag does not exist upstream -- +the repository has 0.9.3.0, 0.9.4.0, 0.9.5.0 -- and all the person saw was +git's own + + error: pathspec '1.0.0' did not match any file(s) known to git + !! phase 'unpack' FAILED (exit 1) + +which reads like a build failure and sends them to a log. The repository is +cloned and standing right there, and it knows exactly what it does have. + +The runner now names the ref, lists the recent tags and the branches, and says +where the wrong value came from ("Fix ref= in the stack entry for X"). The +phase still fails -- explaining is not excusing. + +The stack's pin is corrected to 0.9.5.0, checked to have meson.build so the +entry's build line still applies. libsrtp's v2.6.0 verified too; libnice and +webrtc-audio-processing are on gitlab.freedesktop.org, which this environment +cannot reach, so those two are unverified and may hold the same kind of typo. + +Worth noting for stack entries generally: a pinned ref is data that rots, and +the only thing that can check it is the remote. `git ls-remote --tags` is +cheap; a `packagemanager stack --check-refs` that runs it over every git entry +would turn this class of failure into a five-second report. Not built. + +## Not installed is not the same as not a package (1.14.43) + +The user asked: "is something broken in our install cycles?" No -- and the +evidence they gathered settled it in one line: + + ls -al /usr/src/pkgusr/ | grep tiff -> (nothing) + +There is no p_libtiff account: libtiff was never installed here. gtk4 +requires it, and NO page in that chain lists it -- not gtk4's, not +gdk-pixbuf's, not librsvg's (all three checked). Same family as libaom +needing cmake: the book does not say, so the planner cannot know. + +But the box handled it badly. 1.14.42 resolved a module name to its package +by asking whether the ACCOUNT exists, so with libtiff absent it fell through +to "nothing provides 'libtiff-4' -- packagemanager install libtiff-4", naming +a package the book does not have, and then offered to vendor a copy. Two +wrong answers where the right one was one string operation away. + +A trailing API version that strips to a DIFFERENT name is the signature of a +pkg-config module belonging to a real package (libtiff-4 -> libtiff, +libxml-2.0 -> libxml). The box now names that package, offers to install it, +and says how to check the guess. libnsgif strips to itself, so it keeps the +fallback advice -- the distinction that matters is preserved. + +Three releases on this one message (1.14.39, .42, .43), each fixing what the +previous one got right for one case and wrong for the next. A diagnostic that +prescribes should be tested against every shape it can meet, not the one that +prompted it. + +## One answer per failure (1.14.42) + +gtk4 stopped on `Dependency 'libtiff-4' is required but not found`, and the +box told the user to allow a vendored copy: + + [gtk4] + wrap_mode = default + +libtiff-4 is the pkg-config MODULE of libtiff -- a BLFS package, installed on +that system. Following that advice would have built a second libtiff inside +gtk4 and buried a pkg-config problem under it. The advice added in 1.14.39 +was right for gegl and wrong here, and nothing distinguished the two. + +Two faults. meson has a SECOND wording -- "Dependency 'x' is required but not +found", single quotes -- which matched none of the four extraction patterns, +so the whole .pc diagnosis was skipped and only the fallback note survived. +And the fallback note ran BEFORE that diagnosis, so it could not have known +better even with the name in hand. + +Now the diagnosis runs first and resolves the module name to the package that +owns it (`libtiff-4` -> `libtiff`, trying the name and then with a trailing +API version dropped -- libxml-2.0, ncursesw6 are the same shape). When there +is an owner the box says so and offers the rebuild; the fallback note stays +quiet. When there is none, the fallback note is the answer and the generic +"nothing provides it" line stays quiet instead. + +Saying two contradictory things in one box is worse than saying neither: the +person has to decide which half to believe, and the wrong half was the one +with a command under it. + +## A test runner is a command, not a substring (1.14.41) + +librsvg died on `Program 'cargo-cbuild' not found`, and cargo-c -- which +provides it -- had installed nothing. Its one install line: + + install -vm755 target/release/cargo-{capi,cbuild,cinstall,ctest} /usr/bin/ + +`\bctest\b` matched inside the brace list, because a comma is a word +boundary. The line that installs four binaries was commented as "runs the +test suite"; verify was RIGHT to call cargo-c not installed; and the failure +surfaced two packages later in something that needed one of those binaries. + +`ctest` and `pytest` now need a lookbehind rejecting a name that is part of a +word, a path or a list. Being a test runner is about being the COMMAND. + +Found while checking the whole book for the same shape: Whois installed +nothing either. It builds with a bare `make` and installs with `make +prefix=/usr install-whois`, and `_comment_orphan_doc_installs` skips +`make install-X` unless something ran `make X` -- there is no `make whois` +target. A bare make builds the package; the exception is the DOCUMENTATION +targets (man, html, info, pdf...), which is the case that filter exists for -- +git has a bare make too, and its manuals still need the conditional +`make man`. Both verified. + +The whole book now: 954 scripts, 0 syntax problems, 0 install phases that do +nothing. That last number is the one to keep in the suite -- it is the +cheapest possible check for this entire family of filters, and it would have +caught the pip wheels (1.14.16), cargo-c and Whois on the day each was +introduced. + +CACHE_VERSION 52. + +## Protect against what the book has (1.14.40) + +1.14.39 told the user how to allow gegl's fallback by hand. They should not +have had to: gegl is a book package that BLFS builds with a plain `meson +setup`, and our flag is what stopped it. + +The rule the protection was reaching for is narrower than "no fallbacks ever": + + libnsgif, poly2tri-c no BLFS page -> nothing can supply them; allow + freetype2, fontconfig in the book -> a fallback would shadow a package + we manage; refuse + +`--force-fallback-for` overrides the wrap mode for exactly the named +subprojects, so gegl now generates + + meson setup --force-fallback-for=libnsgif,poly2tri-c --wrap-mode=nofallback ... + +and cairo is untouched. `_FALLBACK_ALLOWED` carries the list, with a test +asserting every name in it really has NO page -- so if the book gains one, the +suite says so rather than a vendored copy appearing quietly. CACHE_VERSION 50. + +The machine.conf `wrap_mode` override from 1.14.39 stays for the cases the +list has not caught yet, and the failure box still explains itself. A default +should fit the common case; an escape hatch is for the rest, not for a +package the book supports. + +## A commented block does not move the stream (1.14.39) + +gegl failed with + + ninja: error: loading 'build.ninja': No such file or directory + +Its generated script had an EMPTY build phase and did the whole build inside +install_pkg -- so when `meson setup` failed, the `ninja install` on the next +line ran regardless and produced that message instead of the real one. + +`extract_commands_phased` latches: once a ROOT block has appeared, everything +after it joins the install stream. That is right, and GLib needs it (build, +install gobject-introspection as root, then reconfigure). But gegl's first +root block is conditional -- "If you are installing over a previous version +... remove it" -- so it is emitted COMMENTED and nothing has happened, and it +flipped the latch anyway. A block that does not run cannot mark a boundary. +CACHE_VERSION 49. Across the book: 954 scripts, 0 syntax problems, GLib's +interleaving intact. + +The real dependency error underneath was ours too. blfs adds +`--wrap-mode=nofallback` to every `meson setup` so a package cannot silently +vendor a dependency (cairo once installed its own freetype, and the planner +then believed freetype2 was done). gegl needs libnsgif and poly2tri-c, which +have NO BLFS page -- the fallback is the only way to get them, and meson's +message ("Use of fallback dependencies is disabled") never says who disabled +it. So: `wrap_mode` is settable per package in machine.conf, and the failure +box says the flag is ours and prints the three lines that override it. + +A protection that cannot be turned off for the case it does not fit is a bug +with a good reason. + +## A report names the thing it is about (1.14.38) + +After 1.14.37 the tree came to 232 installed and six lines needing attention. +Two of those six were the report's own fault. + + root broken has a home dir but no user + +That sent the user looking for /usr/src/root, which is not where it is. The +path was in hand when the line was printed and thrown away; it now reads "no +account for the directory <path>", which is the whole of what anyone needs to +act. + +And the advice under the list said "remove with: packagemanager remove +<name>". For a directory with no account the fix is to delete the DIRECTORY, +and suggesting a remove command against a name like `root` is an invitation to +an accident. It names the line above instead. + +Two process notes from this release. The version bump was done with a blanket +`sed s/1.14.37/1.14.38/` over the tools, which rewrote HISTORICAL references +inside comments -- "fixed in 1.14.35" became "fixed in 1.14.38", and a test +whose sed range quoted one of those comments stopped matching. Bump the +version CONSTANTS only; a comment naming a version is a fact about the past. + +Small, but this is the fourth report in the session whose remedy was "print +what you already know" (1.14.6 the file paths, 1.14.32 the cycle, 1.14.33 the +manifest owner). The information is almost always in hand at the moment of +the message. + +## An install that leaves no record is invisible (1.14.37) + +verify reported beautifulsoup4, certifi, charset_normalizer, idna, requests, +soupsieve and urllib3 as "not installed -- nothing installed yet", while blfs +was using bs4 to parse the book on that same machine. Their homes held +build.conf and .project and nothing else. + +`packagemanager pip` installed the module and wrote NO record: no pkg.lst, no +install_last. Every downstream reader -- verify, the planner, gather_user -- +looks at those two files, so the bootstrap set the tools depend on was +invisible to the tools. And then verify offered to rebuild them with a +command that could not work, because they are not BLFS packages either. + +The user's question was the right one: "if we use pip to install something, it +should somehow be unified". It is now. `pip_install_module` writes the +manifest (write_pkg_list), an install_last carrying name_version -- with the +version asked of the MODULE via importlib.metadata, not of the request -- and +a validate_cmd that can confirm it later. So a pip module is a package like +any other from the moment it lands. + +For the ones already installed without records, `packagemanager pip record +[names]` writes the paperwork without reinstalling; with no names it finds +every account whose module imports but has no record. It REFUSES to record a +module python cannot import, which is what keeps it from stamping a lie onto +the exact state it was written to fix. verify names that state and prints +that command instead of "nothing installed yet". + +The pattern, for a future session: state lives in pkg.lst + install_last, and +anything that installs must write both. Three install paths existed +(pm-install, the runner, pip) and only two of them did. + +## Silence is the one answer a command must never give (1.14.36) + +Pasting verify's own suggested command: + + packagemanager install beautifulsoup4 cargo-c ... --run --recursive + note: no book matched 'stable-sysv'; using the cached BLFS-BOOK... + (nothing else, prompt returns) + +`blfs order` exits 1 for a name the book does not have, and +`blfs_order_anchors` turned that into `sys.exit(1)` for the WHOLE run -- +before any plan was printed. One name that is not a BLFS package (soupsieve, +pkgusr, linux-headers) killed a twelve-package command, and the person could +not tell whether it had worked, refused, or crashed. + +It reports the name and returns an empty order now; the caller records it in +`not_found` and carries on with the rest. The recursive branch also never +recorded such a name at all -- the non-recursive one always had. + +And verify no longer suggests a command that cannot work. Its list is split: +BLFS packages get the install command; LFS steps (linux-headers, pkgusr) are +named as chroot-build work; anything else is an account with no package. +Offering a command that fails is worse than offering none -- 1.14.10 again, +and this time the wrong command came from our own health check. + +## One shared value, two defaults (1.14.35) + +1.14.34 fixed the collector-group filter and the groups were STILL reported as +broken packages. The filter was right; the prefix was not. + +`lfs` and `packagemanager` fall back to "sysgroup"; lfs-helper had one line +saying `${COLLECTOR_PREFIX:-nimgnu}`. On a system whose config never set the +key, groups were CREATED as nimgnu_* by lfs-helper and filtered for as +sysgroup_* by packagemanager. A value read from a shared file, with a +different default in each reader, is not shared. + +Three changes: lfs-helper falls back to "sysgroup" like the others; the +account list recognises BOTH spellings, because real systems already carry +groups under either name and the config still decides what new ones are +called; and `_looks_like_an_account` no longer accepts "a user of this name +exists" -- /usr/src/root passed that test because root exists, which is how a +directory became a broken package. + +Also: 1.14.34's summary block was inserted by a string replace that matched in +TWO functions, so verify printed it twice and cmd_script_status got a copy +referring to a variable it did not have. Broad textual edits need the same +care as the code they edit. + +## A health check nobody reads is not a check (1.14.34) + +A real `packagemanager verify` on a working tree: + + broken: 4 not installed: 12 unvalidated: 226 + +Three problems in one line. + +**226 "unvalidated" were normal.** Most pages name no installed_programs or +-libraries, so there was nothing to check against -- and the answer to +"nothing to check" was a word that reads like a fault. The manifest HAS been +checked by then (its files exist, or the state would already be "not +installed"), so the honest answer is `installed`, with the limit stated. +Anything genuinely wrong still comes out as not installed, broken, or a +missing target. + +**Three of the four "broken" were collector groups.** `nimgnu_*` are groups, +not packages. The filter for them read `d.startswith(_cp + "_")` while +`collector_prefix()` already ends in "_", so it tested for `sysgroup__` and +has never matched anything since it was written. One underscore. + +**The fourth was a stray directory.** BASE_DIR is /usr/src, which holds more +than accounts; every directory in it was treated as a package, so `root` was +reported as broken. An account has one of the account prefixes, exists in the +user database, or carries the skeleton -- a bare directory is none of those. + +And verify now ends with the work rather than the evidence: the packages that +need action, with the command to rebuild them, or "nothing needs attention". +The twelve that mattered were in the middle of 242 lines. + +Also worth recording: the grep suggested to the user in chat -- +`verify | grep -vE "installed "` -- hid the "not installed" lines it was meant +to surface. A filter written against the wrong end of a word. + +## The manifests know who claimed it (1.14.33) + +librsvg stopped on `Program 'cargo-cbuild' not found`. The box had the shape +of it right -- a package recorded as installed whose build did not finish -- +and then said "find the package, purge its record, rebuild it", leaving the +person to find it. + +Every installed package has a pkg.lst naming what it installed. One grep +answers, and the answer says WHICH of two situations this is: + +- a record CLAIMS the file -> that record is wrong; + `packagemanager install cargo-c --run --reinstall` +- nobody claims it -> whatever provides it was never installed; + `packagemanager search`, then install. + +Also hardened: the report reads $SRCROOT and now tolerates it being unset. A +failure report runs under `set -u` in whatever state the failure left, and +must not add a second failure to the one it is describing. Same lesson as +1.14.4 (`_retry: unbound variable`), which is why the test for that one +deliberately does not set SRCROOT either. + +## Say what the cycle cost (1.14.32) + +vala stopped at + + configure: error: Package requirements (libgvc >= 2.16) were not met + +libgvc is Graphviz, which vala RECOMMENDS, and graphviz was in the plan -- +twenty packages later. The cause is a real cycle: + + vala -> graphviz -> gegl -> babl -> librsvg -> vala + +every edge recommended, so one has to give and vala loses. The breaker has +done this correctly since 1.13, and silently, so the consequence arrived as a +configure error with no connection to the decision that caused it. + +`_CYCLE_DROPS` records each dropped edge; `blfs order --anchors` emits it as +`#cycle<TAB>dependant<TAB>dependency`; the plan reports the ones that are real +in the FINAL order (a two-package cycle drops both edges, so reporting both +directions makes one line false every time) and offers the rebuild: + + 10 package(s) are built before something they recommend (a dependency cycle): + vala is built before graphviz + freetype2 is built before harfbuzz + ... + rebuild afterwards if you want it: + packagemanager install babl gdk-pixbuf glib2 vala ... --run --reinstall + +Also: 1.14.26 knew meson's wording for a missing dependency and pkg-config's, +but not autoconf's -- "Package requirements (libgvc >= 2.16) were not met" -- +so every autotools package in the book missed the .pc diagnosis entirely. +Three spellings of one event, and a pattern that matched two of them looked +like it worked. + +## A link in an aside is not a dependency (1.14.31) + +Asked whether the plan could be trimmed: installing Xwayland -- an X server +for Wayland -- planned 235 packages, among them LibreOffice, OpenJDK, +PostgreSQL, TeX Live and a print stack. + +One sentence on the harfbuzz page: + + Graphite2-1.3.14 (required for building texlive-20250308 or + LibreOffice-26.2.1.2 with system harfbuzz) + +Those name what NEEDS graphite2. `_dep_xrefs` took every `<a class="xref">` +in the paragraph, so harfbuzz "recommended" LibreOffice and TeX Live -- and +from them came Qt6, Cups, Java, OpenJDK, Postgres, the lot. poppler +"recommended" okular the same way, from "Qt-6.10.2 (required for PDF support +in okular)". + +The qualified package is always the link BEFORE the parenthesis, so links +inside one can be dropped without losing anything real. `_in_parenthetical` +counts unclosed brackets in the text preceding a link. CACHE_VERSION 48. + + xwayland 235 -> 53 mesa 41 dbus 35 pipewire 110 gtk4 125 + required-only: 25, unchanged -- nothing real was lost + +Checked by hand against the book: harfbuzz keeps GLib, Graphite2, ICU and +FreeType (exactly what its Recommended line says), poppler keeps Qt6 and loses +only okular. + +This also retired the 1.14.24 test, which used harfbuzz/libreoffice as its +example of a prunable branch -- an edge that was never real. It uses +poppler/qt6 now (185 -> 61). A test written against a bug reads as a +specification for it. + +The user's avoid list, and the ignore machinery of 1.14.24, were both +compensating for this. Worth asking, when a workaround gets long, whether it +is working around a mistake. + +## Ask the way the build will ask (1.14.30) + +A plan printed itself twice and refused twice: + + cargo needed by 2 package(s), first: cargo-c + adding to the plan: rust -- its commands call `cargo` + ... the same plan again, the same refusal ... + +Rust was installed. BLFS puts it in /opt/rustc and adds /opt/rustc/bin from +/etc/profile.d/rustc.sh; a build runs through `su -`, a LOGIN shell, which +reads that. `shutil.which` reads OUR path, which does not. So a tool the +build could see perfectly well was reported missing. + +And the 1.14.14 repair could not help: adding an already-installed package to +the plan changes nothing, so the second pass produced an identical stop. A +repair that cannot change the state it is repairing has to notice, or it +loops. + +`_tool_available` asks the way the build asks -- PATH, then `bash -lc +'command -v'` (bounded, memoised), then /opt/*/bin. A provider that is +already installed is never "added" again. And when the package IS installed +while the tool is not on our PATH, the stop says so and names the cause: + + NOTE: rust is installed, so `cargo` exists but is not on PATH here. + BLFS puts it outside /usr (e.g. /opt/rust/bin) and adds it in + /etc/profile.d -- which a login shell reads and this one did not. + +The general lesson: a check must run in the environment of the thing it is +checking. This is 1.14.7 again from the other side -- there the build could +not see what we had set, here we could not see what the build had. + +## The setup for a test is part of the test (1.14.29) + +An ordinary Xwayland build cloned piglit, cloned the X Test Suite, started an +Xvfb server and tried to build xts: + + install_Xwayland-24.1.9: line 54: cd: xts: No such file or directory + Fatal server error + +The page puts all of it in the BUILD block, and only the last line +(`ninja test`) looks like a test run, so the 1.13 filter commented that one +line and left everything it depends on running. Commenting a test but not its +setup is worse than commenting neither: the whole cost is still paid and the +result is thrown away. + +The setup is bracketed by pushd/popd, so the group can be taken as a unit -- +but only when it is recognisably a test suite (`_TEST_SETUP_RE`: piglit, xts, +fate-suite, jtreg, XTEST_DIR...), never a plain pushd group that happens to +sit before a test. A `mkdir` on its own line above the pushd goes with it, +since the directory exists only for the suite. The book's text is commented, +not deleted, so it can be read and re-enabled. + +Across the whole book this changes exactly one page, and all 954 still parse. +That narrowness is the point: the rule fires on evidence, not on shape. + +## Build it, do not just name it (1.14.28) + +The 1.14.27 check worked -- and then asked the person to type the thing the +stack exists to do: + + stack stopped at git 'elogind': it needs a tool that is not installed + packagemanager install git --run --recursive + +A PLAN already handles this itself: 1.14.14 adds a missing provider to what +was asked for and rebuilds. A git entry is not a plan, so it stopped. git is +a book package and the stack installs book packages all day. + +A git or tar entry now installs any missing tool the BOOK provides +(`--recursive`) and carries on. A tool with no BLFS page -- meson, ninja, +gperf, which come from LFS -- still stops the run, because nothing here can +install those and pretending otherwise would loop. + +The distinction that matters, and it has now come up three times (1.14.10, +1.14.14, here): a message telling someone to run a command is only right when +they know something the tool does not. When the tool has everything it needs +to do the thing, printing the command instead of doing it is not caution, it +is an unfinished feature. + +## An old account can point at the wrong home (1.14.27) + +elogind ended up with two half-populated homes: + + /usr/src/p_elogind/ install_elogind, install_last, log + /usr/src/pkgusr/p_elogind/ build, build.conf, src + +Its .bash_profile was dated November 2023 -- an account carried over from an +earlier system. Every tool resolves the home through `pkgusr_home_for` (the +canonical path), but `su -` starts in the PASSWD home, so the build ran in one +directory while its sources were staged in the other, and the failure box +printed two different paths for the same script without either being wrong. + +`_repair_misplaced_home` has existed since 1.11.x and ran only when an account +was CREATED. An account made by an older version was never re-examined. +cmd_pm_install now repairs an existing account's home before staging, in the +LOUD form -- a home wrong since 2023 is not the routine just-created case +1.14.4 quieted. + +Second fix from the same report: `git: command not found`, after the account +was made and the phase started. A stack entry of kind=git sets `is_git="1"` +and the RUNNER does the clone, so `git` appears in no command text and the +1.14.9 scan could not see it. It is read from `is_git` now, git is in the +tool map (`Git-2.53.0` -- checked against the book, per 1.14.14), and a git +entry runs the same prerequisite check as a book package before it creates +anything. + +## Three reasons a dependency is "not found" (1.14.26) + +mesa stopped with meson's entire account of the problem: + + ERROR: Dependency "libdrm_intel" not found, tried pkgconfig and cmake + +libdrm was installed -- 2.4.131, found two lines earlier in the same log. It +had been built before libpciaccess existed, so it has no libdrm_intel.pc and +never will until it is rebuilt. The book only RECOMMENDS Xorg-Libraries for +libdrm, so nothing put them in that order. + +The .pc files on disk tell the three cases apart: + +- the .pc EXISTS and was not found -> a path problem (PKG_CONFIG_PATH, + pc_path), not a missing package; +- the .pc is absent but its PROVIDER's is present (libdrm_intel -> libdrm) + -> the provider was built without that component, almost always because one + of its own dependencies was missing at the time; rebuild it; +- neither -> nothing provides it, and which-package/install is the way. + +Same shape as 1.14.22's undeclared symbol, and the same principle: the tools +cannot fix a build error, but they can nearly always say which KIND it is, +and that is most of the work. + +Note for a future session: libdrm-before-Xorg-Libraries is an ordering the +book's own dependency lists do not express, exactly like libaom needing cmake +(1.14.12). The command-scanning fix cannot see this one -- there is no +`libdrm_intel` in the commands -- so it stays a diagnosis rather than a +prevention. + +## Where the page's own commands left off (1.14.25) + +Xorg Libraries downloaded all thirty-two tarballs, verified every checksum, +and then: + + the list this build loops over is missing or empty: ../lib-7.md5 + +Its prepare step writes lib-7.md5 in the build root, then `mkdir lib && cd +lib` and downloads into it; the build loop reads `../lib-7.md5`, which only +resolves from INSIDE lib. `_pp_unpack` ran prepare_pkg and then `cd +"$BUILD_ROOT"` -- correct for a package with a tarball, which must be looked +for from the build root, and fatal for a page whose own commands ARE the +unpack. + +The directory prepare_pkg ends in is now remembered (`_pp_prepared_in`) and +recorded as the resume point for a page with no tarball of its own, so build, +install and configure all continue where the page left off. A package with a +real tarball still enters its unpacked source, as before; both are asserted. + +The runner's own guard was what caught this, and it named the file it could +not find -- which is the only reason the cause was findable at all. A test +that fails with the path it looked for is worth writing. + +## An avoided package is not walked into (1.14.24) + +The JT_JAVA plan had cups, Java, OpenJDK and apache-ant in it, every one +labelled "required by libreoffice" -- with libreoffice in --ignore. Ignoring +dropped the NAMED package and kept everything it alone wanted, so a sway +desktop was about to build a JDK. + +1.12.82 tried this and was reverted, for a real reason: it pruned the FINISHED +list, `blfs order` reports one parent per package, and graphite2 -- recorded +under an ignored parent but genuinely needed by harfbuzz -- was dropped: + + Dependency 'graphite2' is required but not found + +That note ended "pruning needs every edge, which this output does not carry". +So it moved to where the edges are. `walk_deps` does not descend into an +ignored anchor; anything another package needs is still reached through that +package. Measured on harfbuzz with libreoffice ignored: + + before 230 packages (planner: 463) + after 46 packages (planner: 101) + kept graphite2, icu, libxml2, fontconfig, curl + gone PostgreSQL, unixODBC, CLucene, Redland, OpenJDK, apache-ant, Cups + +A package named as the TARGET is still built, ignore list or not -- otherwise +`install libreoffice --ignore libreoffice` would silently do nothing. + +sway.stack's avoid list is eighty names long because that is what it took to +say this by hand; its own comment says what was meant -- "everything they +alone pull in goes with them". The list can shrink to the branches +themselves now, though it costs nothing to leave. + +The wider point: a revert with a reason attached is a bug report. 1.12.82 +recorded exactly why its approach could not work, and that sentence was the +design for the fix two years of versions later. + +## A commented line is not a command (1.14.23) + +A plan refused to start: + + JT_JAVA needed by 1 package(s), first: openjdk + packagemanager env --set JT_JAVA=... + +JT_JAVA appears on OpenJDK's page only inside the TEST block -- which the +generated script comments out -- and the page exports it there itself: + + # export JT_JAVA=$(echo $PWD/build/*/jdk) && + # jtreg/bin/jtreg -jdk:$JT_JAVA ... + +So `_required_env_guard` demanded a value for a variable nothing live would +ever read, and the person was asked to invent one for a test suite that does +not run. The stop added in 1.14.12 turned that into a refusal to build +anything at all -- a correct mechanism amplifying a wrong input. + +Two rules, both obvious in hindsight: + +- Scan only lines that will RUN. Everything commented is the book's prose, + kept for the reader, and not the shell's business. +- A variable the commands ASSIGN is not one to ask about, live or commented: + the page is telling us the value, not asking for it. (TeX Live sets + TEXLIVE_PREFIX="/opt/texlive/2025" itself.) + +Across the whole book the guards now come to exactly XORG_PREFIX (14 pages) +and XORG_CONFIG (23) -- the two the Xorg build environment page genuinely +asks the reader to export, and nothing else. Before this, every page whose +commented-out test block mentioned a capitalised name could stop a build. + +## An undeclared symbol usually belongs to someone else (1.14.22) + +cairo stopped with twelve errors, all like + + error: 'FC_HINT_NONE' undeclared (first use in this function) + error: 'FC_RGBA_RGB' undeclared ... + +Every one of them is fontconfig's. This is not a bug in cairo: it is a +DEPENDENCY whose headers are missing or too old. The box said "this is a +COMPILE error in the package's own source ... the tools cannot repair it" -- +true about repairing, wrong about the diagnosis, and it left the person to +work out whose symbols those were. + +The tools can look. The first undeclared symbol is grepped for under +/usr/include (bounded: `timeout 20`, `grep -rlm1`, first hit only): + +- FOUND -- the header is installed and the build did not see it, so the + problem is an include path or a missing .pc file; the header's directory + usually names the package, so `pkg-config --cflags fontconfig` and + `packagemanager info fontconfig` are offered. +- NOT FOUND -- nothing on this system declares it, so the package that + provides it is missing or out of date, and `packagemanager which-package` + is the way to it. + +The machine.conf advice stays for what it was written for: a real compile +error needing a build flag. + +Also fixed in the suite: the harness that tests the failure report sliced it +from a marker HALFWAY DOWN, so every branch above that marker -- including +this one and the compile-error branch it sits in -- was untested. It now +builds the whole report. A test that covers half of what it names is worse +than none, because it reads as coverage. + +## One door for every build (1.14.21) + +glib2's install stopped on + + PermissionError: [Errno 13] '/usr/share/gettext/its' + +Through lfs-helper that failure is REPAIRED -- grant-and-retry offers a +collector group for the directory and runs the phase again, which is exactly +what happened the first time. Through `packagemanager script install glib2` +the identical traceback repeated forever: that path built its own `su` and had +none of the wrappers, the file tracking, the failure box or the repair around +it. And `packagemanager script install <pkg>` is the command the failure box +itself prints. + +It now calls `lfs-helper build <user> --script <file> --phase <phase> --force` +when lfs-helper is present, falling back to the old direct run when it is not. + +This is the SECOND bug from that same split -- 1.14.7 was the build +environment not reaching the build, from the same reimplemented `su`. The +note then said "packagemanager script should call lfs-helper run_phase_as +rather than reimplement it"; it took a wasted evening to make that concrete. +When a note says two implementations will drift, believe it. + +Also here: this path still used install_<user>, the redundant copy 1.14.16 +removed everywhere else, so a failure said "ran: install_p_glib2" while +telling the person to edit install_GLib-2.86.4 -- and an edit to the file they +were pointed at did not affect the file that ran. It uses the canonical +versioned script now, and the regeneration message says what actually happened +instead of claiming that file "is never regenerated" while regenerating it. + +## Not every name is an LFS step (1.14.20) + + lfs-helper build glib2 --phase install --force + !! no script for 'glib2' at /usr/src/lfs-pkgusr/scripts/glib2.sh + From outside the chroot: + lfs build-system gen-chroot-scripts --run --overwrite + (then, running exactly that) + You are inside the LFS chroot, where it does not apply. + +"that hint was dumb", and it was. glib2 is a BLFS package: there is no +$SCRIPTS/glib2.sh and there never will be. cmd_build looked only at the step +scripts, assumed a missing name meant a missing STEP script, and prescribed +the cure for that -- a command that refuses to run in the chroot the person is +standing in. Meanwhile glib2's script was in /usr/src/pkgusr/p_glib2/, which +the failure box's own retry hint had printed a screen earlier. + +Two changes: + +- Before giving up, cmd_build looks for the package user's own script + (install_<Name-Version>, newest first, .edited and .bak excluded, then + install_last). A BLFS package now builds through `lfs-helper build <name>` + as well as through packagemanager. +- The message that remains asks WHAT THE NAME IS. `in_step_order` decides: + a real step with no script keeps the regenerate advice; anything else is + told to use packagemanager, or to check `lfs-helper list` for the step it + might have meant. + +1.12.30 fixed the same class of thing from the other end (the retry hint +printed the anchor, and `lfs-helper build` wanted the account). The lesson +both times: a tool that accepts a name from three different namespaces owes +the user an answer that says which one it looked in. + +## A typo in a group name is permanent (1.14.19) + +The collector-group prompt offered `nimgnu_gcc` or `nimgnu_lib`, the user +typed a name of their own and wrote `nimnu_gdb` -- one letter short of the +prefix. Perfectly valid as a group name, so it was created and the directory +handed to it; and nothing that looks for `${COLLECTOR_PREFIX}_*` will ever see +it again. + +The prompt sanitised CHARACTERS (1.13.x, after a leading space produced +`--nimgnu_gcc`) but never asked whether the name meant what it said. It now +compares the part before the first underscore with the configured prefix and, +when it is one edit away, offers the correction: + + 'nimnu' is not 'nimgnu' -- collector groups start with 'nimgnu_'. + use 'nimgnu_gdb' instead? [Y/n]: + +One edit exactly: a name that is nothing like the prefix is a deliberate +choice and is left alone. + +For the ones already made, `lfs-helper rename-group <old> <new> [--run]`. +Renaming is the whole repair because the GID does not change -- every +directory already handed to the group and every member still apply. It lists +what it will affect, refuses a name that exists, and has a dry run. + +The general point: a prompt that accepts free text is a place where a typo +becomes permanent state. Validating the SHAPE of an answer is not +second-guessing the user; it is the difference between a question and a trap. + +## Ask the book once, not once per package (1.14.18) + +"why is the lookup for the script taking that long? arent we somehow indexing +all the package names?" + +There is an index. `book_versions_all()` reads the whole book in one call and +returns anchor -> name_version -- and only `update` used it. `book_version()`, +which everything else calls, ran `blfs debug <anchor>`: a subprocess that opens +and reads the book cache to answer ONE lookup. The sway stack preview asks it +56 times; the planner asks it for every package it considers. Hence the +minutes of + + looking up pipewire in the book + +Measured on 26 names: 7.07s of subprocesses against 0.79s for the map, and +every lookup after the first free. The map is loaded on first use; a name it +does not have (an alias, a URL) still falls back to the old call and is +remembered afterwards. + +Worth noticing WHY this survived: the fast path existed and was used in one +place. A helper written for one caller does not advertise itself to the next +one, so the slow obvious thing stays. When something is slow, look for the +bulk version before writing one. + +## A fix that only applies to new records is half a fix (1.14.17) + +1.14.16 excluded /var/mail from the manifest SCAN, and the next run failed the +same way: Hatchling, `No module named 'pathspec'`, pathspec skipped again. +Every pkg.lst already on disk still had that one line in it, and the reader +still counted it -- one path, one alive, "installed". + +The exclusion belongs on both sides. `_manifest_has_real_paths` skips account +furniture (/var/mail, /var/spool/mail) as it reads, so a record written by any +earlier version is judged on what it actually claims about the install. A +mail-spool-only manifest now reads as not installed, and the planner rebuilds. + +The lesson, and it is the same shape as the "advice must work where the person +is standing" one: a fix that only applies going forward leaves every existing +system broken, and those are the systems people are running. When a rule +changes, ask what the data already on disk says under the new rule. + +## Build a wheel, install nothing (1.14.16) + +The user pasted an `ls` of p_pathspec and its manifest: + + /usr/src/pkgusr/p_pathspec/pkg.lst -> /var/mail/p_pathspec + +One path, created by useradd. Two independent faults had met. + +**The install step was commented out.** `_comment_orphan_pip_installs` +comments a pip install whose `--find-links` directory nothing builds -- right +for brotli, whose wheel step is optional and commented. But it judged each +command block ALONE, and for a Python module the wheel is built in the BUILD +phase and installed in the INSTALL phase. It never saw the producer, so it +commented the only line that installs anything. ALL 76 python-module pages +in the book: build a wheel, install nothing. Each block is now filtered with +the other in view. + +**The mail spool passed for evidence.** useradd creates /var/mail/<user> +owned by the package user, the manifest scan picked it up, and `state` +accepts one existing path as proof of an install. So a package that installed +nothing looked installed, the planner skipped it, and Hatchling died on +`No module named 'pathspec'` twenty-two packages later. /var/mail is excluded +from the scan: an empty record is honest, a record of the account's own +furniture is not. + +**A second opinion, from data already on disk.** Every generated script +carries the page's own Installed-programs/-libraries/-directories: + + installed_directories='/usr/lib/python3.14/site-packages/pathspec ...' + +`_declares_content_that_is_missing` checks them, and when a script names paths +and none exist the package is not installed, whatever install_last says. Only +when the page names paths -- otherwise the old rule stands. + +Note for the next session: every python module "installed" before 1.14.16 is +suspect. `packagemanager verify` or a rebuild of the module packages is the +way back. + +## Progress that is real should be visible (1.14.15) + +After three runs of the same stack: "its like always starting from fresh". + +It was not. The plan had gone 181 -> 151 -> 135 across those runs: forty-six +packages built and kept. But a re-run recomputes the whole tree and prints +only what is LEFT, so three real advances read as three identical fresh +starts. Nothing was wrong except that the tool never said what it had +already done. + +`build_install_plan` counts what it skipped as up to date, and the plan header +says so: + + ### Install plan (15 to build) ### + 4 of the 19 packages in this tree are already installed -- they are skipped. + +Worth stating as a rule, because this is the third report in this session of +the same shape (1.14.2 silence during the claim, 1.14.8 silence during +regeneration, this one): a long-running tool must account for the time it +takes AND for the work it is not doing. Silence reads as failure, and an +unchanged-looking plan reads as no progress. + +## One file to edit, one to run (1.14.16) + +"im also irritated cause of the different files in script and staged, is it +wanted that we have the p prefix?" No. One script had four names in one +home: install_Hatchling-1.28.0 (the one the user is told to edit), +install_p_hatchling (the one that actually RAN), install_hatchling (a staged +copy) and install_last (the record). Every failure box spent three lines +explaining which was which, and the file that ran was the one the user was +told not to edit. + +The canonical versioned script is runnable, so it is what runs. +install_p_<user> is removed when it is a copy and kept with a warning when +someone has edited it; a script already living in the owner's home is not +staged to a second name; and the failure box prints a `staged:` line only when +there really is a second file. What is left: install_<Name-Version> to edit +and run, install_last as the read-only record of what last ran. + +Also: a `ModuleNotFoundError` in a build log now names the module and the +command that installs it. pip prints a full traceback and ends with +"metadata-generation-failed", and the log scanner picked exactly the two lines +that say nothing. + +## Name a package the book actually has (1.14.14) + +The 1.14.12 stop worked, and then said: + + cargo needed by 3 package(s), first: rust-bindgen + packagemanager install rustc --run --recursive + +`rustc` is not an anchor in this book -- `rust` is (`Rustc-1.93.1`), and +`book_version("rustc")` returns None, so that command would have answered +"not in the BLFS book". Exactly 1.14.10 again, one layer down: the +tool -> package map was written from memory rather than checked against the +book. `ninja`, `meson`, `gperf` and `libtoolize` were wrong in a second way +-- they have no BLFS page at all, they come from LFS, so no command should be +offered for them. + +`_provider_of` resolves a tool through `book_version` and returns nothing when +the book has no such page; `_install_cmd_for` builds a command only when there +is one, and the report says plainly that meson and ninja come from LFS instead +of inventing an install for them. The test asserts EVERY named provider +resolves in the book, so the next entry added is checked by the suite rather +than by a user's failed build. + +The deeper fix: a tool whose provider is not in the plan AT ALL is now ADDED +to it. Reordering (1.14.12) can only help when both are present, and +rust-bindgen's page does not list Rust, so no ordering would have saved it. +The provider is a book package like any other -- it joins what was asked for +and the plan is rebuilt, once, so a provider that is itself missing something +cannot loop. Only with --recursive: a plain `install X` means that package +and nothing else. + +## The stack is the same door (1.14.13) + +Asked whether `packagemanager stack ... --run` gets the 1.14.12 ordering fix +too. It does, and the reason is worth recording: a stack's `book` entry runs +`_install_via_self`, which re-executes THIS tool as + + packagemanager install <target> --run --recursive [--ignore ...] + +so the plan, the ordering, the prerequisite checks and the stop all come from +the same code. One door, as HANDOFF has it elsewhere -- and a test now +asserts it, because the day someone reimplements the install inside cmd_stack +is the day the ordering fix silently stops applying to the only command +anyone actually runs. + +`--ignore-prereqs` is forwarded through it: a stack is a long unattended run, +and a decision to push past a missing tool has to reach the install it +delegates to. + +Also corrected in the advice given to the user: with 1.14.12 there is no +longer any reason to install cmake by hand first. The plan contains it and +now orders it correctly, which is the whole point of the fix. + +## Build the tool before the package that calls it (1.14.12) + +The user asked the right question: "are we even sure we recursively install +programs?" Yes -- and that was never the problem. Measured on their plan: + + libaom-3.13.1 position 89 + CMake-4.2.3 position 151 + +--recursive worked; CMake WAS in the plan. It was ordered 62 packages too +late, because the order comes from the book's dependency lists and libaom's +page lists no Required at all. Right answer, wrong facts -- 1.14.9 read the +scripts' commands but only to WARN, when the same reading should have fed the +ordering. + +`_order_by_build_tools` moves a package down to just after the one providing +the tool its commands call. The CONSUMER moves, never the provider: moving a +package later keeps every dependency it already has in front of it, while +moving one earlier could jump it over its own (CMake needs cURL). Only when +the tool is not already on the system, and the plan is now ORDERED BEFORE IT +IS PRINTED -- the list is a promise about what happens next. + +A tool the plan itself builds is no longer reported missing; it was only late. + +Two more from the same report: + +- `resolve_script` has three exits and only two checked prerequisites. The + third -- a script already in the package user's home -- is every package + that has been attempted once, which is why libaom's missing cmake was never + announced at plan time. +- A plan that cannot finish no longer starts. Warning and building anyway is + how a missing cmake was found twenty-two packages deep, long after the + warning had scrolled away. `--ignore-prereqs` overrides. + +Also answered: the p_cmake home with scripts and a src/ but nothing installed +is a HUSK, and expected. The account and its script are created before the +build; that build failed (cURL missing), so the home is what the attempt left. +It is not a sign that cmake is half-installed. + +## The book sets them together (1.14.11) + +Reported seventeen packages into a plan: + + this package needs XORG_PREFIX to be set + +The person had set XORG_CONFIG hours earlier, from the same BLFS page that +sets both: + + export XORG_PREFIX="/usr" + export XORG_CONFIG="--prefix=$XORG_PREFIX --sysconfdir=/etc ..." + +Nothing mentioned the other half, for a reason worth remembering: `env --set` +asks the CACHED SCRIPTS which variables matter, and at that moment every one +of them was from an older blfs, so `_env_wanted_by_scripts` returned nothing +and the "ignored 183 script(s)" line was the only trace. A check whose input +was empty reported nothing and looked like a pass. + +Three changes: + +- `_ENV_COMPANIONS`: XORG_CONFIG and XORG_PREFIX are each other's other half, + so setting one names the other, with the book's own value. +- When EVERY cached script is stale, say that nothing could be determined and + list the common variables, instead of a note about ignored files. +- The plan warns once per VARIABLE (it was once per package -- the same + XORG_PREFIX notice dozens of times, scrolled past) and prints a summary + beside the missing-tools one: which variables, how many packages want each, + and the exact command. + +Same shape as 1.14.9's missing build tools, and the fix belongs beside it: +everything knowable before the first package is built should be said before +the first package is built. + +## A suggested command has to work (1.14.10) + +Straight out of 1.14.9. The new hint said + + packagemanager install cmake --run + +which builds cmake ALONE, and cmake then fails with + + CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! + +because cURL is one of its four RECOMMENDED dependencies and BLFS builds +CMake with --system-libs. packagemanager had already said this itself, above +the plan -- "12 dependencies of CMake-4.2.3 are NOT installed ... to build +them too: packagemanager install cmake --run --recursive" -- and the hint sent +the person past its own correct advice. + +Every suggested install now carries `--recursive`, here and in the +"these are NOT INSTALLED, to build them" hint. The test greps for a bare +`install %s --run` anywhere in the tool, so the next one added is caught. + +The pattern across 1.14.3, 1.14.7 and this one: a suggestion is code that +runs on someone else's machine, and it has to be correct there, not just +plausible here. + +## The commands are the fact; the dependency list is an opinion (1.14.9) + +Reported from a real chroot, three packages into a 151-package plan: + + install_libheif: line 33: cmake: command not found + +libheif's BLFS page has NO Required section -- only Recommended (libaom, +libde265, x265). CMake reaches libheif only THROUGH x265, and x265 was +already installed on that system, so nothing pulled cmake into the plan. The +resolver was right about the book; the book does not say what the commands do. + +The script is right there and its commands are the truth. +`_warn_missing_build_tools` reads the command word of each line, keeps the +build-system drivers (cmake, meson, ninja, cargo, autoreconf, qmake, ...) and +reports the ones `shutil.which` cannot find, with the BLFS package that +provides each. Scanned across 200 generated scripts it flagged only real +absences -- no parse noise. + +Warned once per TOOL, not per package (37 packages calling cmake would print +the same three lines 37 times), with a block under the plan naming every +missing driver, how many packages want it, and one command to install them +all. + +Fixes a hole 1.14.8 opened as well: `_warn_unset_env` lived only on the +freshly-generated path in resolve_script, so pre-generating a plan in one pass +skipped it. Both checks are now `_check_script_prereqs`, called for a script +whether it was just written or came from the cache -- they belong to USING a +script, not to writing one. + +## One book load, not one per package (1.14.8) + +Reported from a real chroot: a 151-package plan printed + + cached script is from blfs 1.14.3, this is 1.14.7 -- regenerating + +twice, then sat silent for five minutes and was killed. + +`resolve_script` generates ONE SCRIPT PER PACKAGE, each a `blfs script` +subprocess that loads and indexes the whole book again. With a warm cache +that is ~0.2s each; on a fresh system, where the first call indexes a 25MB +book, it is the difference between a wait and a hang -- and it presented as a +hang, because the output was one line per package with all the work between +the lines. + +`blfs script` has ALWAYS taken `nargs="+"` and used only `_t[0]`. It now +writes every target from one book load (5 packages: 1.14s as separate calls, +0.27s in one), and `_prewarm_scripts` generates everything a plan needs in a +single pass before the plan is printed, announcing the count first. A local +script in a package user's home still wins and is never touched. + +Also fixed, same family as 1.14.7: `_warn_unset_env` asked `os.environ` only. +After `packagemanager env --set XORG_CONFIG=...` every Xorg package in the +plan was still announced as missing it -- and the cure it printed, `export`, +is exactly the one that cannot work, since `su -` drops the caller's +environment. It now reads /etc/pkgusr/build.env, the file the build actually +reads, and prints the command that writes there. + +That is three bugs in two releases from the same root: a value has one home +(/etc/pkgusr/build.env) and several readers, and each reader had its own idea +of where to look. + +## The build environment has to reach the build (1.14.7) + +The end of the util-macros report, and the actual cause of all of it. The +person ran + + packagemanager env --set XORG_CONFIG=... + +was shown it written to /etc/pkgusr/build.env and echoed back in the table, +and the very next command still stopped with "this package needs XORG_CONFIG +to be set". + +lfs-helper's `run_phase_as` pastes `build_env_pairs` in front of the command +it runs, so the pm-install path had this right. packagemanager's +`script <phase>` path builds its OWN `su` and passed nothing -- it set +variables in subprocess's `env`, and `su -` starts a login shell that drops +them. So a file written by one command was honoured by one of the two paths +that run builds. + +Now the same pairs go INSIDE the command (`env K='v' ... bash install_x +build`), single-quoted, which is the trap `build_env_pairs` already documents: +XORG_CONFIG is a flag list, and unquoted it split into words with the second +run as a command. The header line also names what was applied and where it +came from, so a missing variable is visible before the build starts rather +than after it fails. + +Two tools, two implementations of "run a phase as the package user", is what +allowed this. Worth collapsing: `packagemanager script` should call +`lfs-helper run_phase_as` rather than reimplement it. Not done here -- it +touches the phase records and the log plumbing too -- but this is the second +bug from that split (the first: phase records, 1.11.55). + +## A failure has to say where (1.14.6) + +Continuing the util-macros report. Its build had stopped for a missing +XORG_CONFIG, so the source was unpacked and unconfigured. Running the install +phase then gave, in full: + + make: *** No rule to make target 'install'. Stop. + !! phase 'install' FAILED (exit 2) + phase 'install' failed for p_util-macros (exit 2). + logs: /usr/src/pkgusr/p_util-macros/log + +The package's own words for "there is no Makefile", and nothing else: not +which of the three copies of the script had run, not the file the person is +told to edit, not the source directory, not the actual reason. The user's +words: "we do not even tell the user the file path". + +Two halves. + +The RUNNER now records that a build FINISHED (`.pkgusr-<nv>.built`, written +only on a zero exit) separately from where it finished (`.cwd`, written even +on failure so a retry can resume). A later phase with no such record says so +before the package's tools fail in their own vocabulary -- a warning, not a +refusal, because a tree built by an older runner has no record either and +refusing there would be wrong. + +packagemanager's report names `ran`, `edit`, `source`, `logs` and `retry`, +and for install/configure/test adds that no later phase can stand in for a +build that never finished. `_versioned_script_in` finds the editable +install_<name>-<version> (not the copy, not a .edited backup) and +`_unpacked_source_dir` the tree, so the three files a person has to tell +apart are named for them. + +## One package, several filenames (1.14.5) + +Reported from a real chroot. util-macros had unpacked and built; the retry +of just the install phase said + + !! no unpacked source in /usr/src/pkgusr/p_util-macros/src + +with the unpacked source sitting right there. + +1.14.0 named the build markers from `$0`. This project runs ONE script under +three filenames by design -- the book copy (install_util-macros-1.20.2), the +staged copy (install_util-macros) and the account's copy +(install_p_util-macros), and the failure box says so itself. Unpack recorded +`.pkgusr-install_util-macros.dir`; install looked for +`.pkgusr-install_p_util-macros.dir`. The old scaffolding keyed markers to the +PACKAGE (`.cc-dir-<name>`, `.pm_build_cwd`), which is why this never happened +before -- a regression introduced by taking an identity from the wrong place. + +Markers are now `.pkgusr-<name_version>.{dir,cwd}`: name_version is in every +copy of the script and is the same in all of them. + +And `_pp_enter_source` no longer gives up when the marker is missing. It +looks for `$BUILD_ROOT/$name_version`, and failing that for a single unpacked +directory, before saying "run unpack" -- which is the wrong answer when the +tree is in front of it, and for a big package throws away a compile that +succeeded. Markers written by 1.14.0-1.14.4 are recovered this way rather +than being re-unpacked. + +`clean_stale_source` clears `.pkgusr-*` by glob now: the build root belongs to +one package, so whatever is in it is that package's, under whichever naming. + +## The last thing you see cannot be the thing that breaks (1.14.4) + +Reported from a real build. util-macros stopped because XORG_CONFIG was not +set, the box diagnosed it correctly, printed the one command that fixes it, +and then: + + ! Edit the script, then retry just this phase: + /usr/bin/lfs-helper: line 5596: _retry: unbound variable + +`$_retry` -- the account name a retry command has to use -- was computed +INSIDE the permission-failure branch and used by three hints outside it. +Under `set -u` every failure that was NOT a permission problem crashed the +reporter on the line telling you how to retry. It has been there since +1.12.30, hidden because the branch that sets it is the common case. + +Now resolved once, before any hint runs. The regression test builds the +reporter into a standalone function and runs it under `set -u` against six +kinds of log -- env-var, permission, compile error, missing program, missing +library, and one it does not recognise -- asserting no crash and a retry +command in each. That is the shape to keep: a diagnostic path is code, and +the only code guaranteed to run on a bad day. + +Also quieted: `add_package_user` takes no home argument, so every account it +creates lands in /usr/src/<name> and is moved to /usr/src/pkgusr/<name> a +line later. That is expected and instantly repaired, but it was reported as + + misplaced home: p_util-macros is recorded at /usr/src/p_util-macros + every tool looks at /usr/src/pkgusr/p_util-macros + +opening every single install with two lines of alarm about a non-problem. +`_repair_misplaced_home` now takes a third argument for "this account was +just created": in that case, and only when the recorded home is exactly +add_package_user's own, it is one dim line. A misplaced home found any other +way is still a warning -- there it means a real account has been pointing at +the wrong directory for who knows how long. + +## The bootstrap circle, explained from where you stand (1.14.3) + +Reported from a real chroot, at `packagemanager bootstrap --run` stage 2: + + wget: there is no BLFS book on this system to look it up in. + Download it (needs working networking): + blfs fetch + +Told to a system whose entire problem is that it has no download tool. That +is the circle itself -- fetching needs wget, wget is looked up in the book, +the book is what is missing -- and the advice was to go round it once more. + +The cause is one step earlier and on the other side. lfs's +`bootstrap_sources()` asks `packagemanager bootstrap --sources`, which +resolves wget's URL FROM THE BOOK (`blfs sources wget`). A host with no +cached BLFS book therefore staged no wget tarball into $LFS/sources either. +Both halves go missing together, from one cause, and the tree cannot supply +either from inside. + +Two changes: + +- `install-tools` said "BLFS book: (none cached here)" as a dim aside in a + list of successes. It is now a `warn` that names the consequence, because + it IS the later failure, announced early enough to fix while there is still + a network. +- The no-book message asks what this system can actually do. With a download + tool: fetch or import, as before. Without one: the recovery is on the + HOST, both halves in one trip, and it says which of them is really absent + rather than letting them be discovered one at a time. + +The general shape, and it has come up before: advice has to be answerable +from where the person is standing. A remedy that requires the thing that is +broken is not a remedy. + +## Silence is not progress (1.14.2) + +Reported mid-build, at chapter 8 GCC: + + ===== [37/106] gcc ===== + # user : p_gcc (package user) phase: all + +...then ten minutes of nothing, one bash process holding a core, and no child +process at all. Between those two lines sits exactly one thing: +`_claim_earlier_stages`, which takes over the files an earlier stage of the +same package left behind. It forked three or four times PER PATH -- a +command substitution for `strip_host_prefix`, a `stat`, a `chown` -- across +every earlier stage's manifest. For GCC that is gcc-pass1 + gcc-pass2 + +libstdcpp, and it prints nothing until it is finished. + +Measured: 3,000 paths took 9.3s the old way and 0.08s now -- one `stat` and +one `chown` for the whole manifest, batched through `xargs`, with the +owner filter moved into an `awk` that reads the batch. Semantics are +unchanged: a path another package owns is still left alone, and a path +already owned by the account is still not counted. + +Two things this also fixed: `libstdcpp` was never in the list of earlier +stages (chapter 5 builds "Libstdc++ from GCC", and `pkg_owner_name` maps it +to the gcc account), so those files stayed root-owned; and the step now says +what it did. + +The lesson is the one in the title. A step that can run for minutes has to +say it started, not only that it finished -- `detail` before the work, not +`say` after it. + +## A fresh build on a host we built (1.14.1) + +Reported from a real fresh build, on the user's own LFS machine: + + root ~ # lfs run + You are inside the LFS chroot, where `lfs build-system run` does not apply. + +It was not. `inside_chroot()` fell through to its last-resort inference -- +"/usr/src/lfs-pkgusr exists AND /mnt/lfs/usr/src does not" -- and both halves +were true on the host: the machine was BUILT by these tools, and the freshly +mounted partition was still empty. + +1.11.x had already fixed the first half of this ("an inference that is true of +the destination cannot identify the journey") and added `LFS_IN_CHROOT`. The +guard bolted onto the old inference was the part that stayed wrong, and it +failed at the one moment the inference is consulted at all: before a build has +put anything in the tree. + +The kernel knows. A chroot IS a root that is not PID 1's, so +`_root_differs_from_pid1()` compares `/` with `/proc/1/root` and answers +outright. Order: `LFS_IN_CHROOT`, then the kernel, then (only when the kernel +cannot be asked -- no /proc, or not root) `$LFS`, then the inference, now +against the CONFIGURED mount point rather than a hardcoded `/mnt/lfs`. + +Being wrong toward "inside" blocks a host; being wrong toward "outside" fails +later with a message about the real problem. So the fallbacks all lean +outside, and only `LFS_IN_CHROOT` or the kernel can say "inside". + +## The runner moved out of every script (1.14.0) + +A generated install script was ~230 lines for five lines of book commands: +source lookup, the symlink farm, the unpack, a heredoc-per-phase trick, build +markers, the dispatcher and its usage text -- emitted by FOUR generators +(`_crosschain_script_body`, `_plain_root_script`, blfs `render_script`, +packagemanager `_TEMPLATE`) in three dialects, and text-patched by +`_chrootify` for the chroot. Every consumer then grepped for a piece of that +scaffolding (`#### CONFIGURE ####`, `.cc-build-*`, `.pm_build_cwd`). + +Now a script is its variables at the top, one function per phase +(`prepare_pkg`, `build_pkg`, `install_pkg`, `configure_pkg`, `test_pkg`), and +one last line: + + . "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@" + +`lfs-phases` is the shared runner. It ships with the tools (Makefile, +`TOOLCHAIN_SCRIPTS`, `PKGUSR_TOOLS`), is copied into the store for the host +`lfs` user, and is found on PATH or through `PKGUSR_LIB`. What it keeps from +the old scaffolding, and why: + +- **Each phase is its own process.** bash disables `set -e` inside any + function reached from a tested context and subshells inherit that, so a + failing make was silently ignored (the day-long bug). The runner re-executes + the script itself as `PKGUSR_PHASE=build bash script </dev/null`; the phase + then runs at top level under `set -e`, and stdin is closed so a book command + that reads it (Expect's PTY check) swallows nothing. This is why a script + must be run BY PATH: `cmd_bs_crosschain` no longer pipes it through + `bash -s`, and the runner refuses stdin with a message. +- **`pkgusr_stage`** (`cross` | `chroot` | unset) replaces `_chrootify`'s text + patching: it decides `$LFS`, `$LFS_TGT`, `/tools/bin` on PATH, and the + defaults for `SOURCES_DIR` / `BUILD_ROOT`. +- **One marker pair**, `.pkgusr-<step>.dir` and `.pkgusr-<step>.cwd` in + BUILD_ROOT; `clean_stale_source` and `clean-sources` sweep both the new and + the old names. +- **Function bodies are not indented.** The books are full of heredocs and + an indented terminator is no terminator. A body of comments alone gets a + `:` -- bash rejects an empty function. +- **The phase names keep their `_pkg` suffix.** `install` is coreutils and is + called inside install phases; `test` is a builtin. +- **Sourcing a script does nothing.** pm-install sources scripts for their + metadata; `pkgusr_run` returns when `BASH_SOURCE[1]` is not `$0`. Every + name in the runner carries a `pkgusr_`/`_pp_` prefix because it comes along + into the sourcing shell. + +Consumers that changed: `script_phases_of_file` and `lfs install --reinstall` +look for `^configure_pkg()` (the marker still recognised for old scripts); +`_version_from_script` now reads `name_version=` -- it read a `# package:` +comment no generator ever wrote, so every pkgusr.info `version=` was empty; +`run_phase_as` names a missing runner in one message instead of letting `su` +report it; the test switch is `PKGUSR_TESTS=1` (`LFS_RUN_TESTS`/`PM_TEST` still +accepted); packagemanager's stack generator fills function bodies and +`extract_phase_cmds` reads them. + +Two bugs the old scaffolding hid, fixed here because they surfaced the moment +it was gone: blfs's `_repair_dangling_operators` ran before five later +commenting filters, so a body ending `cmd &&` + commented line was a syntax +error in install_pkg (cachecontrol and 145 others failed `bash -n` under the +OLD generator too -- in build_pkg the trailing `&&` chained into the +scaffolding's `pwd > .pm_build_cwd` line and passed by accident); and the +BLFS mirror of last resort was the literal `13.0`, now `pkgusr_book_mirror` +from the book file's name. + +Formats: lfs v18, blfs v17. Old scripts are self-contained and keep working; +`--regenerate` writes the new shape. Every LFS package (81) and every BLFS +page (954) in the 13.0 books renders, parses and sources. + +## stdout is the return value (1.13.69) + +1.13.68's new messages made the real cause readable at last: + + chgrp # /usr/share/gstreamer-1.0/presets is inside /usr/share/gstreamer-1.0, + which is shared through nimgnu_gst\n# -- using that group\nnimgnu_gst + /usr/share/gstreamer-1.0/presets FAILED + +choose_collector_group ANSWERS on stdout. 1.13.46 added two explanation +lines with plain `say`, so the caller captured the whole paragraph as the +group name and chgrp failed on it -- silently, which is why it read as +"nothing to grant" for six rounds. The prompt body was always safe: it lives +inside a `{ ... } > /dev/tty` group, for exactly this reason. + +Both lines now go to stderr, and the suite captures the function's output and +requires it to be a bare group name. + +THE LESSON, and it is mine to keep: the bug was introduced by a FIX (1.13.46, +inheriting the ancestor's group -- which was correct and the user's idea) and +then hidden by the silence of the code around it. Every one of the six +rounds added a message; the round that added the RIGHT message solved it. +When a repair is silent, make it speak before changing what it does. + +## The prompt could not run, so nothing was chosen (1.13.68) + +FOUND IT. The same failure appeared on a new directory +(/usr/share/gstreamer-1.0/presets, owned by p_gst10-plugins-good) after the +user fixed the first one by hand -- same shape, same "nothing to grant". + +choose_collector_group PROMPTS when the group does not exist yet. During a +build stdin is not a terminal, so the prompt cannot run and the function +returned NOTHING. Then: + + chgrp "" /usr/share/gstreamer-1.0/presets -> fails + grant_dir_access -> return 1, no message + the loop -> "nothing to grant" + +Two fixes, both about not being quiet: + * an empty name falls back to the OWNER-NAMED group (the recommended answer + anyway) and says why: "could not ask ... (no terminal here) -- using + nimgnu_gst10-plugins-good, named after its owner"; + * a failed chgrp reports itself, with `getent group` to check. + +Six rounds on this one package, and the cause was a prompt in a place that +cannot prompt. 1.13.33 made --yes stop answering name questions -- correct +for a person at a terminal, and it left the BUILD with no way to answer at +all. Worth re-reading every prompt for "what happens when nobody can +answer": the answer must never be "nothing". + +## Extras last, except when they are dependencies (1.13.67, cont.) + +Adding the WebRTC chain broke a rule this session had made: 1.12.75 put all +git entries LAST so --no-git could skip the extras. libnice and friends are +git entries that must come FIRST, because gst-plugins-bad compiles webrtcbin +only if libnice is present. + +Two corrections: + * the invariant is that nothing after the extras MARKER is a book entry -- + not that no git entry may precede a book one; + * `--no-git` now says what it costs when it skips them: + NOTE: libnice, libsrtp, usrsctp, webrtc-audio-processing are skipped, + and gst-plugins-bad / webkitgtk link them. Built without them there + is no WebRTC ... adding them later means recompiling both. + +Which matters immediately: the user has been running --no-git all session. + +## WebRTC has to be decided before the build (1.13.67) + +The user asked whether streaming support can come later. It cannot, cheaply: +webrtcbin -- the element a browser uses for WebRTC -- is compiled into +gst-plugins-bad ONLY if libnice is present, and webkitgtk links the result. +Adding it afterwards means recompiling both. + +BLFS 13.0 carries NONE of the pieces, so they are git entries, placed before +gst10-plugins-bad and webkitgtk: + + libnice ICE / NAT traversal -- WebRTC cannot work without it + libsrtp encrypted media transport + usrsctp data channels + webrtc-audio-processing echo cancellation (the log's "found: NO") + +The camera side was already there: v4l-utils for the device, pipewire for +routing. + +UNTESTED: the build lines follow each project's README and none has been +built here. Two things learned writing them: + * `git` entries must be ONE line -- the parser has no continuations + ("No escaped character" at the first backslash); + * `prog=false` as a placeholder made all four report themselves INSTALLED, + because `false` is a real program. The suite now rejects it, and checks + that libnice precedes the two packages that link it. + +STILL OPEN, and worth saying plainly: BROWSER SCREEN SHARING on Wayland goes +through xdg-desktop-portal, which this desktop avoids. Camera capture and +playback do not need it. If screen sharing is wanted later, the portal and +its wlr backend are the addition -- and that one CAN be added afterwards +without recompiling anything. + +## Interactive, like the others (1.13.66) + +Per the user: every config command should have an interactive mode. Two +already did -- `packagemanager config` asks for what is missing, and +`packagemanager env --ask` walks the build variables. The one that did not +was the one a person most needs to set deliberately: the NEW SYSTEM'S. + + lfs config --target --ask + prefix for package users in the new system (p -> p_mesa; empty -> mesa) + pkgusr_prefix [this host's value]: p + folder under the base for package homes (pkgusr) + pkgusr_subdir [this host's value]: packages + +Enter keeps the current value, '-' clears it (which MEANS none, and is shown +as "(none)" afterwards -- the absent/empty distinction from 1.13.64). Each +key is explained where it is asked, rather than in documentation nobody has +open at that moment. + +REMAINING for the unification: `lfs config` itself (the session keys) is +interactive only through `lfs build-system session --run`. Once the storage +is unified (1.13.56), all four should offer the same three modes -- show, set +one key, walk them all -- with the same layout. + +## A config for the build system, written before it exists (1.13.65) + +The user's request, and it was the missing piece: a file on the HOST that IS +the new system's config, so it can be configured before there is a tree to +configure -- and `lfs run` copies THAT in rather than the host's own +settings. + + /usr/share/lfs/target-config.json + + lfs config --target # show it + lfs config --target pkgusr_prefix p # set a key + +Precedence, printed where it is applied: the host's session config is the +floor, target-config.json overrides it, and a config the TREE already has +wins over both -- it is the system's own choice, made after that file was +written. `lfs run` lists what it took from the target config as it copies. + +Laid out like `packagemanager config --show`: the file, then value and +source. That is the first step of the user's second request -- ONE LAYOUT +AND ONE VOCABULARY for the config commands across all four tools. The rest +belongs with the unification proposed under 1.13.56: while the shared keys +still live in two files with precedence rules, the commands cannot be made to +look alike without lying about which file wins. Do that first, then align +`lfs config`, `lfs config --target`, `packagemanager config --show` and +`packagemanager env` on the same shape. + +## Which config is which, and an empty prefix (1.13.64) + +The user asked whether /usr/share/lfs/config.json is the build system's +config. It is not, and the distinction matters: + + /usr/share/lfs/config.json the HOST's -- drives the build + /mnt/lfs/usr/share/lfs/config.json the BUILT system's -- decides for it + once it exists (1.13.51/63) + +The session shows the host's, correctly (it holds the device, the mount, the +makeflags). What it also holds is the prefix the build will CREATE ACCOUNTS +with -- and on this machine that is now empty, because the user emptied it +on the host. With no tree config yet, nothing overrides it. + +"(unset)" read as "nothing chosen yet" when it was an empty value on +purpose. The two are now distinguished: + + pkgusr_prefix = (none -- accounts will be named without a prefix) + pkgusr_prefix = (unset -- will ask) + +Third time this exact ambiguity has cost a round (1.13.26 the reader, +1.13.63 the source, this the display). Anywhere a setting may legitimately +be empty, ABSENT AND EMPTY MUST LOOK DIFFERENT. + +## The session config took the host's prefixes (1.13.63) + + Session config: + pkgusr_prefix = (unset) + +That is the empty prefix the user had just set on their HOST, in the session +that creates the accounts of the NEW system. 1.13.51 stopped the host's +prefixes reaching the chroot's environment; this is the same leak one step +earlier, in the session config itself -- and it would have named every +account of the build system after a decision made on a different machine. + +A tree that already has a config now decides its own prefixes there, and +says so: + pkgusr_prefix = p <- from the tree's own config + +And the header names the file, because "which config is this?" was exactly +the question that could not be answered: + Session config: (/usr/share/lfs/config.json) + +Two leaks found by the user asking the same question three ways. Worth +asking it of every place that reads a shared key: WHOSE value is this, the +machine I am on or the machine I am building? + +## A retry hint that cannot work (1.13.62) + +The user asked whether the script-version warning had been the cause all +along. No -- and the question was fair, because our own failure box sent +them there: + + ! lfs-helper build gst10-plugins-bad --phase all --force + !! no script for 'gst10-plugins-bad' at .../scripts/gst10-plugins-bad.sh + !! These build scripts were generated by 1.12.30 ... + +`lfs-helper build` takes the ACCOUNT (p_gst10-plugins-bad). The box printed +whatever name it was called with, and for a BLFS package that is the anchor, +so the command fell through to the LFS BASE-STEP path and blamed chroot step +scripts that have nothing to do with BLFS packages at all. + +The box now resolves the name to a real account before printing it. + +THE ACTUAL CAUSE of that package, for the record: /usr/include/gstreamer-1.0/ +gst/audio was group p_gst10-plugins-base (mode 775) and the installing +package is in install, nimgnu_gir and nimgnu_gst -- not in that group. The +user fixed it directly: + chgrp nimgnu_gst .../gst/audio && chmod g+w .../gst/audio +which is exactly what grant_dir_access should have done: its parent gst/ is +already nimgnu_gst. It reported "nothing to grant" instead, so ONE of its +six early returns fires wrongly. NEXT SESSION, FIRST TASK: run it under +`bash -x` and find which -- five patches to that function were made from log +text and three of them addressed cases that were not happening. + +## The directory was never the problem (1.13.61) + +One line of new output ended five rounds of guessing: + + /usr/include/gstreamer-1.0/gst/audio -- nothing to grant (already writable) + +The package CAN write there. What it cannot do is finish installing over a +FILE owned by another package: meson copies the contents (the directory is +writable) and then chmod/chown/utime the destination, which only its owner +may do. Python raises EPERM; meson prints "Unhandled python OSError" with no +path; and for five rounds that looked like a directory-permission problem +because that is the only class the repair knew. + +It is the same shape as tar's "Cannot change mode" (1.13.35) -- and the same +answer as the rest of this system: OWNERSHIP FOLLOWS THE INSTALLER. +meson_takeover_from_log hands the files a package is installing over to that +package, using meson's own "Installing SRC to DIR" lines to know which. +root's files are never taken. + +The lesson worth keeping: the repair had ONE explanation for a failure and +kept applying it. The fix that mattered was 1.13.60's -- making it say what +it decided -- because that is what turned a guess into a diagnosis. + +## A group on a leaf is no use if the chain is closed (1.13.60) + +The user's own `ls` chain, and their conclusion, were the fix: + + /usr/include/gstreamer-1.0/gst/ p_gstreamer10:nimgnu_gst drwxrwxr-x + /usr/include/gstreamer-1.0/gst/audio p_gst10-plugins-base:... drwxrwxr-x + +"isn't it normal that a collector group will not work without its parent? so +why not directly set it until we get to an accessible parent" -- yes. +grant_dir_access now walks UP from the directory it is granting and makes +each ancestor searchable, stopping at the first one that already is. Search +happened to be open in this tree, but a package that tightens its own +directory would break everything installing below it. + +AND, the reason four fixes were made by guessing: the grant loop printed a +header and then NOTHING when every candidate was skipped, so a failing +package looked like a repair that had never run. Every candidate now reports +its outcome, including "nothing to grant (already writable, or not a +directory)". + +If gst-plugins-bad still fails after this, the output will finally say which +directory was considered and what was decided about it -- which is what the +last four rounds were missing. + +## Grant every destination meson announced (1.13.59) + +The plan was "the wrapper that fails records the directory". It cannot: +MESON INSTALLS THROUGH PYTHON, not through install/cp/mkdir, so nothing on +our side ever sees the EACCES. That is why four rounds of log-parsing were +needed and why the fifth would have been too. + +What meson does give us is every destination it ANNOUNCED before dying. +Granting all of them settles the package in one round instead of one round +per header directory, and costs nothing: grant_dir_access now returns +immediately for a directory the package can already write (`su -c "test -w"`), +so the package's own directories and shared ones it already belongs to are +skipped rather than growing new groups. + + Installing ... to /usr/include/gstreamer-1.0/gst/uridownloader + Installing ... to /usr/include/gstreamer-1.0/gst/audio + ERROR: Unhandled python OSError + -> both granted, one round + +STILL TRUE for the wrapper-based builders (make/install/cp): recording the +directory at the point of failure would beat parsing, and those DO go through +our wrappers. Worth doing for them -- but meson needed the other answer, and +meson is most of BLFS now. + +## One grant per directory, not per tree (1.13.58) + + auto-repair: rounds run: 2 + /usr/include/gstreamer-1.0/gst/audio drwxrwxr-x p_gst10-plugins-base + +The repair ran twice, granted something each time, and the build failed on +the same directory. The dedup was the bug: it collapsed every failing path +to the top of its tree, so once ONE directory under +/usr/include/gstreamer-1.0 had been granted, every other directory under it +was skipped -- including the one that failed. + +A collector group applies to the directories it is put on -- the prompt says +exactly that -- so each distinct directory needs its own grant. The problem +the tree-top dedup was written for (the same directory named by nine +different paths) is solved by deduping on the directory itself. + +FOURTH fix to this one repair path tonight: relative paths (1.13.36), +meson's wording (1.13.49), the ancestor's group (1.13.46), and now the dedup. +Every one was a patch to reconstructing information the BUILD already had. +The design note stands: the wrapper that fails should record the directory it +failed on, and the repair should read that instead of parsing prose. + +## Flags for the layout (1.13.57) + +The directories became configurable in 1.13.52/53 and could still only be +changed by editing a file: + + packagemanager config --base-dir /usr/src + packagemanager config --pkgusr-subdir packages + packagemanager config --cfguser-subdir config + +with the prefix guard's twin -- moving the layout does not move the homes: + + account homes already exist under the current layout: + /usr/src/pkgusr (37 entries) + Changing the layout does NOT move them. Every tool would look in the + new place and find nothing. + +Also: the report named /usr/share/lfs/config.json whatever it had actually +read. It names the real file now, which matters as soon as LFS_STORE points +elsewhere -- and pointed at the wrong file while explaining where settings +come from. + +## Saved where nobody reads (1.13.56), and the config question + + packagemanager config --pkgusr-prefix "" + pkgusr_prefix -> '(no prefix)' + Saved /etc/pkgusr/packagemanager.conf + packagemanager config --show + package-user prefix p_ the lfs config (/usr/share/lfs/config.json) + +The setter wrote one file; the reader reads the other. Nothing changed and +it said Saved. _save_shared_key() now writes the AUTHORITATIVE file (the lfs +config, when it exists) and says which file it wrote. + +### THE DESIGN QUESTION THE USER ASKED -- worth doing properly + +"Is there even a good reason for those different config paths?" No. It is +history: `lfs` has its store (config.json, travels with the tree) and +packagemanager has its own file, and the SHARED keys ended up in both with +precedence rules. Every config bug this session came from that: + + * an empty prefix from the environment beating the config (1.13.26) + * the host's prefix baked into the chroot's exports (1.13.51) + * the layout readable from one file only (1.13.52/53) + * a setting saved where nobody reads it (this) + * and `--show` needing a "source" column at all, to explain the mess + +PROPOSED, for a fresh session: + * /usr/share/lfs/config.json is the ONLY home for system-level settings: + the prefixes, the layout, the books, the device/mount. It travels with + the tree, which is what makes a build system's config its own. + * /etc/pkgusr/packagemanager.conf keeps only what belongs to the person + using the tool on this machine: editor, difftool, main_user. + * packagemanager reads the lfs config directly (it already can) and stops + duplicating those keys; the precedence rules disappear with them. + * the environment stays as an override for one-off runs, and `--show` + keeps the source column for exactly that reason. + +That is a contained change -- one reader, one writer, one file -- and it +removes a whole class rather than the five instances above. + +## purge deleted the wrong name and said it worked (1.13.55) + + $ userdel mako + mako purged. + $ grep p_mako /etc/passwd + p_mako:x:10764:10767:p_mako:/usr/src/pkgusr/p_mako:/bin/bash + +`remove --purge mako` uninstalled 85 files, removed the home, ran `userdel +mako` -- an account that does not exist -- and reported success. The system +was left with an account owning nothing and a report saying otherwise. (The +1.13.19 fix corrected the passwd LOOKUP in the same function and missed the +delete.) + +Now: the account name is used, a failed userdel is reported, and the final +message checks what actually remains: + mako: files removed, but the account p_mako, /usr/src/pkgusr/p_mako + remain -- NOT fully purged + +For the user's host right now: p_mako is still there and can be removed with +`userdel p_mako`. After that the tree has no package users and +`config --pkgusr-prefix ""` will be accepted. + +## A flag for the prefix, and a refusal (1.13.54) + +`config` could set the collector and user prefixes but NOT the package-user +one -- the only way was editing config.json by hand, and it is the one +prefix whose change orphans every account on the system. + + packagemanager config --pkgusr-prefix "" + packagemanager config --cfguser-prefix cfg + +and, when accounts already use the current prefix, a refusal that says why: + + 2 account(s) already use the 'p_' prefix: p_gcc, p_wget + Changing it now does NOT rename them. Every tool would look for + accounts named '<none>' and find nothing: the packages would read as + not installed, their manifests would orphan, and a rebuild would make + a second set. + If you are certain (a fresh tree, or you will rename them yourself): + packagemanager config --pkgusr-prefix '' --force + +The user asked for exactly this safety before making the change, which was +the right instinct: on their HOST the accounts are p_*, so the answer is +that the change belongs on a fresh tree, not that one. + +## Say what the keys are, and mean it (1.13.53) + +The user ran `config --show` after 1.13.52 and saw no difference -- correctly. +The values were unchanged (nothing had been set), and the FOOTER still said + + set a directory: export PKGUSR_BASE / PKGUSR_SUBDIR / CFGUSR_SUBDIR + +which was true before the keys existed and stale afterwards. A report that +does not mention the thing that changed looks like nothing changed. + +It now lists the actual keys in both files. Writing that text also caught +that the claim was not yet TRUE: _early_conf read only packagemanager.conf, +so the layout could not travel with the tree. It reads the lfs config first +now, as the prefixes do -- verified from both files, with the environment +still winning over both. + +The lesson, again: when adding a setting, the place that TELLS people about +settings has to change with it -- and writing the sentence is a good way to +discover the code does not do what the sentence says. + +## The directories are configurable too (1.13.52) + +The last leak from 1.13.50/51: BASE_DIR has read `pkgusr_home` from +packagemanager.conf for a long time, but the two SUBDIRECTORIES were +environment-only. A tree with a different layout therefore depended on the +environment being right on every invocation, and could not be made +tree-specific the way the prefixes now are. + + pkgusr_subdir=packages + cfguser_subdir=config + +are read from packagemanager.conf now (environment first, then the file, then +the default -- as everywhere else), reported by `config --show` with their +source, and added to the keys that travel with the tree. + +So the whole answer to the user's question is now: the build system has its +own prefixes AND its own layout, set once, and the host cannot overwrite +either. + +## The build system's config is its own (1.13.51) + +The user asked whether the host's config gets copied into the build system. +Half of it did not (config.json has been MERGED since an earlier fix -- host +values only where the tree is silent). The other half did: + + export LFS_PKGUSR_PREFIX="{pkgusr_prefix()}" + +written into the chroot scripts, evaluated in the HOST process. So a prefix +changed on the user's own machine was baked into the build system's +environment -- and that is exactly how an empty LFS_PKGUSR_PREFIX reached a +chroot whose config says "p" (see 1.13.26, which treated the symptom). + +_tree_prefix() reads the TREE's config for those exports; the host's value is +only a fallback for a tree that has not been configured yet. Verified all +three cases. + +So the answer to the question: yes, the build system can have its own +prefixes and directories, and after this they stay its own. The one +remaining leak is the DIRECTORIES, which have no config keys at all and are +environment-only -- the open item from 1.13.50. + +## Where the prefixes and directories actually live (1.13.50) + +The user tried to check their prefixes and the package directories and could +not find them. Fairly: prefixes came from /usr/share/lfs/config.json, then +packagemanager.conf, then the environment, then a default -- and the +DIRECTORIES were environment-only (PKGUSR_BASE, PKGUSR_SUBDIR, +CFGUSR_SUBDIR), documented nowhere. `packagemanager config` existed but +PROMPTED for the main user before printing anything, and showed neither the +package prefix nor any directory. + +`packagemanager config --show` prints the lot, with the source of each: + + package-user prefix p_ the lfs config (/usr/share/lfs/config.json) + config-user prefix cfg_ the built-in default + collector prefix nimgnu_ packagemanager.conf + shared-user prefix u_ the built-in default + base directory /usr/src the built-in default + package homes /usr/src/pkgusr $PKGUSR_BASE/$PKGUSR_SUBDIR + config-user homes /usr/src/cfg $PKGUSR_BASE/$CFGUSR_SUBDIR + +...and how to change each, with the warning that a prefix is a decision taken +before the first account. + +STILL WORTH DOING: the directories have no config-file keys at all -- they +can only be set by exporting variables, which means a tree built with a +non-default layout depends on the environment being right every time. They +belong in the same config as the prefixes. + +## meson says "OSError" and names nothing (1.13.49) + + Installing .../gstnonstreamaudiodecoder.h to /usr/include/gstreamer-1.0/gst/audio + ERROR: Unhandled python OSError. This is probably not a Meson bug + +A permission failure in another package's directory -- the class the +collector-group repair exists for -- and it ran TWO rounds with nothing to +grant, because meson's error names no path. The directory is on the line +BEFORE it, which is the only place meson puts it. (The failure box already +worked it out for the human: "what those paths actually are" named +/usr/include/gstreamer-1.0/gst/audio, owned by p_gst10-plugins-base.) + +unwritable_dirs_from_log now takes the destination of the last `Installing +... to <dir>` line when an OSError / Errno 13 / Permission denied follows. + +Third extractor added for this one repair (absolute paths, a relative path +plus its `cd`, and now meson's). Each build system reports the same failure +differently, and the repair is only as good as the parsing. + +## glycin needs librsvg (1.13.48) + + ERROR: Dependency "librsvg-2.0" not found + +Third instance of one shape: the book RECOMMENDS it, the build REQUIRES it, +and both packages sit inside one strongly-connected component -- so the +cycle-breaking is free to drop exactly that edge (glycin 197, librsvg 206). + + [glycin] + needs = librsvg +librsvg 205 -> glycin 211, with libass/ffmpeg unchanged. + +The three are ffmpeg->libass, mesa->{glslang,rust-bindgen,libclc} and now +glycin->librsvg. Each was one line, and each was found by a build failing. +The real fix remains the one noted in 1.13.43: inside an SCC, drop the edge +that CLOSES the cycle rather than any recommended edge in it. Tarjan already +identifies the back-edges; using them would remove this whole class rather +than treating it one package at a time. + +## A feature missing from an installed package (1.13.47) + + Run-time dependency cairo found: YES 1.18.4 + Run-time dependency cairo-ft found: NO + ERROR: Problem encountered: No Cairo font backends found + +cairo is installed and working; it simply has no FreeType backend, because +it was built BEFORE freetype2 existed under an older ordering. The ordering +is right now (freetype2 112 -> fontconfig 116 -> cairo 117), but a package +already on disk does not rebuild itself. + +Nothing detects this class -- a package that is present and less capable +than it should be -- so the failure box now recognises the shape (a +`<pkg>-<feature> found: NO` beside a `<pkg> found: YES`) and says: + + ! 'cairo' is installed, but the feature this package needs is missing + ! from it -- it was probably built before its own optional dependency + ! existed. Rebuild it, then retry: + ! packagemanager install cairo --run --yes --reinstall + +WORTH CONSIDERING: every package built before tonight's ordering fixes may +be missing optional features the same way. A sweep -- rebuild anything +installed before a dependency it recommends -- would find them, and would be +cheap to compute from the install timestamps we already keep. + +## An ancestor's group already answered (1.13.46) + + /usr/lib/python3.14/site-packages drwxrwxr-x p_python nimgnu_python + ... + Which group should share /usr/lib/python3.14/site-packages/gi/overrides? + +The user's point: the enclosing tree is already shared through +nimgnu_python, so a nested directory inside it needs no new decision. +share_new_dir_with_tree_owner has inherited a parent's group since 1.12.44; +choose_collector_group -- the path the auto-repair uses -- never learned it. + +It walks up to the nearest ancestor carrying a COLLECTOR group now and uses +that, saying so. Only collector groups count: root or install on a parent +says nothing about sharing. With no such ancestor it still asks. + +Also visible in that output and NOT chased: "retrying at-spi2-core (phase +all), round 6" -- the auto-repair re-ran the whole phase six times, unpacking +the source afresh each round. Each round granted one directory. Granting +all the directories the log names in ONE round would turn six unpacks into +one; worth doing. + +## polkit was an orphan (1.13.45) + + ERROR: Dependency "libsystemd" not found + +polkit wanted libsystemd because this is the systemd flavour of the book and +its meson defaults to logind session tracking. But the user asked the better +question -- do we need it at all? -- and the parent check answered: polkit's +only parents are colord and systemd, both avoided. A pure orphan, like +colord and linux-pam before it. Avoided, with libgusb (colord's too). + +205 packages. bluez/pipewire/wireplumber stay: bluez has a live parent +(pipewire). + +For the record, had it been wanted: polkit on a SysV system needs + -Dsession_tracking=libelogind +which is one line in package-fixes.conf. Not added, because nothing wants +polkit. + +The parent check is now the standard move for "do I need this?" and it has +found five orphans (colord, linux-pam, xdg-utils, freeglut, polkit+libgusb). +Automating it needs every edge, which is still the open item from 1.12.83. + +## vala wanted Graphviz, not graphite2 (1.13.44) + + configure: error: Package requirements (libgvc >= 2.16) were not met + +libgvc is GRAPHVIZ (the user read it as graphite2, which is installed and +fine). Vala needs it only for valadoc, its documentation generator, and the +book says so: "--disable-valadoc: This option is required if Graphviz is not +installed." That flag was already in package-fixes.conf -- and I added a +SECOND [vala] section for it, which made configparser raise and the WHOLE +FILE be ignored, silently disabling every fix in it. + +Duplicate removed, and the suite now parses package-fixes.conf on every run: +one bad section would otherwise turn off ffmpeg's libass edge, mesa's three +`needs`, cairo's ctime_r and the rest at once, with no message beyond a +single warning line. + +(Check the file before adding to it -- the sections have grown past the +point where they fit on one screen.) + +## Break inside the cycle, release only the knot (1.13.43) + +libass really was not installed, and the recursive plan put ffmpeg FIRST +(0/32) -- so the sort had dropped ffmpeg's edge to libass. Two faults, both +in the cycle handling: + + * "fewest blocking dependencies" is not "in the cycle". It picked ffmpeg + and dropped its recommended libass edge, while the actual cycle was + harfbuzz <-> freetype2. Cycle membership is computed properly now + (Tarjan's SCC), and only edges with BOTH ends in one component are + dropped. + * the required-cycle fallback emitted EVERY remaining node in its original + order -- one unbreakable knot threw away the ordering of everything + after it. It releases just the smallest component now and keeps sorting. + +The isolated case sorts correctly (harfbuzz, freetype2, libass, ffmpeg). In +the FULL book graph ffmpeg and libass still land in one huge component, so +the recommended edge between them remains fair game -- and ffmpeg's configure +hard-requires libass. So: + [ffmpeg] + needs = libass +Order: libass 204 -> ffmpeg 209, with every earlier case intact. + +NOTE for next session: a big SCC means the "drop recommended edges inside +it" rule can still drop something a build needs. The mechanism to declare +the edge required is the safety valve, but a better rule (drop the edge that +closes the cycle, not any edge inside it) would need the cycle's back-edges, +which Tarjan gives -- worth doing properly. + +## A one-package plan with missing dependencies (1.13.42) + + packagemanager install ffmpeg --run --regenerate + ### Install plan (1 to build) + ... + ERROR: libass >= 0.11.0 not found using pkg-config + +libass really was not installed (search, pkg-config and the missing .pc all +agreed). Without --recursive the plan is exactly one package -- correct +behaviour, and the command came from me without the flag. + +But the plan KNOWS its dependencies are missing and said nothing. It now +does: + + 2 dependencies of FFmpeg-8.0.1 are NOT installed: + libass-0.17.4, ... + this plan builds ONLY FFmpeg-8.0.1 and will fail on them. + to build them too: packagemanager install ffmpeg --run --recursive + +Worth remembering when suggesting commands: `install <pkg> --run` is only +right for a package whose dependencies are already in place. For anything +mid-build, the stack run or --recursive is the honest suggestion. + +## Half a command is worse than none (1.13.41) + + install_ffmpeg: line 242: rsync: command not found + +`make fate-rsync` was commented (1.13.39) and the standalone fetch beside it +was not -- and worse, the fetch is split over two lines: + + rsync -vrltLW --delete ... \ + rsync://fate-suite.ffmpeg.org/fate-suite/ fate-suite/ + +Only the SECOND line matched the pattern, so the first stayed active and ran +on its own. A match on a continuation line now walks BACK and comments the +command it belongs to, and rsync:// fetches count as test-suite work. + +STILL OPEN, and the actual build failure: `libass >= 0.11.0 not found using +pkg-config`, with libass recorded as installed. That is the xorg7-lib shape +again -- a record without the files -- and the user is checking. + +## A stale script refreshes itself (1.13.40) + +ffmpeg failed on the identical error one release after it was fixed, and the +box said why: "generated by blfs 1.13.38; the generator is now 1.13.39". + +That warning was added in 1.13.4 and has now cost several rounds EACH on +brotli, rust, xcb-utilities and ffmpeg. The fix existed, the message +printed, and the build ran the old script anyway -- because noticing one +warning in a 200-package plan and acting on it per package is not a +reasonable thing to ask of anyone. + +A script older than the generator is now REGENERATED, unless the user has +claimed it: a `# EDITED` line in the file, or an existing .edited sibling, +means hands off (and then the old message appears, with the command). + +The 1.13.4 reasoning -- "not automatic, because regenerating discards edits" +-- was right about the risk and wrong about the remedy. The answer was a way +to say "this one is mine", not making every user do it for every package. + +## A typesetting loop, and ffmpeg's fate (1.13.39) + + You don't have a working TeX binary (tex) installed anywhere in your PATH + make: *** No rule to make target 'fate-rsync'. Stop. + +Two more in one page: + + * ffmpeg wraps texi2pdf/texi2dvi/dvips in `pushd doc && for ... done &&`. + The 1.12.87 filter only knew `make ... pdf|ps|dvi`, and commenting just + the texi2* lines would have left an EMPTY LOOP BODY -- so the block is + commented as a unit, from the pushd/for that opens it to the done/popd + that closes it. + * `make fate` is ffmpeg's test suite, and `make fate-rsync` fetches its + samples -- tens of gigabytes over rsync -- during an ordinary install. + Added to the test drivers. + +ffmpeg's build is now: configure, make, install. A loop that does NOT +typeset is untouched. + +## Rearranging a directory nothing filled (1.13.38) + + mv: cannot stat '/usr/share/doc/git-2.53.0/git*.adoc' -- with 346 files installed + +FIFTH in the same family, and the last shape of it in git: after the +optional "untar the html docs" block is commented, the page still +reorganises what it would have extracted, with mkdir/mv chains. + +A `mv` (or find/sed) whose directory appears ONLY in commented commands is +moving something nobody produced -- so it is commented, with its chain and +the `mkdir` that led into it. Two things had to be right for this to work: +a trailing slash must not defeat the match, and mv/find/sed must NOT count as +"filling" a directory (they rearrange it; only tar/cp/install/make fill). + +Git's install block is now: `make ... install` and nothing else. Everything +optional is commented, and 346 files were already installing correctly +before this. + +THE FAMILY, for whoever adds the next filter: + libuv's man page, the orphan log reader, brotli's wheel, + install-man/install-html, and this. +Ask, every time: WHAT CONSUMES WHAT THIS BLOCK MADE? + +## Installing docs nothing built (1.13.37) + + /bin/sh: line 1: asciidoc: command not found + make: *** [Makefile:3725: install-man] Error 2 + +Git's page generates its man pages and html docs under "If you have installed +asciidoc and xmlto you can create ..." -- correctly skipped -- and then +installs them unconditionally. `make install-man` therefore tried to BUILD +them, after everything else had installed. + +An `install-<x>` whose `<x>` nothing active builds is now commented, along +with the rest of its && chain (git follows install-html with mkdir/mv of +files only install-html would have created). A package that really does +build its docs keeps both lines. + +Fourth in this family: libuv's man page, the orphan log reader, brotli's +wheel, this. The rule to apply when adding any filter: WHAT CONSUMES WHAT +THIS BLOCK MADE? + +Also note: `make htmldir=... install-html` needed the pattern to allow +variable assignments between make and the target. And I broke the edit +twice by patching in place before rewriting the function whole -- the +1.13.31 note about that stands. + +## A relative complaint needs the cd above it (1.13.36) + +The log settled it in two lines: + + (cd '/usr/lib/perl5/5.42/site_perl' && umask 022 && tar xof -) + /usr/bin/tar: ./Git: Cannot mkdir: Permission denied + +A permission problem in another package's directory -- the class the +collector-group repair exists for -- and it found NOTHING to grant, because +tar reports "./Git" and every extractor looks for absolute paths. The +directory that needs granting is named one line earlier, in the subshell's +own cd. + +unwritable_dirs_from_log now scans the tail in order, remembers the last +absolute `cd`, and emits it whenever a relative complaint follows. Verified +on the real two lines: it yields /usr/lib/perl5/5.42/site_perl. + +(The 1.13.35 tar wrapper was right and not enough: it correctly tolerated the +utime/mode noise, which then let the REAL error -- three lines later and +about a different directory -- become visible.) + +## tar is a wrapped tool too (1.13.35) + + tar: ./zh_TW: Cannot change mode to rwxr-xr-t: Operation not permitted + tar: Exiting with failure status due to previous errors + make: *** [Makefile:3636: install] Error 2 -- with 104 files installed + +Git installs its translations by piping through tar, which then stamps the +destination DIRECTORIES: they belong to other packages and are sticky +because we sealed them. The FILES extracted correctly. + +cp has had this exact wrapper for a long time ("preserving times for +/usr/bin: Operation not permitted"); tar was never wrapped. It is now, with +the same rule: if every complaint is about utime/mode/ownership on something +we do not own, the extraction succeeded; anything else still fails. Verified +both ways. + +Wrapped tools are now: install, chown, chgrp, chmod, mkdir, cp, ldconfig, +tar. Worth asking what else writes metadata -- rsync, cpio, and `install -p` +are the obvious candidates, and none has appeared yet. + +## vulkan-loader needs git (1.13.34) + + FileNotFoundError: [Errno 2] No such file or directory: 'git' + +Its CMake scripts shell out to git to stamp a version. git IS a stack entry +but sits at the end, and vulkan-loader is built inside another entry's plan, +so it was not there yet. One line in package-fixes.conf: + + [vulkan-loader] + needs = git + +Order: git 111 -> vulkan-loader 159. + +Checked the other fifteen packages in the plan for the same shape -- none of +their SCRIPTS call git; vulkan-loader's call is inside its CMake, which no +scan of the book would ever find. That is the limit of this class: a +dependency invisible in both the book AND the generated script, findable only +by building. + +## --yes answers a question; it does not name things (1.13.33) + + # granting access via collector groups and retrying: + # /usr/include/X11 -> group nimgnu_xorgproto + +The user asked why nothing was asked any more. The prompt was there; --yes +skipped it -- and --yes was in the command this session kept recommending. + +The distinction that was missing: reusing an EXISTING group is a yes/no, and +--yes may answer it. Creating a NEW one is a choice of NAME that stays on +the system, and the user wants to keep making it. So a run with --yes now +still stops for a group that does not exist yet, whenever a terminal is +available. With no terminal, the owner-named choice stands and says so -- +including "(this CREATES the group X)". + +Worth applying the same test elsewhere: --yes should carry decisions whose +answer is obvious afterwards, not decisions that leave a name behind. + +## ...and the build must resume where they left off (1.13.32) + +1.13.31 got the downloads running -- 32 tarballs, all md5-verified -- and the +build loop still said + + grep: ../lib-7.md5: No such file or directory + +The page does `mkdir lib && cd lib`; the build phase is a separate function +that starts at the build root, so `../lib-7.md5` pointed nowhere. unpack now +records that directory in .pm_build_cwd (the mechanism already existed for +resuming a build) and _enter_build returns there for a meta page. + +AND THE SECOND HALF, which is the one that mattered: a loop over a missing +list runs zero times and exits 0, so the package was RECORDED as installed +having built nothing. Every `for x in $(... ../file ...)` now gets + + [ -s "../lib-7.md5" ] || { echo "the list this build loops over is + missing or empty: ../lib-7.md5" >&2; exit 1; } + +That was the "loop that runs zero times is not a success" item from 1.13.31's +notes; it took one release rather than a session because the shape was +already written down. + +## A meta page's own commands must run (1.13.31) + + grep: ../lib-7.md5: No such file or directory + # xorg7-lib installed no NEW files ... + # xorg7-lib: done <- RECORDED AS INSTALLED + +The meta-page early return (1.12.61) sat ABOVE the page's own Downloading +section, so Xorg Libraries never wrote lib-7.md5 nor fetched its list. The +build loop read a file nobody had created, iterated zero times, and exited 0 +-- which is how a package that installed nothing got recorded, twice. + +The page's own commands are emitted BEFORE the early return now. A normal +package is untouched (one tar -xf, as before), and nine pages including the +three meta ones generate valid bash. + +TWO THINGS FOR NEXT SESSION, both exposed here: + +1. A LOOP THAT RUNS ZERO TIMES IS NOT A SUCCESS. `for x in $(grep ... file)` + exits 0 when the file is missing. The recorder trusts rc==0 (1.12.7), so + an empty meta build looks identical to a complete one. A meta page should + assert it built something. + +2. I broke the generator three times editing it in place during this fix and + restored it from the shipped zip. When an edit needs more than one + substitution, write the new function whole rather than patching around it. + +## verify checks the package you name (1.13.30) + + lfs-helper verify xorg7-lib --fix + ... + [1/533] binutils-pass1 ^C + +It read only $1 for the flag and ignored the name, so a question about ONE +package started a walk of all 533. (The command came from me, which is how +it surfaced.) + +Arguments are parsed properly now: --fix/--run anywhere, the first non-flag +is the package, and the ownership pass is narrowed to its manifest. An +unknown name fails immediately rather than scanning everything. + +## A function definition is not a command to skip (1.13.29) + + packagemanager search xorg7-lib -> installed + pkg-config --exists x11 -> 1 + +xorg7-lib recorded as installed with no libX11 on disk. The cause was mine: +the Xorg loop defines + + do_build() { make; } + do_test() { make check; } + do_install() { as_root make install; } + +and calls them per package. The 1.13.2 test filter commented the DEFINITION +of do_test, so the loop called a function that no longer existed. + +_is_definition() now guards every filter -- the doc and pip filters had the +same flaw waiting for a page that wraps those commands in a function. A real +`make check` is still skipped. + +The record is a separate problem the user must clear: + lfs-helper verify xorg7-lib --fix (purges the lying record) + packagemanager install xorg7-lib --run --yes --regenerate + +And the lesson: a filter that edits the book's commands can break a script +in ways `bash -n` cannot see -- this one parsed perfectly and did the wrong +thing. Every filter added since 1.13.2 should be re-read with "what if this +line is inside a function definition, or defines one?" in mind. + +## The book writes -D key=value with a space (1.13.28) + + ERROR: Dependency "libdrm_intel" not found + +An INTEL dependency, on a machine.conf that says radeonsi. The merged line +explained it: + + meson setup -Dgallium-drivers=radeonsi ... \ + -D gallium-drivers=auto \ <- the book's own, still there + -D vulkan-drivers=auto \ + +apply_machine_opts matched only `-Dkey=`, and the book writes `-D key=value` +with a space -- so ours was PREPENDED and meson took the last occurrence. +mesa was building every driver, and the three "undeclared dependencies" +found in 1.13.23-27 (glslang, rust-bindgen, libclc) were at least partly +this: an auto build wants far more than a radeonsi one. + +The pattern accepts the space now, so the book's own option carries the +machine's value. Worth re-testing whether mesa still needs all three of +those `needs =` entries once it builds for radeonsi alone -- if not, they are +harmless but no longer necessary. + +## mesa's third undeclared dependency (1.13.27) + + ERROR: Dependency "libclc" not found, tried pkgconfig and cmake + +libclc IS in the book; mesa's page does not list it. radeonsi (from the +user's machine.conf) builds the OpenCL bits and needs it. One line: + + [mesa] + needs = glslang rust-bindgen libclc + +Order: llvm 120 -> glslang 139 -> rust-bindgen 140 -> libclc 143 -> mesa 159. + +Three undeclared dependencies for ONE package, each found by a build +failing. The pattern is now cheap to fix but still expensive to find -- +worth considering whether a failed `Dependency "X" not found` could search +the book for X and suggest the `needs =` line itself. The failure box +already does something similar for a missing PROGRAM (1.12.x); this is the +same idea for a missing pkg-config module. + +## An empty export is not an answer (1.13.26) + +The user checked, and both config files said "p". The note added in 1.13.25 +was therefore a FALSE POSITIVE, and a useful one: lfs-helper really did read +an empty prefix. + + PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-...}" + +`-` means "set but empty wins". A chroot script exports +LFS_PKGUSR_PREFIX="" when the value was unknown when it was written, and +that beat a config file that says "p". Now `:-` for the ENVIRONMENT, while +the FILE keeps its present-and-empty meaning (1.12.95) -- the two are not the +same statement, and only one of them is deliberate. + +Also: mesa needs `bindgen` as soon as it FINDS a rust compiler, and rust is +built here for librsvg. The book's mesa page lists neither it nor glslang, +so both are declared in package-fixes.conf now: + [mesa] + needs = glslang rust-bindgen +Order: rust 128 -> glslang 139 -> rust-bindgen 140 -> mesa 157. Second use +of the 1.13.23 mechanism, and this time it took one line rather than a +release. + +## Strip what an older version wrote (1.13.25) + +mesa failed on -Dneeds= AGAIN after 1.13.24 stopped adding it: the flag was +already in the script on disk, and the merge is idempotent, so nothing took +it out. Stopping a bad write does not undo the writes already made. +apply_machine_opts removes a stale -Dneeds= when it sees one, and says so. + +Also, the stale-script box suggested `packagemanager install p_mesa` -- the +ACCOUNT, where the command wants the ANCHOR. The user pointed out that +p_mesa is correct for their tree (prefix "p"), which is the useful part: +unprefix_pkg_user should therefore have produced "mesa" and did not, meaning +lfs-helper read an EMPTY prefix while packagemanager had "p". + +That is a config disagreement (config.json vs packagemanager.conf inside the +chroot), so the box now prints the anchor AND names the disagreement rather +than hiding it behind a literal strip. Asked the user to check both files. + +## One file, two readers (1.13.24) + + mesa: ../meson.build:4:0: ERROR: Unknown option: "needs". + +1.13.23 put `needs = glslang` in package-fixes.conf for blfs to read as a +dependency edge -- and apply_machine_opts turns EVERY key in a section into +a meson -Dkey=value, so mesa was configured with -Dneeds=glslang. + +The file has two readers and each must ignore the other's keys. +apply_machine_opts drops `needs`; blfs already looks only at `needs`. + +Worth watching if the file grows: a third key type would need the same +treatment, and the failure appears in a package's configure output rather +than anywhere near the config file. + +## A dependency the book does not record (1.13.23) + + libass: configure: error: Package requirements (harfbuzz >= 1.2.3) were not met + +libass's page lists FreeType, FriBidi, Fontconfig and NASM. Its configure +hard-requires harfbuzz. So no ordering could place it -- libass 143, +harfbuzz 199 -- and moving stack entries could not either, because both sit +inside ONE plan. + +package-fixes.conf can now add the edge, and blfs treats it as required: + + [libass] + needs = harfbuzz + [mesa] + needs = glslang # the same class, patched by hand in 1.12.96 + +Result: harfbuzz 198 -> libass 199. + +This is the general answer to a class that has cost several sessions +(glslang/mesa, and every "X not found" where the book never said X). When +the next one appears, it is two lines in package-fixes.conf, which the sync +refreshes -- no code change. + +## Commenting a command stranded the one before it (1.13.22) + + install_p_libpwquality: line 190: syntax error near unexpected token `}' + +The generated script ended: + make install && + # pip3 install ... --find-links dist pwquality +The pip filter (1.13.3) commented the last command of an && chain and left +`make install &&` dangling, so bash refused the entire file. + +Every filter that comments a line can do this -- the test-run filter already +had a local version of the fix (1.13.6, for pipelines). The repair belongs +at the END, once, over the whole block: an ACTIVE line ending in && || or | +with no active line after it loses the operator. Real chains are untouched. + +Swept all 30 packages in the current plan afterwards: no syntax errors. +Worth keeping that sweep in mind whenever a filter is added -- the generator +runs `bash -n` and PRINTS the error, but still writes the file, which is how +this reached a build. + +## Say "regenerate" where the failure is (1.13.21) + +XCB-Utilities failed again with the message 1.13.20 had just fixed, because +the script it ran came from the package's home and was written by an older +generator: + install file: /usr/src/pkgusr/p_xcb-utilities/install_XCB-Utilities + +packagemanager warns about that when it RESOLVES a script -- at the top of a +59-package plan, hundreds of lines before the failure. The failure box now +carries the same news, with the command: + + ! This script was generated by blfs 1.13.6; the generator is now 1.13.21. + ! Fixes made since then are NOT in it -- including any for this very + ! failure. Rebuild it from the book first: + ! packagemanager install xcb-utilities --run --yes --regenerate + +Third time this has bitten in the same session (brotli, rust, this). The +information existed each time; it was just nowhere near the failure. + +## A meta page has nothing to enter either (1.13.20) + + no single tarball -- this page builds a list of packages + no unpacked source in .../src -- run: ... unpack + +Both lines in the same run: unpack_pkg correctly did nothing (1.12.61) and +_enter_build then demanded the directory unpack had not created, so +XCB-Utilities failed in the BUILD phase. Half a fix -- the same shape as +1.12.20/21 (the shared index) and 1.13.11/13 (the profile helpers): the first +half was right and the second path had not been told. + +_enter_build returns at the build root when the page has no tarball of its +own; the page's commands download and cd where they need. A normal package +still fails loudly if its source is missing. + +## A fixed PATH loses everything in /opt (1.13.19) + +The user's own test settled it: + + su - p_cargo-c -c 'command -v cargo' -> /opt/rustc/bin/cargo + (the build) -> cargo: command not found + +The login shell had it; the BUILD did not, because cmd_build replaced PATH +with a hard-coded list that has no /opt in it. So the profile work of +1.13.11-18 was correct and still could not reach a build. + +cmd_build now appends the /opt directories that the packages' own profile.d +snippets name (grep for pathprepend/pathappend, keep the ones under /opt +that exist) after the fixed base -- wrappers and system tools still first. +Read from the files rather than by starting another login shell: cheaper, and +it does not inherit whatever else a profile does. + +Also fixed: `packagemanager remove cargo-c --purge` crashed with +KeyError: getpwnam(): name not found: 'cargo-c' -- it looked the ANCHOR up in +passwd instead of the account, and a missing account is a normal answer here. + +Both suite failures on the way were real: root must not get the wrappers, and +my first version shelled out to `su` to compute the PATH. + +## The profile loop is not cosmetic (1.13.18) + + install_p_cargo-c: line 161: cargo: command not found + +cargo lives in /opt/rustc-1.93.1/bin and gets onto PATH through +/etc/profile.d/rustc.sh -- which never runs, because this system's +/etc/profile does not source /etc/profile.d. 1.13.13 offers to add the +loop, and the offer could only be accepted by exporting LFS_ASSUME_YES, +so it was (reasonably) skipped. + +`lfs-helper install-profile --yes` now accepts the answer on the command +line. The build runs through `su -`, a login shell, so once /etc/profile +reads profile.d every package that installs under /opt is on PATH for every +later build. + +Worth seeing the chain: rust installs to /opt -> writes a profile.d snippet +-> the snippet needs pathprepend (1.13.11) -> and the snippet needs to be +READ (1.13.13/18). Three separate fixes for one package to be usable, and +the last of them was a prompt nobody could answer. + +## An install nobody recorded did not happen (1.13.17) + +Rust installed cleanly (the ~/src fix worked -- x.py ran, every component +installed). The very next stack run then planned it as + + Rustc-1.93.1 new required by cargo-c + +and started compiling it again. `packagemanager script` runs the phase +directly and only pm-install writes the manifest, so nothing recorded the +install and every other command still read rust as absent. + +cmd_script now calls `lfs-helper record-install` after a phase that installs +(install / all / update), and says so. + +This is the fourth bug in cmd_script. The handoff item stands and is getting +more expensive to ignore: that function reimplements what _install_via_self +does properly, and should call it instead. + +## The plan's "recommended by" names one parent, not the reason (1.13.16) + +The user asked why recommendations are still being pulled in, pointing at +lines like "GTK-3.24.51 new recommended by libreoffice" -- with libreoffice +avoided. + +Two different things, and the display hid the difference. `blfs order` +reports the FIRST parent that pulled a package in, so: + + gtk3 labelled libreoffice, really wanted by adwaita-icon-theme + ffmpeg labelled qt6, really wanted by mpv + polkit labelled systemd, really wanted by colord + pulseaudio labelled sdl3, wanted by pipewire and gst too + +Those belong in the plan. But checking every parent (not the printed one) +found four with NO live parent at all -- colord, linux-pam, xdg-utils, +freeglut -- pure tails of cups/systemd/mupdf. Avoided. 204 packages now. + +And to answer the question itself: recommendations stay because required-only +was tried twice (1.12.47, 1.12.70) and broke the build both times -- BLFS +"recommended" often means the build command uses it (cmake has NO required +deps and hard-fails without curl; libxml2 without icu). The rule is: build +the book's graph, name what you do not want. + +WORTH FIXING: the plan should print a parent that is actually IN the plan. +Doing that by hand is what this note is; the tool has the data. + +## Where the sources are, not where the shell started (1.13.15) + + $ su - p_rust -c 'bash ~/install_p_rust install' + install_p_rust: line 237: ./x.py: No such file or directory + +BUILD_ROOT fell back to $PWD, and a login shell starts in the HOME -- so +_enter_build hunted for the unpacked tree beside the script, took the first +directory it found (log/, probably) and rust's x.py was not there. It only +worked during a full `all` run because the build phase had already left the +shell inside the source directory. + +Sources are unpacked into ~/src, so that is the fallback now: + LFS_BUILD_ROOT (the runner) -> BUILD_ROOT (explicit) -> ~/src -> $PWD + +This is the other half of 1.12.90: that fix stopped exporting BUILD_ROOT into +package builds, and this one gives the script a sensible answer when nobody +exports it at all. + +## The whole startup-files chapter (1.13.14) + +The user asked why only the path helpers. Reading the chapter answered it: +"The Bash Shell Startup Files" creates NINE files, and we installed one. +Without the rest a package user's shell has no umask policy, no INPUTRC, no +locale setting and no ~/.bashrc. + +install_shell_startup_files now writes, when absent: + /etc/profile.d/umask.sh readline.sh i18n.sh extrapaths.sh + /etc/bashrc + /etc/skel/.bash_profile .bashrc .bash_logout .profile +plus /etc/profile and the path helpers from 1.13.11. + +They are OUR versions -- shorter, commented for this system -- not copies of +the book's text. Nothing that already exists is touched: each file reports +"kept existing" instead, verified by putting a file in place and re-running. + +## Report what you did, not only what you refused (1.13.13) + + $ lfs-helper install-profile + /etc/profile exists but does not source /etc/profile.d -- left alone. + ... + "it did not installed the files" + +It HAD written /etc/profile.d/00-path-functions.sh -- that happens before the +profile check -- and said nothing about it. A command whose only output is +what it declined to do reads as a command that did nothing. + + * it now reports the helpers it wrote, in every branch; + * an /etc/profile that lacks the profile.d loop is OFFERED the line through + confirm_change (--yes accepts), with a backup at + /etc/profile.before-lfs-helper, instead of a paste-this message. + +Both paths verified: with consent the loop is appended and the backup taken; +without it the profile is untouched and the helpers are still installed. + +## Wired into the command nobody runs (1.13.12) + +1.13.11 put install_profile_helpers in cmd_init_pkgusr -- and the command +people run on a fresh system is `packagemanager setup`, the BOOTSTRAP, which +is a different entry point. The user ran setup, saw the five stages pass, +and got no helpers. + +Two ways in now: `lfs-helper install-profile` on its own, and the bootstrap +calls it and prints what it did. + +The pattern is familiar enough to name: a fix wired into a code path the user +does not travel is not delivered (1.12.85 the file that was never +overwritten, 1.12.86 the script that was never re-read, this). Ask which +command the user will actually type. + +## The startup files BLFS assumes (1.13.11) + +The user's diagnosis, and it is right: BLFS pages drop snippets into +/etc/profile.d that call pathprepend/pathappend, the book defines those in +postlfs/profile ("The Bash Shell Startup Files"), and LFS does not install +them. So rust's rustc.sh said + + pathprepend: command not found + +on every package-user login -- noise at best, a failed configure at worst, +and it will happen for every package that ships a profile snippet. + +install_profile_helpers, run by `packagemanager setup` (cmd_init_pkgusr), +writes /etc/profile.d/00-path-functions.sh with pathremove, pathprepend and +pathappend -- our own implementation, not a copy of the book's files. + +/etc/profile is the administrator's: + * already sources profile.d -> nothing to do + * exists but does not -> LEFT ALONE, with the line to add printed + * does not exist -> a small one is written that sources profile.d +Verified: the helpers actually prepend and append, and an existing profile +survives untouched. + +## A local import that shadowed the module (1.13.10) + + UnboundLocalError: cannot access local variable 'pwd' + +cmd_script imports pwd inside a try in one branch, which makes `pwd` LOCAL +to the whole function -- so the module-level `pwd` (imported at the top, with +a `pwd = grp = None` fallback for non-unix) was invisible, and the later +`if pwd is not None` crashed. The import now lives where it is used, and a +missing account leaves the file owned by root rather than raising. + +The suite gained an AST check for the pattern: any function that imports pwd +locally must not also use a bare `pwd`. + +This is the third bug in cmd_script in three releases (anchor-vs-account +twice, now this). That function is the one place doing by hand what +_install_via_self does properly, and it should probably just call it. + +## Two names, and they are not interchangeable (1.13.9) + +1.13.8 fixed `packagemanager script` by replacing the anchor with the +account everywhere in that function -- and broke the other half: + + Generating install script for p_rust... + blfs script failed for p_rust + +cmd_script needs BOTH: + anchor -- what the book calls it (rust); blfs generates from this + name -- the account (p_rust); su and the script path use this +It used the anchor for both, then the account for both. They are separate +variables now. + +STILL OPEN on the user's chroot: `lfs-helper build rust --phase configure` +did not recognise rust as a package and fell through to the LFS build-step +path ("no script for 'rust' at .../scripts/rust.sh"). That is the account +prefix again -- if the chroot's config carries pkgusr_prefix "" while its +accounts are p_*, lfs-helper (which honours empty since 1.12.95) looks for an +account called "rust". Asked the user for: + grep pkgusr_prefix /usr/share/lfs/config.json /etc/pkgusr/packagemanager.conf +If that is it, the answer is to put "p" back in the CHROOT's config -- and +the deeper question is whether an empty prefix should be allowed at all once +a tree exists with prefixed accounts. + +## script ran as the anchor, not the account (1.13.8) + + packagemanager script install rust + $ su - rust -c 'bash ~/install_rust install' + bash: install_rust: No such file or directory + +cmd_script took args.package verbatim -- so it used the ANCHOR as both the +account name and the script name, neither of which exists. Everything else +in the tool goes through pkgusr_name(); this one path did not. (Found +because a command I recommended to the user was itself wrong.) + +## Writing a login profile is not running it (1.13.7) + +Rust built, installed 2390 files, and failed on the last line: + + /etc/profile.d/rustc.sh: line 3: pathprepend: command not found + +The page writes that snippet and then sources it. The snippet calls +pathprepend/pathappend -- helpers from the BLFS bash startup files, absent +from a build shell -- so sourcing it tests the login shell, not the package. +The file is written and applies at the next login, which is its purpose. + +`source /etc/profile[.d/x]` and a bare `/etc/profile.d/x.sh` are now +commented, in the configuration section as well as build and install. + +Note this was the CONFIG section again (1.12.89 was the first): a part of the +book that runs after a successful install and can fail a package that is +already on disk. + +## x.py test, and a guard that ate the install (1.13.6) + +Rust ran 41425 tests during an ordinary build: `./x.py test` is its test +driver and the 1.13.2 list did not know it (nor `cargo test`, `go test`). +Added -- with `./x.py build` and `./x.py install` carefully NOT matching. + +Two more, found by reading the regenerated script instead of shipping: + + * the test run pipes into rustc-testlog and the next lines grep and awk it. + Commenting only the run left `awk` reading a file nobody wrote, so a + commented line now takes its continuation lines (trailing |, && or \) + with it. + * the book prefaces `./x.py install` with "If sudo or su is invoked for + switching to the root user, ensure ..." -- and the conditional guard + commented THE INSTALL. A sentence about privilege mechanics guards + nothing; sentences starting "if sudo" or "if su" are no longer treated + as conditions. + +That second one is the guard's first FALSE POSITIVE on something essential, +and it was one regenerate away from silently installing nothing. Worth +remembering that the guard's failure mode is not only "runs too much". + +## A question with one answer is not a question (1.13.5) + + 'p_rust' created /opt/rustc-1.93.1/share inside 'p_rust''s tree. + Which collector group should own it? + 1) nimgnu_p_rust (named after the tree ...) + 2) nimgnu_p_rust (named after the package that made it) + +Two bugs in four lines: + + * p_rust created a directory in ITS OWN tree, so there is nothing to share + and both options were the same name. When the tree owner IS the + installing package, no group is created and no question is asked. + * the group kept the ACCOUNT prefix. collector_group_for unprefixes with + PKGUSR_PREFIX, which is empty on that system, so "p_rust" survived whole. + The tools' own default prefix is stripped as well now: nimgnu_rust. + +And, at the user's request, the three things the answer depends on are +highlighted -- the package doing the installing, the directory, and the +recommended name: + + package : p_cmake (it is doing the installing) + directory: /usr/share/vim/vimfiles + in the tree of: p_vim + 1) nimgnu_vim <- recommended: named after the tree ... + +## A fixed generator does not fix the scripts on disk (1.13.4) + +brotli failed on the identical pip line after 1.13.3 fixed it: the script in +the package home was written by the PREVIOUS blfs, resolve_script prefers a +local script (correctly -- it may hold the user's edits), and the only notice +was a one-line "pass --regenerate" hint that vanishes in a 200-package run. + +The mismatch is now stated plainly: + ^ this script was generated by blfs 1.13.2; the current generator is 1.13.4 + fixes made since then are NOT in it. To take them (your edits, if any, + are saved as install_<pkg>.edited): + packagemanager install brotli --run --regenerate + +Deliberately NOT automatic: regenerating would silently discard edits, and +1.12.92 showed how long a silently-ignored edit can go unnoticed. The user +decides; the tool makes sure they know there is a decision. + +This is the same shape as the /tmp cached-script check (1.12.57) and the +stale env questions -- a version stamp exists on these files, and every +consumer of them has had to learn to read it. + +## A pip install whose wheel nothing built (1.13.3) + +brotli installed its C library perfectly and then failed on + + WARNING: Location 'dist' is ignored: ... non-existing path + ERROR: No matching distribution found for Brotli + +Its optional Python bindings are two steps: `pip3 wheel -w dist ...` (which +the conditional guard correctly commented) and `pip3 install --find-links +dist Brotli` (which it did not). Producer skipped, consumer live -- the +same shape as the orphan log reader (1.12.62) and libuv's man page. + +A pip install reading from a --find-links directory that no ACTIVE command +fills is now commented with its producer. A package that really does build +the wheel keeps both lines. + +Third instance of this family, so it is worth naming: whenever the guard +comments a block, ask what CONSUMES what that block made. The generator has +no dataflow model -- each case has been found by a build failing on the +consumer. + +## Running the tests is not building the package (1.13.2) + + Tests summary: Passed: 77549 Failed: 12 + ! p_nss FAILED (phase 'all') + +An ordinary install ran NSS's entire test suite -- hours of it -- and then +failed the package on a leftover test artifact. The book puts the run in +the BUILD block: + cd tests && + HOST=localhost DOMSUF=localdomain ./all.sh + cd ../ +and the generated script already HAS a test phase, gated on PM_TEST=1. + +Test drivers (make check/test, ctest, meson/ninja test, pytest, ./all.sh, +./runtests, make -C tests) are commented out of build and install now, along +with the `cd tests &&` that leads in and the `cd ..` that leads back out -- +or the && chain would break. + +A bug worth remembering from writing it: the regex began with `\b(`, and +there is no word boundary before "./all.sh", so that alternative could never +match. It compiled, it caught the make-based drivers, and it let the one +that mattered through -- a filter that half works looks like a filter that +works. + +## Tails of avoided packages, named by hand (1.13.1) + +The plan still held 255 packages after the named exclusions, and the parent +column said why: most of the remainder existed ONLY to serve something +already avoided. + + poppler, mupdf, gs, qpdf, pdfio <- libcupsfilters <- cups (avoided) + clucene, redland, rasqal, raptor <- libreoffice (avoided) + jasper, libmng, double-conversion <- qt6 (avoided) + exiv2 <- libkexiv2 <- okular (avoided) + the whole xfce/lxqt/gnome stack <- xdg-desktop-portal's four backends + +Each was checked against its parents before being added -- rust and llvm +STAYED, because glycin genuinely requires rust and llvm is a stack entry for +mesa. 273 -> 208, with webkitgtk, gtk4, mpv, pipewire, mesa, cairo, +harfbuzz, graphite2, librsvg and gdk-pixbuf all still in. + +This is the manual version of the prune reverted in 1.12.83. The real fix +remains: `blfs order --anchors` reports ONE parent per package, so "reachable +only through an avoided package" cannot be computed from it. Emitting every +edge would make this list unnecessary -- and is the next session's first +task, tested against the five ordering cases already in the suite. + +## The status counted what the run skips (1.13.0) + + packages: 108 of 273 installed + ... qt6, okular, libreoffice, cups, openjdk, postgresql, xfce4-panel ... + +all of which the stack AVOIDS. The avoid list reaches the installer +(1.12.78) and never reached the status display, so the user was shown the +unfiltered graph and reasonably asked whether the exclusions had come undone. +They had not; the report was wrong. + +--status now drops avoided packages from the count and says how many: + packages: 108 of 214 installed + avoided : 24 named in the stack (apache-ant, cups, gegl, ...) + +Third time a status has reported something other than what the run does +(1.12.71 the graph, 1.12.72 the stale file, this). A display that disagrees +with the action is worse than no display -- worth a look at whether any other +report is computed differently from the thing it describes. + +## Staging a file onto itself (1.12.99) + + install: '/usr/src/pkgusr/p_nss/install_p_nss' and + '/usr/src/pkgusr/p_nss/install_p_nss' are the same file + !! could not stage the install script + +The script is staged TWICE -- pm-install copies it into the account's home, +and cmd_build stages it again to the same name. That was invisible while the +account's home was MISPLACED (/usr/src/p_nss vs /usr/src/pkgusr/p_nss, two +different paths); the moment fix-home repointed the home mid-run they became +one file and install(1) refused. + +Both staging sites now skip the copy when source and destination are the same +file (and the package-user branch still confirms ownership). A latent bug +that only surfaced because another fix worked. + +STILL WORTH DOING: the double staging itself. One copy, in one place, would +be better than two that happen to agree -- see the note under 1.12.75 about +account creation ignoring PKGUSR_SUBDIR, which is the same story one layer +down. + +## Break the least (1.12.98) + +ghostscript failed on a missing FreeType, and the order explained it: +gs 112, freetype2 130 -- with gs RECOMMENDING FreeType. + +The topological sort (1.12.69) breaks cycles by dropping recommended edges, +and it dropped ALL of them among the remaining nodes at once. gs sits in a +genuine cycle (gs -> cups -> cups-filters -> gs), so when that stalled the +sort, gs's unrelated FreeType edge went with it. + +Now it breaks the LEAST that frees a node: pick the node closest to ready +(fewest blocking required dependencies, then fewest recommended), drop only +ITS recommended edges, sort again. Every other edge survives, and each pass +either frees a node or gives up. + +Verified on the real book, all five cases that have cost a session each: + freetype2 150 -> fontconfig 151 -> gs 153 + libxcb 143 -> xorg7-lib 155 + glib2 173 -> shared-mime-info 174 + +## A finished host is not the chroot (1.12.97) + +On the user's own machine: + + root ~ # lfs run + You are inside the LFS chroot, where `lfs build-system run` does not apply. + +inside_chroot() asked "does /usr/src/lfs-pkgusr exist" -- which is true of +ANY system these tools built, so a finished host was refused every build +command it has. An inference that is true of the DESTINATION cannot identify +the journey. + +The chroot says so itself now: `chroot enter` sets LFS_IN_CHROOT=1 in the +environment it starts. A set $LFS means we are outside (there is a build +target to work on). The old directory test survives only as a fallback for a +chroot entered by hand, and only when there is no $LFS and no /mnt/lfs. + +Worth checking next session: this probably explains other odd behaviour on +the host, since every command that guards on inside_chroot() was refusing +there -- possibly including the `packagemanager install Mako` hang, which +has never been diagnosed. + +## An entry's position does not move a dependency (1.12.96) + +texlive built at 3/102 despite `book texlive` being the LAST entry: it is +harfbuzz's RECOMMENDATION, so it arrives inside an earlier entry's plan, and +where the entry sits in the file has nothing to do with it. (1.12.88 said +"TeX last" and was wrong about the mechanism.) + +Its own failure is upstream anyway -- texlive-source's configure looks for +`freetype-config`, which FreeType removed years ago. + +So texlive is now `avoid`ed: nothing in this stack needs it (the book's +`make ... pdf ps` manual builds are commented out, asymptote and biber are +avoided), and it can be built on its own later: + packagemanager install texlive --run + +STILL UNRESOLVED: the `packagemanager install Mako` hang on the HOST. It +stops after "# user 'p_mako' already exists" with both tools now agreeing on +the name. Asked twice for `ps -ef --forest` / `/proc/<pid>/wchan` and not +yet received; my hypothesis is the pre-build tree snapshot, which on a full +host system (millions of files, unlike a chroot) may simply be slow and +silent -- but that is a guess and the process tree would settle it in +seconds, as it did for the `bash -e` hang. + +## Present-and-empty is an answer (1.12.95) + +The user set `"pkgusr_prefix": ""` and the two tools disagreed in the same +line of output: + + $ /usr/bin/lfs-helper pm-install mako ... <- packagemanager: no prefix + Install p_mako <- lfs-helper: prefix "p" + +packagemanager honours an empty value ("no prefix", documented since +prefixes were introduced). lfs-helper's reader tested `[ -n "$v" ]`, so an +empty value looked like a MISSING key and fell through to the default `p`. +It now checks whether the KEY EXISTS and takes its value, empty or not. + +WARNING recorded for the user: their tree was built with `p_`, so setting +the prefix empty now makes every tool look for accounts that do not exist. +The prefix is a decision taken before the first account, not a runtime +switch. For that machine the answer is to put "p" back. + +Not changed: the collector-prefix reader has the same shape, but an empty +collector prefix would produce group names like "_vim", so "present and +empty" is not obviously meaningful there. Left alone deliberately. + +## "Cannot judge" is not an answer (1.12.94) + + packagemanager update --reinstall --yes --recursive mako + ... 1 it cannot judge: Mako + +True, and useless: Mako has no install record because it is NOT INSTALLED, +and the user was asking to build it. When packages are NAMED on the command +line, the report now says so and gives the command: + + These have no record here because they are NOT INSTALLED. To build them: + packagemanager install Mako --run + +(The first attempt read `args` inside a helper that never had it -- a +NameError the suite caught on the same run. The package names are passed in +as an argument now.) + +## Type what you were shown (1.12.93) + + blfs search mako -> Mako Mako-1.4.1 + packagemanager update mako -> ERROR: 'mako' not found + +BLFS anchors are not all lower case: the Python module pages keep their +project capitalisation (Mako, PyYAML, Cython). `search` matched +case-insensitively and every other command looked the anchor up exactly, so +a user who had just been shown a name could not use it. + +resolve_anchor() returns the book's own spelling for whatever case was +typed, and is applied at every command that takes an anchor from the user +(debug, order, script, deps, install...). A name that really is absent comes +back unchanged, so the error message still says what was asked for. + +(Two mistakes of mine while doing it, both caught immediately: a blind +string replacement rewrote a loop body, and the resolver assumed `book` was +a dict when it is a Book object with a .packages map.) + +## The edited script was never even looked at (1.12.92) + +The user edited install_xmlto-0.0.29 -- the file the tools tell them to edit +-- and the run still used the generated one, without the "using LOCAL +script" line. + +_script_name_version matched only + + name_version="..." + +and since 1.12.18 the generator emits book strings SINGLE-quoted (a dollar +sign in prose is not a variable). So it returned None, the local-script +condition never held, and EVERY user edit has been silently ignored since +that release -- including the ones this session told people to make. It +accepts either quote now. + +The 1.12.91 backup message was treating a symptom of this: edits were not +being overwritten so much as never read. Both fixes stand. + +Checked for the same assumption elsewhere: the only other double-quote-only +reader (the book-set variables in _env_wanted_by_scripts) matches what the +generator actually emits there, so it is correct. + +## An edit must not vanish in silence (1.12.91) + +xmlto's download 404s at pagure.io AND at the LFS mirror -- the book's URL is +simply dead. The user fixed the link in a script and found it broken again, +and asked the right question: do we overwrite the install files? + +Three files, and only one is safe to edit: + install_<name>-<version> the canonical one -- resolve_script prefers it + and never rewrites it. EDIT THIS. + install_<name> regenerated whenever the tools move on + install_p_<name> the runner's copy, rewritten every run +The failure box has always named the first one, but the other two look just +as editable, and losing an edit to them was silent. Now a differing file is +saved as install_<name>.edited with a message saying where edits belong. + +STILL OPEN for the user: xmlto's tarball has to come from somewhere. Options +are dropping it into /sources by hand (the fetch checks there first), or +correcting the url= in install_xmlto-0.0.29. Nothing in this desktop needs +xmlto directly -- it arrives via xdg-utils -- so `avoid xmlto` is also +reasonable. + +## Our variable name broke someone else's build (1.12.90) + +libvpx configured correctly, created its headers, and then every compile +said `vpx_config.h: No such file or directory` -- with the file sitting in +the build directory, readable. + +Its build/make/Makefile: + BUILD_ROOT?=. + CFLAGS+=-I$(BUILD_PFX)$(BUILD_ROOT) -I$(SRC_PATH) +`?=` yields to the ENVIRONMENT, and run_phase_as exported BUILD_ROOT (the +package's source directory) into every build -- so libvpx compiled with +-I<source root> instead of -I. and never saw its own config header. + +Ours is now LFS_BUILD_ROOT; the generated script still calls it BUILD_ROOT +internally (`BUILD_ROOT="${LFS_BUILD_ROOT:-${BUILD_ROOT:-$PWD}}"`) but as a +plain shell variable that is not exported to make. + +The general lesson: a name in the environment belongs to whoever else might +use it. Every variable this project injects should be prefixed -- +LFS_CC_PHASE and PM_* already are; BUILD_ROOT was the exception, and it took +a package whose Makefile happened to share the name to find it. Worth a +sweep of what else run_phase_as passes. + +(Found by reading libvpx's actual Makefile from the tarball's repository +rather than guessing -- three commands, one cause.) + +## A conditional block is conditional wherever it appears (1.12.89) + + make: *** No rule to make target 'install-saslauthd'. Stop. + +The book introduces that command with "If you need to run the saslauthd +daemon at system startup, install the saslauthd.service unit included in the +blfs-systemd-units package" -- a conditional block, and one that needs a +package this system does not have. + +extract_commands_phased has guarded conditional blocks since 1.12.6, but +extract_commands -- used for the CONFIGURATION section -- did not, so every +"If you want..." command in a Configuration section has been running all +along. The guard is applied there too now; cyrus-sasl's line is emitted +commented, with the book's sentence above it. + +(Read the script first: it named the failing line and where it came from in +one look. Same as libassuan.) + +## TeX last, because nothing needs it (1.12.88) + +The user asked whether TeX is needed by other packages or can wait. With +1.12.87 the answer is clear: NOTHING in this stack needs it. The only +commands that wanted TeX were the book's `make -C doc pdf ps` manual builds, +now commented out, and the packages that genuinely require it (asymptote, +biber) are avoided. TeX is useful to the USER, not to the build. + +So `book texlive` is the LAST book entry -- after everything the desktop +needs, before the git block (the suite enforces that the cloned entries stay +last, and caught the first attempt at putting texlive after them). + +## Read the script before fixing it (1.12.87) + +The user pasted the generated script and it settled four releases of +guessing in one line: + + 153: ./configure --disable-doc --prefix=/usr <- the fix HAD applied + 159: make -C doc pdf ps <- this is the failure + +--disable-doc does not exist in libassuan (autotools accepts unknown +--disable-* silently), so the fix was WRONG, not undelivered -- and three +releases went into delivering it better. Reading the script on the first +failure would have shown the cause immediately. + +Two changes: + * `make ... pdf/ps/dvi` lines are emitted COMMENTED -- typesetting a + manual is not installing a package, and those targets need TeX. html + and info builds are untouched (texinfo is in LFS). + * texlive is BUILT now, at the user's call: it is generally useful and it + removes the whole "needs TeX" class. It goes to /opt/texlive/2025, a + value the generator lifts from the book's own PATH page (1.12.49), so + nothing has to be set by hand. + * the useless --disable-doc entries are gone from package-fixes.conf, with + a note saying why. + +## And it must reach a script that already exists (1.12.86) + +package-fixes.conf arrived on the system (the user checked) and libassuan +failed identically for the THIRD release. apply_machine_opts ran only where +a script was GENERATED -- and libassuan's script was already in its home +from the first failure, so resolve_script returned it untouched. Every +package that has failed once was in that state. + +The merge now runs on the local path too, and is idempotent (an option +already present is not added again, verified over three applications). + +Also: TOOL_VERSION still read 1.12.82 while the build id moved -- the +version bumps had been editing a string that no longer matched. The user +spotted it in --version output. + +Three releases for one fix, each blocked by a different link in the chain: +the file was not shipped, then not overwritten, then not applied. A fix is +not delivered until it reaches the code path that runs. + +## A fix must be able to arrive (1.12.85) + +1.12.84 shipped libassuan's --disable-doc in machine.conf, and libassuan +failed again with the identical error. machine.conf is NEVER OVERWRITTEN by +the sync -- correctly, it holds the machine's mesa options -- so a workaround +shipped in it can never reach an existing system. The fix was in the +release and not on the machine. + +Two files now, and the difference is the point: + package-fixes.conf -- workarounds the TOOLS ship (cairo's ctime_r, + libassuan/libgpg-error --disable-doc, gpgme). + Refreshed with every release. + machine.conf -- the MACHINE'S choices (mesa's drivers). Never + overwritten, and it WINS where both name a key. + +The general lesson, which cost two releases: a file the user owns cannot +also be a delivery channel. + +## machine.conf reaches autotools (1.12.84) + +libassuan's `make` builds assuan.pdf, which needs TeX -- and TeX is on the +avoid list on purpose. The package has a switch for it (--disable-doc), but +machine.conf could only inject into `meson setup` lines, so an autotools +package had no way to take a machine option at all. + +`configure_args` is appended to the ./configure line now; every other key is +still a meson -Dkey=value. Shipped for the three packages in this plan that +build TeX manuals: libassuan, libgpg-error (--disable-doc) and gpgme +(--disable-gpg-test). + +This is the same shape as the cairo ctime_r workaround (1.12.16): the cure +lives in machine.conf so it survives a regenerated script, and it is per +package rather than a rule about all of them. + +## The prune was wrong, reverted (1.12.83) + +1.12.82's reachability prune broke harfbuzz: + Dependency 'graphite2' is required but not found +graphite2 IS a stack entry, and it was dropped because `blfs order +--anchors` reports ONE parent per package -- the first that pulled it in. +When that recorded parent is an ignored package, a dependency several +packages share is pruned with it. Pruning needs every edge; that output +carries one. + +Reverted. The KDE leaves that okular alone pulls in are named in the stack +instead (plasma-activities, libkexiv2, kwindowsystem, libfm-qt, the lxqt and +gnome portals, gnome-desktop) -- more lines, but each one true. + +Two attempts at automatic exclusion (deps=required in 1.12.47/70, this +prune) have both been reverted after breaking a build, and both times the +naming approach the user asked for turned out to be the correct one. Worth +remembering before a third attempt: the book's graph does not carry enough +information to prune it safely from one walk. + +## Prune the graph, not the list (1.12.82) + +plasma-activities failed on a missing ECM -- a KDE package, in a plan where +okular is avoided. This was the limit written down in 1.12.78: --ignore +removed the named packages and left the tails only they pulled in. okular +was skipped; plasma-activities, which nothing else wants, was built anyway, +straight into the KDE Frameworks page that this book edition does not carry. + +The plan now prunes by REACHABILITY: `blfs order --anchors` reports each +package's parent, so a package whose only path to the target runs through an +ignored one is dropped with it. Verified on the shape that failed -- +ignoring okular drops plasma-activities and qtbase, keeps poppler and +webkitgtk. + +That closes the honest gap left in 1.12.78, and with it the KDE branch. + +## Say what to do about it (1.12.81) + +util-macros stopped with "this package needs XORG_CONFIG to be set" -- the +guard doing its job (1.12.47), but saying WHAT and not HOW. The value had +been set before the snapshot restore and went with /etc/pkgusr/build.env. + +The failure box now prints the command: + ! XORG_CONFIG is not set for builds. Set it once and it applies to + ! every package (builds run as a package user, so exporting it in your + ! shell does not reach them): + ! packagemanager env --set XORG_CONFIG=... # or: packagemanager env --ask + +NOTE for a fresh tree: build.env is not part of the snapshot, so the Xorg +variables have to be set again after a restore. Worth considering whether +`packagemanager setup` should ask for them. + +## Do not throw away a build that succeeded (1.12.80) + +Two bugs, one of them expensive: + +**--ignore twice, the second by letter.** The command line read + --ignore texlive,libreoffice,... --ignore t,e,x,l,i,v,e,,,l,i,b,... +A leftover second block re-joined the already-joined STRING, one comma per +character. Removed; there is one list and one flag. + +**LLVM recompiled from scratch.** It had built all 4343 targets and failed +on the systemd-run line at the very end -- and the retry wiped the source +tree and did it again, because `--phase all` always called +clean_stale_source. That is right for a fresh build and wrong for a retry. +The phase records already know what finished (1.11.55), so a retry with +'build' on record keeps the tree and says so: + # keeping the source tree: 'build' already finished here + # (rebuild from scratch with: lfs-helper build llvm --phase unpack --force) +A package with only 'unpack' recorded, or nothing, is still cleaned. + +## The whole systemd family (1.12.79) + + [4343/4343] Linking CXX executable bin/obj2yaml + # skipped (no systemctl on this system): systemctl --user start dbus + install_llvm: line 166: systemd-run: command not found + +LLVM built every one of its 4343 targets and then died on the last line but +one. systemctl was shimmed (and did its job, visibly); systemd-run was not, +and on a SysV system a missing command aborts the phase. + +The shim now covers the family the systemd book actually uses: systemctl, +systemd-run, systemd-tmpfiles, systemd-sysusers, systemd-machine-id-setup, +systemd-hwdb, loginctl, busctl, journalctl, timedatectl, hostnamectl, +localectl, udevadm. Each is defined ONLY when the real binary is absent, so +nothing changes on a systemd host. Verified in an isolated shell where the +binary genuinely does not exist: + # skipped (no systemd-run on this system): systemd-run --user foo + +(The user is building with the systemd-flavour book on a SysV system -- see +1.12.4, where the default was made init-aware. Switching the book would +avoid this class entirely; shimming keeps the current build going.) + +## Follow the book, name what you do not want (1.12.78) + + CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! + Dependency "icu-uc" not found, tried pkgconfig + +Two packages in a row, same shape: the book calls a dependency +"recommended", and the package's own build command hard-requires it. +--ignore-recommended is global, so it strips those too -- and naming them one +failure at a time is a treadmill. The user asked the right question ("are we +still installing required deps?"): yes, and that was never the problem. + +So deps=required is off the shipped stack. The book's graph is followed as +written, and the branches this desktop does not want are NAMED -- which is +the rule the user set for texlive and which the stack format already had: + + avoid qt6 / okular / phonon / vlc / cups / openjdk / java / apache-ant + avoid postgresql / openldap / unixodbc / graphviz / gegl + avoid texlive / libreoffice / asymptote / biber + +And a real bug found while doing it: `avoid` was ONLY A MESSAGE. It printed +"never built by this stack" and passed nothing to the installer, so those +packages were built anyway. It is now handed to every entry's resolve. + +Honest limit: --ignore removes the named packages, not the dependencies they +alone pull in, so some of qt6's tail may still build. If that shows up as +wasted hours, the fix is to prune the graph rather than the list -- worth +doing properly, not at the end of a session. deps=required remains +available per entry for anyone who wants it. + +## The deps=required trade-off, paid (1.12.77) + + CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! + +cmake declares NO required dependencies and four recommended ones (cURL, +libarchive, libuv, nghttp2) -- and its own build turns the system libraries +ON, so a recommendation is a hard requirement in practice. This is exactly +the cost of deps=required that was flagged when it went in, and the answer +is the one that was agreed: NAME what the build needs. Those four are stack +entries now; the plan goes 99 -> 102. + +Expect a few more of these as the build climbs -- each arrives as a clean +configure error naming the package, and costs one line. That is the deal +deps=required makes: 170 packages not built, in exchange for naming the +handful the book calls optional and the code does not. + +## A book entry can become a clone (1.12.76) + +--no-git skipped the 14 declared git/tar entries and then cloned elogind +anyway. elogind is `book elogind url=...` (1.12.5): a BOOK entry that falls +back to git when the book does not carry it -- and that rewrite happens +during the judging, after the early filter has run, so its kind was still +"book" when the filter looked. + +The decision is taken in the run loop as well, where the kind is final: + elogind: skipped (--no-git) +Same lesson as the ordering passes: a filter that runs before the data is +final only filters what it happened to see. + +## The extras come last (1.12.75) + + install_elogind: line 54: git: command not found + +git is itself a BLFS package, and the stack was cloning things before it +existed. The git and tar entries -- elogind, seatd, wlroots, sway, foot, +the Sw* tools, dejavu -- are now grouped at the END of sway.stack, after a +`book git` entry, so the book part of the desktop finishes first and the +extras follow. `packagemanager stack sway --no-git` skips them entirely, +which is what the user wanted for now. + +Noted while reading that output, not chased: the failure box shows + script: /usr/src/p_elogind/install_p_elogind + staged: /usr/src/pkgusr/p_elogind/install_elogind +-- p_elogind's home is /usr/src/p_elogind, the hint's default, which +fix-home is supposed to repair at account creation (1.11.55). It was +created by the git-entry path, which may not go through _pm_add_account. +Worth a look next session. + +## Where the stacks actually are (1.12.74) + +1.12.73 refreshed the stacks -- from `os.path.dirname(__file__)/stacks`. The +running tool is /usr/bin/lfs, so that is /usr/bin/stacks: nothing, in +silence. The host had 31 book entries and the chroot kept 23, and the only +clue was the ABSENCE of a line. + +It looks in both places now, nearest first: beside the tool (a checkout) and +/etc/pkgusr/stacks (an installed system), never at the chroot's own copy. +And finding none is no longer silent: + # no stack files found to refresh (looked beside the tool and in + # /etc/pkgusr/stacks) + +Three fixes for one sync (1.12.72 stacks at all, .73 one path + no +downgrade, .74 the right source). Each was found only because the user +checked a number rather than trusting the tool's own report -- worth doing +again the next time something claims to have refreshed itself. + +## One sync path, and never backwards (1.12.73) + +Per the user: `lfs build-system run`, `lfs run` and `lfs chroot enter` must +all keep the chroot current. Two of them did (both call +_refresh_chroot_tools); `sync-tools` installed the helper and the skeletons +and left the other three tools and the stack files to whatever was already +there -- so WHICH COMMAND YOU USED decided what got updated. sync-tools +calls the same function now: one sync path, every entry point. + +And it never goes backwards. Stack files carry `# stack-version:` and are +skipped when identical, kept when the copy in the chroot is NEWER (a file +edited in there, or a newer release), replaced only when the host's is +newer. machine.conf is still placed only when absent -- it is the machine's +own options. + + # kept the newer sway.stack already in the chroot (9.9.9 > 1.12.73) + # refreshed in the chroot: kernel-sway.sh, sway-session.sh + +## The stacks travel with the tools (1.12.72) + +--status still said 272 after 1.12.71, and the entry list gave it away: 39 +entries, none of the ones added in 1.12.70. The chroot was reading a +sway.stack from days ago. + +_refresh_chroot_tools copied the four BINARIES and nothing else, so a chroot +kept whichever stack file it was first given -- the user updated the tools +four times and kept reading the same stale file, checking a number that +could never move. The stack files are refreshed with the tools now. +machine.conf is the exception: it is the machine's own build options, so it +is placed only when absent, never overwritten. + +(Both --status bugs in a row were the same mistake in different clothes: +reporting something other than what the run would do.) + +## The status must count what the build will build (1.12.71) + +The user ran --status after 1.12.70 and saw "4 of 272 installed" -- the old +number. The build would have used the required graph; the STATUS asked +blfs_order_anchors without the entry's deps=, so it counted a plan nobody +was going to build. It passes no_recommended now and reports 99, the same +graph the run uses. (Also: the entry-table header still printed in status +mode; gated.) + +A status that does not match the run is worse than no status -- it was the +number the user was asked to check before committing hours of build. + +## The graph you asked for (1.12.70) + +`--status --full` showed 272 packages for a sway desktop: LibreOffice, +OpenJDK, CUPS, Qt6, okular, VLC, texlive, PostgreSQL, the xfce and lxqt +portals -- every one of them arriving as a RECOMMENDATION, and one branch +(KDE) unbuildable because its dependency page is out of book. + +So the book entries now carry `deps=required`, and the recommendations this +desktop actually needs are NAMED as entries instead of inherited: harfbuzz, +graphite2, pango, gdk-pixbuf, hicolor-icon-theme, adwaita-icon-theme, +glib-networking, and the gstreamer family that was already there. + + 272 packages -> 99 + +The first attempt at this (1.12.47) was reverted for a good reason and is +worth remembering: texlive's own dependencies are all recommended, so +dropping the class left it with none and the ORDER put it first. That was +an ordering bug, fixed in 1.12.69 by the topological sort -- not an argument +against choosing a graph. The mechanism is per-entry, so anything that +turns out to be needed is one named line, and the failure it causes first +("X not found" at configure) says exactly which. + +The user wants LibreOffice eventually: `book libreoffice` as its own entry, +placed last, once the desktop runs. Deliberately not now. + +## A sort, not a repair pass (1.12.69) + +The user asked the right question after the third ordering fix: is this a +general ordering problem? Yes -- and the approach was wrong in kind. +_dependencies_come_first was a REPAIR PASS: walk the list, notice a +dependency that landed too late, move it earlier. That can only fix the +cases it happens to look at, and the next pass undoes them -- which is +exactly what happened three times (texlive, GLib, libxcb). + +It is Kahn's algorithm now: a node is emitted only when everything it +depends on has been emitted, ties keep the walk's order so the result stays +stable. Cycles are broken by STRENGTH -- recommended edges inside a cycle +go first; a cycle of required edges is the book's own knot and those nodes +are emitted in their original order rather than pretending it was solved. + +Verified on the real book, all four cases that cost a session each: + libxcb 117 -> xorg7-lib 189 glib2 103 -> shared-mime-info 132 + freetype2 130 -> fontconfig 170 cairo 107 -> texlive 108 + +### mesa/glslangValidator is NOT an ordering bug + +The book's mesa page lists required: Xorg-Libraries, Libdrm, Mako, PyYAML -- +glslang is not there. mesa needs glslangValidator only because +machine.conf sets -Dvulkan-drivers=amd: the USER'S OPTION creates a +dependency the book does not declare, and no dependency order can place +what nothing declares. sway.stack now names `book glslang` before mesa, +with the reason in a comment. Any future machine option that pulls in a +tool needs the same treatment. + +## Required has the last word (1.12.68) + +libX11 could not find xcb, and the user's three commands settled it in one +go: `pkg-config --exists xcb` -> 1, `search libxcb` -> NOT INSTALLED, no +xcb.pc on disk. Not a ghost record, not a PKG_CONFIG_PATH problem: libxcb +was never built, because the plan had + + 32: xorg7-lib + 37: libxcb + +and xorg7-lib REQUIRES libxcb, with no edge back at all. + +_dependencies_come_first (1.12.52) ran the required pass FIRST and the +recommended pass second -- and the second pass could move things the first +had just fixed. Reversed: recommended first, required LAST, so nothing +weaker can put a required dependency after its dependant again. Verified on +the real book: libxcb 32 -> xorg7-lib 33, and the earlier cases still hold +(freetype 25 -> fontconfig 26 -> texlive 42; GLib before shared-mime-info in +three different plans). + +Third ordering fix in this session (1.12.33, 1.12.52, this one). The +lesson each time: a pass that reorders must be the LAST word on the +strongest constraint, or a weaker pass silently undoes it. + +## An absolute path walks past every wrapper (1.12.67) + +The ldconfig wrapper (1.12.66) sat unused: the book's loop says + + as_root /sbin/ldconfig + +and the wrappers live in a directory on the package user's PATH, so an +absolute path never reaches them. This is not about ldconfig -- ANY of the +wrapped tools (install, chown, chgrp, chmod, mkdir, cp) written as /sbin/x +or /usr/bin/x bypasses the entire package-user mechanism, silently. + +Generated commands now call the wrapped tools by NAME: /sbin/ldconfig -> +ldconfig, /usr/bin/install -> install, and so on. Comments are left alone +and a longer name (/sbin/ldconfigXYZ) is not a match. + +The wrapper mechanism has been in this project since the beginning; a book +command with an absolute path defeated it the whole time, and only a +sticky-directory failure made that visible. Worth remembering when a +wrapper "does not fire": check how the command was spelled. + +## The library cache, and what ldconfig also does (1.12.66) + + /sbin/ldconfig: Renaming of /etc/ld.so.cache~ to /etc/ld.so.cache failed: + Operation not permitted + +/etc/ld.so.cache is rewritten by every package that installs a library, and +ldconfig writes a temp file and RENAMES it over -- which a sticky /etc allows +only to the file's owner (here p_e2fsprogs). Unsealing /etc to fix that +would let every package delete every other package's configuration, so +instead package users get a wrapper that skips ldconfig, and the tools +refresh the cache as root when the install finishes -- the mandb treatment. + +THE USER CAUGHT THE DANGER IN THAT: ldconfig does two jobs. Besides the +cache it creates and updates SONAME symlinks (libfoo.so.6 -> +libfoo.so.6.2.1) in the library directories -- and run as ROOT those land +root-owned inside package-owned trees. That is exactly how +libfreetype.so.6 came to be owned by p_libpaper earlier in this build. So +the refresh is `ldconfig -X`: rebuild the cache, touch no links. The +symlinks belong to the package that installs the library, and its own `make +install` creates them. + +## as_root, in a model with no root (1.12.65) + +The loop finally ran, and reached + install_xorg7-lib: line 198: as_root: command not found +The book defines as_root() in a NOTE -- prose, kept as a comment -- and the +loop calls it. + +In the package-user model there is nothing to elevate: the package user owns +the directories it installs into, and the wrappers cover what it may not do. +So when the commands call as_root, the script defines it as + as_root() { "$@"; } +placed before the commands that use it. A package that never calls it gains +nothing, and a mention inside a comment does not count as a call. + +## The book says "type this into a subshell" (1.12.64) + +The user's process tree found it in one look: + + bash install_xorg7-lib all + \_ bash -e <- idle, no children, holding the build + +Not a prompt after all. The Xorg pages use an interactive idiom: type +`bash -e`, paste the loop, type `exit`. A SCRIPT that runs `bash -e` gets a +shell reading its own stdin -- the terminal -- and waits there forever. + +The wrapper line and its matching `exit` are now removed: the phase IS that +shell and already runs under set -e. (Leaving the `exit` would have ended +the phase silently at that point, which is worse than hanging.) A package +without the idiom is untouched. + +Worth noting for the next session: the 1.12.63 "WAITING FOR AN ANSWER" +lines were written on a hypothesis that turned out wrong. They are still +right -- a prompt behind a pipe would have been invisible -- but the lesson +is the one this file keeps repeating: ps and /proc answered in seconds what +two rounds of reasoning did not. + +## A prompt behind a pipe looks like a hang (1.12.63) + +xorg7-lib printed "its own commands fetch them" and then nothing for twenty +minutes. `ps` showed lfs-helper alive with NO child compiler -- so it was +not building, it was waiting. + +Every prompt writes its question to /dev/tty (so it survives the pipes) and +reads the answer there, but the build's OUTPUT goes through tee -- so a +waiting prompt is invisible in the place the user is watching. All three +prompts now also announce themselves on the ordinary output: + + !! WAITING FOR AN ANSWER: which group should share /usr/share/X11/...? + +DIAGNOSIS STILL OPEN for the user's current hang -- asked for: + cat /proc/<pid>/wchan # what it is blocked on + ls -l /proc/<pid>/fd/0 # a terminal read? + ps -ef --forest # any child at all +If it is a prompt, pressing Enter releases it and 1.12.63 makes the next one +visible. If wchan says something else (a stuck wget, an NFS read), that is a +different bug and the evidence will say so. + +## A command whose input nothing produces (1.12.62) + +Xorg Libraries got past the tarball problem and died on + grep: *make_check.log: No such file or directory +The line that WRITES those logs is the optional test step, correctly +commented out; the line that READS them stayed active, and set -e failed the +page. + +Same shape as libuv's man page (1.12.19), one layer earlier -- at generation +instead of at run time. A reader (grep/cat/tail/...) of a .log that no +active command writes is commented out with a note. Deliberately limited to +.log files: a build artifact with a real name might be produced by make +itself, and guessing there would do harm. A page that does write a log +keeps its reader. + +## A meta page has no tarball (1.12.61) + + 2026-08-31 (313 KB/s) - 'index.html' saved [328504] + tar: ../: Cannot read: Is a directory + +"Xorg Libraries" is not a package: no pkg=, a DIRECTORY for link=, and its +own commands fetch the list (wget -i from an embedded md5 file, then a +for-loop over every tarball). The generator treated it like any package, +downloaded the directory index and tarred "..". + +unpack_pkg now returns early when there is no pkg -- there is nothing to +fetch or unpack, and the page's own commands do all of it. Xorg +Applications, XCB-Utilities and KF6 Frameworks are the same shape and get +the same treatment. Normal packages are unchanged (nettle still fetches and +extracts). + +## Another language's variables, and a prompt that says who wants it (1.12.60) + +The user, asked for PACKAGE_PREFIX_DIR: "i totally dont know what to set +there". Neither would anyone -- it is a CMAKE variable, written by a sed +expression into a .cmake file: + + sed -e '/PACKAGE_INIT/i set(SAVE_PACKAGE_PREFIX_DIR "${PACKAGE_PREFIX_DIR}"... + +Inside single quotes the shell never expands it, which is a provable rule +rather than a heuristic: single-quoted spans are stripped before looking for +variable names. Both false positives are gone; KF6_PREFIX (a real shell +variable in the same command) is still asked. + +And the prompt now says WHERE the question comes from, because "a script +needs it" is not information: + KF6_PREFIX is needed by: kf6-frameworks-6.19.0 + the book's page for that package explains it: blfs debug kf6-frameworks + press Enter to skip -- the build stops at that package rather than + using an empty value +Book values are offered for KF6_PREFIX (/usr), QT6PREFIX (/opt/qt6), +TEXLIVE_PREFIX (/opt/texlive/2025) and XORG_PREFIX (/usr). + +## A value with spaces is one value (1.12.59) + +The first package built after setting XORG_CONFIG died at line 1: + + env: '--disable-static': No such file or directory + +build_env_pairs (1.12.54) printed NAME=value pairs straight into +`env $envpass bash ...`, and the shell split the flag list on whitespace, so +its second word was run as a command. Every package would have failed this +way -- the variable was set correctly, the plumbing lost it. + +Each pair is single-quoted now, with the one escape single quoting needs, so +values containing spaces AND apostrophes survive. Pinned in the suite with +exactly such a value. + +## An empty list is not a promise (1.12.58) + +With 219 stale scripts ignored, `env` said "nothing set, and no generated +script asks for anything" -- true, and useless: the user still could not set +anything BEFORE the run, because a guard only exists once the package's +script has been generated, which happens during the build. + +Two changes, neither of them a claim that the list is complete: + * `--refresh` drops scripts from an older blfs (they regenerate on demand), + so the current generator's questions can be seen; + * an EMPTY list now says what will actually happen -- a variable becomes + visible when its package's script is generated, the build warns before + that package starts (1.12.48) and stops rather than passing an empty + value -- and offers the one value we already know (XORG_CONFIG). + +The honest shape of the real fix, for a later session: ask blfs for the +variables of every package IN THE PLAN before the run starts, rather than +reading whatever scripts happen to be in /tmp. `blfs env <anchor>` using +_required_env_guard's detector is the missing piece; packagemanager then +walks the plan once and asks everything up front. + +## Old scripts ask old questions (1.12.57) + +The user reran `packagemanager env` after 1.12.56 and saw the SAME eleven +variables. The fix was fine; `env` was reading the scripts in +/tmp/packagemanager/install_files, which the previous blfs had written. It +now skips any script whose "### generated by blfs X" header does not match +the current generator (falling back to packagemanager's own version when +blfs cannot be asked) and says how many it ignored. + +### The user's second question: should those prefix directories exist? + +Yes -- /opt/texlive/2025 and /opt/qt6 are the packages' own prefixes, and an +install into a missing directory fails (`tar: /opt/texlive/2025: Cannot +open`). cmd_build's repair creates them today, but under the SHARED rule +(root:install), because /opt is an install directory -- while the user is +right that a package's own prefix belongs to that package. + +NOT changed here, deliberately: the directory-ownership rules were rewritten +three times this session (1.12.41/42/44) and each attempt broke something. +The shape of the fix is clear -- a directory created as a package's own +PREFIX (the build's --prefix, not a shared tree) should be owned by that +package, with the shared rule reserved for directories more than one package +writes -- and it wants a fresh session and its own tests. + +## Ten of eleven questions were mine to answer (1.12.56) + +`packagemanager env` on the real system listed eleven variables, and the +user asked the right question: how am I supposed to know what to set for the +ones with no default? Answer: they should never have been asked. + + EUID -- the shell's own + DOCNAME -- a for-loop variable in the script's install text + PACKAGE_PREFIX_DIR -- set by the script itself + SAVE_PACKAGE_PREFIX_DIR, JT_JAVA -- likewise + XORG_PREFIX -- "the book says <PREFIX>", which is a placeholder + +Three causes, all in the guard: shell built-ins were not excluded; the +"what does the script set itself" scan only read the BUILD text, so loop +variables and assignments in the install section looked like the user's job; +and a prose placeholder (<PREFIX>) was accepted as a value. Now: a list of +shell variables is excluded, assignments/`for`/`read` are collected from the +whole script, and any value containing <> is refused. + +Across util-macros, texlive, qt6, kf6-frameworks and poppler exactly ONE +question remains -- XORG_CONFIG -- and --ask offers the book's value for it. +A prompt the user cannot answer is worse than no prompt: it teaches them to +ignore the ones that matter. + +## A value in two halves is not a value (1.12.55) + +util-macros wanted XORG_CONFIG. The book does define it, in "Introduction +to Xorg-7": + export XORG_PREFIX="/usr" + export XORG_CONFIG="--prefix=$XORG_PREFIX --sysconfdir=/etc \\ + --localstatedir=/var --disable-static" +1.12.49 rejected it because the value names another variable. That rule was +too blunt -- the other variable is defined on the same page -- so definitions +now FOLLOW THE CHAIN (up to four links) and are emitted dependency-first, +XORG_PREFIX before XORG_CONFIG. + +The first attempt then produced broken bash: the value continues over lines +with backslashes and a single-line regex captured half of it. Continuations +are joined now, and anything still ending in "\\" or carrying an unbalanced +quote is DROPPED rather than emitted -- a fragment in a build script is +worse than a question. + +XORG_CONFIG is exactly such a case (quoted, multi-line), so it stays a +question -- but `packagemanager env --ask` now offers the book's standard +value, so the answer is one Enter: + XORG_CONFIG [--prefix=/usr --sysconfdir=/etc --localstatedir=/var --disable-static]: + +## One place for the book's build variables (1.12.54) + +Exporting TEXLIVE_PREFIX or XORG_CONFIG in a shell never reached the build: +package builds run through `su -`, which drops the caller's environment. +So they live in one file, /etc/pkgusr/build.env (LFS_BUILD_ENV overrides), +and run_phase_as prepends it to every phase -- one place, every package, +every tool. + + packagemanager env # what is set, what is still needed + packagemanager env --ask # asks for each one a script needs, + # offering the book's own value + packagemanager env --set NAME=VALUE # one at a time + +--ask reads the guards out of the generated scripts (the same +"needs NAME to be set" lines the build would fail on) and pre-fills what the +generator already found in the book (1.12.49), so most answers are a press +of Enter. + +STILL WANTED, and not built (the user's second ask): a way to define a GIT +package interactively -- name, ref/branch, build and install commands -- +without editing a generated script full of template. The stack file already +carries exactly those fields + git foo url=... ref=... prog=... build="..." install="..." +so the command is a small one: ask four questions, write or update that +line. `packagemanager recipe <name>` is the obvious shape. Doing it half +now would be worse than not doing it; it is the next session's first task. + +## Status about the tree, not the file (1.12.53) + +`--status` reported "1 of 39 entries done" and reprinted the entry table -- +which says nothing about the hundreds of packages those entries pull in, and +was the wrong question. It now resolves each book entry's dependency order +and reports PACKAGES: + + ### stack sway ### + entries : 3 of 39 done + packages: 41 of 196 installed + next : llvm + building next: libpng, pixman, docutils, icu, libxml2, ... + (full tree: --status --full) + state : /usr/src/lfs-pkgusr/progress/stacks/sway.done + +Short by default, `--status --full` lists every package with its state, and +the entry table is suppressed in status mode. The installed judgement is +install_tag -- the same one `search` and the install plan use, so all three +agree rather than inventing a third answer. + +## A recommendation is a dependency (1.12.52) + +The user's upload settled it: the install plan had + texlive-20250308-source new recommended by harfbuzz <- #1 + FreeType-2.14.1 new required by fontconfig <- #20 +and `blfs order dbus` agreed (texlive 26, FreeType 46). So the ORDER was +wrong, not the plan's use of it -- packagemanager does use `blfs order`. + +_required_edges_win (1.12.33) moved only REQUIRED edges. texlive's +dependencies are ALL recommended (Cairo, libpaper), so nothing pulled it +after them. Third time this project has treated a recommendation as +optional and been wrong: _dependencies_come_first now applies both kinds -- +required first (it wins any disagreement), then recommended, skipping only +genuine mutual pairs, which the required pass has already settled. + +Verified on the real book: freetype2 25 -> fontconfig 26 -> cairo 40 -> +texlive 42, and GLib still precedes shared-mime-info (the 1.12.33 cycle). + +Also added, as the user asked: `packagemanager stack <n> --status` -- +how many entries are done, what is next, and where the state file lives, +without reading the scrollback. + +## No exclusions at all (1.12.51) + +Per the user, after 1.12.50 named texlive and LibreOffice as ignored: +"dont ignore thoose packages! just install them with their deps as we do +with the other ones." + +The shipped sway.stack carries no ignore= and no required-only. The book's +graph is built as the book states it, recommendations included. It costs +hours (texlive, LibreOffice, OpenJDK, Qt and their tails) and buys a system +that matches the book -- and it removes the class of failure that both +attempts produced, where a package arrived before something it needed +because the graph had been pruned. + +`ignore=` remains available as a per-entry option for anyone who wants it; +nothing ships using it. + +## Never drop a class of dependencies (1.12.50) + +texlive built before FreeType and died on + did not find freetype-config required for system freetype2 library +The user's `packagemanager deps texlive` showed FreeType in the plan, +required by Fontconfig, long before texlive -- so why? + +Because of MY OWN 1.12.47 change. `required-only=1` +(--ignore-recommended) was meant to keep texlive and LibreOffice out of a +sway build; it also removed texlive's OWN dependencies, which are ALL +recommended (Cairo, libpaper, the PATH page). Left with no dependencies, +texlive was scheduled FIRST and built against a freetype that did not exist. +I traded a long build for a broken one. + +A recommendation is still a dependency; the book knows why it is there. +Unwanted packages are NAMED instead: `ignore=texlive,libreoffice` on the +entry (--ignore), one explicit decision per package, with the rest of the +graph built in full. required-only is gone, and the suite fails if it +returns. + +## The book already answered it (1.12.49) + +The user pointed at the page: TeX Live's PATH section writes +/etc/profile.d/texlive.sh and sets + TEXLIVE_PREFIX=/opt/texlive/2025 +So asking the user to export it was asking them to repeat the book. + +_page_env_definitions collects, for every variable the build commands use +and the script does not set, a simple assignment from the book -- searching +the package's own section first and then the WHOLE BOOK, because this one +lives in a different section ("Setting the PATH for TeX Live", anchor +tex-path, which our parse lists as its own page). Definitions that +themselves depend on another unknown variable are ignored: that is the same +question moved, not an answer. Only what remains unanswered is asked of the +user (1.12.47/48). + +Verified on the real book: texlive's script now opens with + TEXLIVE_PREFIX="/opt/texlive/2025" # set by this package's own page +and the guard no longer fires; nettle and other ordinary packages gain +nothing. + +## The warning belongs before the build (1.12.48) + +The user asked for the general safety: not just TeX Live, and not only in +the log afterwards. Two layers now, and both are general -- they act on +whatever the BOOK's commands reference, package by package: + + * the generated build phase refuses at the top, naming the variable + (1.12.47), instead of handing configure an empty string; + * packagemanager warns THE MOMENT THE SCRIPT IS GENERATED -- which is + before unpacking, patching and configuring -- if such a variable is + unset in the environment: + texlive needs TEXLIVE_PREFIX set before it can build: + export TEXLIVE_PREFIX=... + and says nothing when it is set. + +resolve_script is the one place every script is born, so every package gets +this, including local and git ones. + +Still worth doing (not done): collect these across a whole PLAN and print +them once before the run starts, so a 200-package stack asks for everything +it will need up front rather than at package 47. The pieces are in place -- +_warn_unset_env's regex reads the guards out of any generated script. + +## An empty expansion is not a value (1.12.47) + + configure: error: expected an absolute directory name for --datarootdir: + tar: --strip-components=1: Cannot open: No such file or directory + +TeX Live's page tells the USER to `export TEXLIVE_PREFIX=...` first. Unset, +the shell expanded it to nothing, configure got an empty flag and tar read +an option as a filename -- nonsense far from its cause. Generated build +phases now refuse at the top and name the variable: every $NAME the commands +use that the script does not set itself and that is not part of its own +vocabulary. A normal package gains nothing (nettle: 0 guards). + +And the deeper question the failure raised: WHY is texlive in a sway build +at all? harfbuzz RECOMMENDS texlive -- and LibreOffice. Both are +enormous, neither belongs in this desktop, and a recommendation is a +preference. Stack entries can now say `required-only=1` +(--ignore-recommended); webkitgtk, gtk4 and mpv carry it in the shipped +sway.stack. + +Watch for the trade-off: required-only also drops recommendations that +matter (harfbuzz's Graphite2, icu). If something later fails for a missing +recommended dep, name it explicitly as its own `book` entry rather than +dropping the flag. + +## cairo built its own freetype (1.12.46) + +The four diagnostics answered it in one line: + + ls -l /usr/lib/libfreetype* + -rwxr-xr-x 1 p_cairo p_cairo 964504 /usr/lib/libfreetype.so.6.20.1 + grep: /usr/src/pkgusr/p_freetype2/pkg.lst: No such file or directory + +There is no p_freetype2 AT ALL -- freetype2 was never built. cairo's meson +fell back to its BUNDLED SUBPROJECTS and installed its own freetype and +fontconfig: owned by p_cairo, with no development symlinks. The planner +then saw a libfreetype on disk and never scheduled the real package. Every +"cannot find -lfreetype" in the last four sessions -- ghostscript, cairo, +the "missing dev symlink" diagnosis, the ghost-record hunt -- traces to that +one silent fallback. + +Generated meson lines now carry --wrap-mode=nofallback, so a package FAILS +and names the dependency instead of vendoring it. Added only where not +already specified, never to a commented line. SCRIPT_VERSION 8. + +USER ACTION on the tree: remove cairo's vendored copies and let the real +packages build -- + rm -f /usr/lib/libfreetype.so* /usr/lib/libfontconfig.so* + lfs-helper build cairo --phase all --force # after freetype2/fontconfig +then resume the stack (it will schedule freetype2 and fontconfig properly). + +STILL OPEN: the planner accepted "a library exists on disk" as proof a +package is installed, with no account, no manifest and no VERSION. That +check should require a RECORD, not a file someone else installed -- the +wrap-mode fix removes the cause, this would remove the class. + +## Ownership is a fact, a manifest entry is a claim (1.12.45) + +cairo failed on -lfontconfig / -lfreetype and the box told the user to +"rebuild cairo" for a FONTCONFIG library. 1.12.30 had made the manifests +the primary source of ownership, and a failed cairo round had swept those +files into cairo's manifest (the over-claiming species of 1.12.13). The +file's OWNER is a fact; a manifest entry is a claim, and claims have been +wrong before -- so stat comes first now, manifests only when the owner is +root. + +STILL OPEN, and the important one: on a REBUILT tree, freetype2 and +fontconfig are installed yet libfreetype.so / libfontconfig.so (the +development symlinks) are missing, exactly as on the old tree. That is no +longer explicable as old damage. Next session starts here: + ls -l /usr/lib/libfreetype* + grep -c . /usr/src/pkgusr/p_freetype2/pkg.lst + grep libfreetype /usr/src/pkgusr/p_freetype2/pkg.lst + tail -40 /usr/src/lfs-pkgusr/logs/freetype2-all.log +The suspects, in order: (1) the install wrapper of 1.12.19 skipping a real +install, (2) something in the phase split dropping the symlink step, (3) the +package genuinely not creating it under meson. Do not guess -- read the +freetype log first. + +## The parent directory decides the group (1.12.43/44) + +The directory came out right and the GROUP was wrong: + + /usr/share/vim/vimfiles p_cmake nimgnu_cmake + +named after the package that created it, because grant_dir_access names the +group after the DIRECTORY'S owner -- which by then was cmake. The group +belongs to the tree, not to whoever installed first. + +The user then stated the rule exactly, and it is simpler than what I built: +when a package creates a directory inside another package user's directory, +**if the parent already carries a collector group, use the parent's; if not, +ASK** -- the ordinary question, not a silent default. My first attempt +asked "does a group named after the tree owner exist anywhere", which is not +the same thing and would reuse an unrelated group. + +share_new_dir_with_tree_owner now: + * parent carries <prefix>_* -> inherit it, no question (the decision was + made when it was put there; a subdirectory of a shared directory is + shared the same way) + * otherwise -> ask, offering the tree's name or the creator's or a typed + one. Asked even under --yes, because naming a group is a decision, not + a confirmation; with no terminal it falls back to the TREE's name, never + the creator's. + +Also fixed in passing (1.12.42): mkdir -p left intermediates root:root, and +the next round read that artifact as "root's tree" -- the ancestor walk now +skips past root:root directories that are not real install directories, and +every created component gets the rule. + +## A directory we made is not evidence (1.12.42) + +1.12.41 chose the rule by asking the nearest EXISTING ancestor, and the +user's ls showed why that is not enough: + + /usr/share/vim/vimfiles/ root:root <- made by mkdir -p + /usr/share/vim/vimfiles/syntax root:install <- wrong rule, round 2 + +`mkdir -p` creates intermediates as root:root. Round 1 made vimfiles that +way; round 2 read that artifact OF OUR OWN MAKING as "root's tree" and +applied the system rule inside p_vim's tree. cmake then could not chmod it, +which is the same failure as 1.12.41 one directory deeper. + +Two fixes: the ancestor walk continues past a root:root directory that is +not a real install directory (it decides only on an install directory or a +package-owned one), and EVERY component created gets the rule -- not just +the last one -- so no root:root intermediates are left behind to mislead the +next round or to block the package that must install into them. + +The suite now replays the user's exact three-round sequence (indent, then +syntax, then a real install dir) and checks all three outcomes. + +## Two rules, and I applied the wrong one (1.12.41) + +The created directory failed cmake a second way: + cannot set permissions on "/usr/share/vim/vimfiles/indent": + Operation not permitted +and the user named the design error exactly: "the base dir belongs to p_vim, +so the dir created within it should belong to nimgnu_vim ... as we do for +all other dirs". + +root:install is the rule for the SYSTEM'S OWN install directories -- the +ones in installdirs.lst, which no package owns. /usr/share/vim/vimfiles is +not one of those: it is a new directory inside ANOTHER PACKAGE'S tree, and +the model has always answered that case with collector groups. Applying the +system rule there was wrong twice over: chmod requires OWNERSHIP, so an +installer cannot set modes on a root-owned directory; and it invented a +second rule where one already existed, which is what made vimfiles behave +unlike everything else on the system. + +create_missing_dirs_from_log now looks at the nearest existing ancestor: + * an install directory, or root-owned -> root:install, group-writable + * another package's tree -> owned by the package INSTALLING into it, then + grant_dir_access() shares it with the tree's owner through the ordinary + nimgnu_* group question +Verified both branches side by side in one run. + +USER ACTION: they are removing /usr/share/vim/vimfiles by hand, so the next +run recreates it under the right rule. If any other directory from the +1.12.39/40 runs is root:install inside a package's tree, the same applies -- +`lfs-helper verify` will not flag these, which is worth a rule of its own if +it recurs. + +## Ask before changing the system (1.12.40) + +The repair worked at last -- /usr/share/vim/vimfiles/indent exists, +root:install, group-writable -- and the user's objection was right anyway: +"that should never happen in any way if not 100%% required and I confirmed." + +The OWNERSHIP is deliberate and stays: giving the directory to p_cmake would +make cmake the owner of a subtree inside vim's tree, and block the next +package that installs vim runtime files. A directory two packages share +belongs to root:install; that is the model. + +The SILENCE was wrong. Creating a directory the book never mentions is a +change to someone else's tree, so confirm_change() now asks, and +create_missing_dirs_from_log goes through it. assume_yes (--yes / PM_YES / +LFS_ASSUME_YES) is that permission given in advance and the action is still +ANNOUNCED; with no terminal the answer is NO, because silence is not +consent. + +Worth applying the same rule to the other automatic repairs (takeover, +share_shared_files, collector grants) in the next session -- they change the +system too, and only the group grant currently asks. + +## An empty variable is not a prefix, it is a wildcard (1.12.39) + +The box from 1.12.38 answered it in one line: + auto-repair: enabled, gate: recognised, rounds run: 0 +and no "directories it could create" line at all -- so the detector returned +NOTHING on the real system while finding the path in every sandbox. The +difference was the root: + + case "$d" in "${SNAP_ROOT%/}"/*) # in a chroot SNAP_ROOT is "/", + # so this is ""/* and matches nothing + case "$d" in "$SRCROOT"/*) continue # with SRCROOT unset this is /* + # and excludes EVERYTHING + +Every sandbox used a root like /tmp/xx, where both patterns behave; every +real chroot uses /, where neither does. Three sessions of "the repair does +not run" came from those two lines. + +under_dir() is now the one implementation of "is this path under that +directory", with an empty directory meaning "/" rather than a wildcard, and +missing_dirs_from_log, takeover_files_from_log and the failure-box path +scanner all use it -- so the file takeover was equally broken in the real +chroot and is fixed by the same change. The suite now runs the detector +with SNAP_ROOT=/ and with SRCROOT unset, which is what would have caught +this on day one. + +## Put the evidence in the box (1.12.38) + +cmake failed a third time on the same missing directory, and the 1.12.37 +explanation line did not appear either -- so the question "did the repair +run?" was STILL unanswerable from the user's paste. Three sessions of +guessing from outside is the actual failure here; the repair may well be +fine. + +The failure box now states its own reasoning: + ! auto-repair: enabled, gate: recognised, rounds run: 2 + ! directories it could create: /usr/share/vim/vimfiles/indent + ! (they are still missing, so creating them did not work ...) +Whatever the next paste shows, it distinguishes the three possibilities that +have been indistinguishable: an old binary on the tree, a gate that refused, +or a creation that failed. If it says "gate: recognised, rounds run: 0", +the loop's own conditions are the bug; if "created ... " appears and the +directory is still missing, mkdir is failing and the reason must be shown. + +## The gate, not the repair (1.12.37) + +cmake failed on /usr/share/vim/vimfiles/indent AGAIN -- the exact case +1.12.15/17 exists to repair -- with no attempt and no explanation in the +failure box. The repair was fine (verified: the detector finds cmake's +path, the creator makes it). The GATE was wrong: _perm_ish is set by a +regex listing "cannot (create|remove|change|touch)", and cmake says "cannot +MAKE directory". The second condition +(`[ -n "$(missing_dirs_from_log ...)" ]`) should also have caught it, so +this needs confirming on the real tree -- see below. + +Two changes: "make" joins the regex, and a refusal now SAYS SO -- + # no automatic repair attempted: the failure does not look like + # a permission or missing-directory problem +A decision not to act is a decision, and it was invisible; two sessions were +spent guessing whether the loop had run. + +STILL OPEN: if cmake fails a third time WITH the new gate, the missing-dirs +detector is not seeing $log at that moment (wrong path? log not yet +flushed?) and the next step is to print what missing_dirs_from_log returns +right there, rather than reason about it from outside. + +## Retrying a dead mirror is not a retry (1.12.36) + + HTTP request sent, awaiting response... 502 Bad Gateway + download failed (try 1/3) -- retrying in 5s: https://ftpmirror.gnu.org/... + ... 502 ... 502 ... and libunistring failed + +Not a tool bug in the strict sense -- ftpmirror.gnu.org was down -- but +asking the same broken host three times, five seconds apart, is not a retry, +and the file was sitting on every other GNU mirror. _fetch now walks a +mirror list: the book's own URL first (always), then for GNU hosts +ftp.gnu.org, mirrors.kernel.org and mirror.csclub.uwaterloo.ca with the path +rewritten properly, and for ANY host the LFS downloads mirror by filename as +a last resort. Backoff grows (15s, 30s) instead of a flat 5. + +A guessed GNOME rewrite was written and then deleted rather than shipped: +its path shape was not verified, and an invented mirror that 404s is worse +than none. Verified: a GNU url yields 5 candidates, an unknown host yields +exactly 2 (itself and the LFS mirror), and the book's url is always tried +first. + +## One meaning for "yes", and the whole cache family (1.12.35) + +**`--yes` did not answer the group prompt.** `stack sway --run --yes` still +stopped at "Which group should share THIS DIRECTORY?". The prompt honoured +LFS_ASSUME_YES; packagemanager --yes sets PM_YES; pm-install read PM_YES. +Three spellings of one decision -- so `assume_yes()` is now the single +predicate (either variable), used by every prompt in the file, and the +automatic choice is ANNOUNCED ("granting <dir> to group <g> (--yes)") +because an answer taken on someone's behalf must still be visible. + +**desktop-file-utils:** "The databases in [/usr/share/applications] could +not be updated" -- update-desktop-database rewriting mimeinfo.cache. The +info index (1.12.20), the XML catalogs (1.12.31) and this are one question +asked three times, so the whole family goes in at once rather than one +session each: applications/mimeinfo.cache, mime/mime.cache, +glib-2.0/schemas/gschemas.compiled, icons/hicolor/icon-theme.cache. Each is +REGENERATED from what other packages installed, by whichever package runs +the tool last; none is installed by one package, so none may be owned by +one. Proven with two real package users on the desktop cache. + +## The book's order survives the split into phases (1.12.34) + +The ordering fix worked -- GLib now builds before shared-mime-info -- and +GLib itself then failed: + ERROR: Program 'g-ir-scanner' not found or not executable + +Not a dependency cycle in the plan. The BOOK resolves the GLib / +gobject-introspection cycle inside GLib's own page, and it does so by going +user -> root -> user: + build glib with -D introspection=disabled (user) + ninja -C gi-build install (root) + meson configure -D introspection=enabled && ninja (user) +extract_commands_phased sorted blocks into two streams, keeping each +stream's order and LOSING THE INTERLEAVING, so the reconfigure landed in the +build phase -- before gobject-introspection was installed. + +Now: once a root block has appeared, every later block joins the install +stream, in the book's own order. Nothing is lost by moving a command +across, because both streams run as the same package user here (a package +user installs its own files). Verified on the real book: glib's install +phase reads ninja install / tar / meson setup gi-build / ninja -C gi-build / +ninja -C gi-build install / meson configure -D introspection=enabled / +ninja / ninja install -- exactly the page. nettle, cmake and libxml2 are +unchanged. The suite generates glib2 from the shipped book and asserts the +gi install precedes the reconfigure. + +## A required edge beats a recommended one (1.12.33) + +docbook-xsl passed (1.12.31 works). shared-mime-info then died on + Dependency "glib-2.0" not found, tried pkgconfig +on a FRESH tree, so not damage -- a real ordering bug, and the same one that +was blamed on a "glib ghost" two sessions ago. + +shared-mime-info REQUIRES GLib; GLib merely RECOMMENDS shared-mime-info. +walk_deps marks a node visited on entry, so a cycle is broken by whoever was +entered first: exploring GLib's recommended list reached shared-mime-info +while GLib was still in progress, its required GLib edge was skipped as +"already visited", and it was emitted first -- + ... libxslt, shared-mime-info, GLib, desktop-file-utils + +A recommendation is a preference; a requirement is a fact. +_required_edges_win moves a required dependency in front of its dependant +after the walk, bounded, leaving all-required cycles alone (those the book +must resolve, and reordering forever would hide them). Verified on the real +book: GLib precedes shared-mime-info in the orders for desktop-file-utils, +shared-mime-info and glib2 itself. + +NOTE for the next session: this bug means ANY package caught in a +required/recommended cycle was mis-ordered, not just these three. Worth a +sweep of the stack's book targets with `blfs deps <t>` once the build +settles. + +## `make install` was breaking the system (1.12.32) + + (lfs chroot) ls -ald /usr/bin/ + drwxr-xr-x 1 root install 9886 /usr/bin/ + +775 -> 755: group-write gone, and with it every package user's ability to +install anything. The user suspected the stack cycle; it was OUR MAKEFILE. +`install -d DIR` applies its mode to a directory that ALREADY EXISTS, so +`$(INSTALL) -d $(BINDIR)` re-moded /usr/bin on every `make install` -- five +such lines (bin, stacks, completions, skel, share). Every failure in the +build that followed was a symptom of installing the tools. + +`mkdir -p` leaves an existing directory exactly as it is; all five now use +it. Reproduced (775 -> 755 with install -d, unchanged with mkdir -p) and +pinned in the suite, which runs a real `make install DESTDIR=...` over a +775 directory and checks the mode survived. + +Repair for a tree already hit: `lfs-helper verify --fix` restores the +install directories. This also explains the docbook-xsl catalog failure +repeating at 1.12.30 -- the run was made with the tools that had just +re-moded /usr/bin. + +## The XML catalog is an index too (1.12.31) + + could not open /etc/xml/catalog for saving + +docbook-xsl, on the REBUILT tree -- so not old damage, a gap. Exactly the +info-index failure one directory over: docbook-xml creates /etc/xml/catalog +as its own package user, and every later docbook package rewrites it with +xmlcatalog. 1.12.20 said the membership test for shared_files_list is "is +this file REWRITTEN by many packages rather than installed by one" -- the +catalogs qualify, so /etc/xml/catalog and /etc/xml/docbook join the list. + +And a real hole in that rule, found while testing it: share_shared_files +fixed the shared DIRECTORY's mode but not its group, so /etc/xml came out +root:root 775 -- group-writable for group ROOT, which helps no package user. +/etc/xml is not in install_dirs_list, so nothing else was going to give it +the group. It is chowned root:install now, like every other shared +directory. Proven end to end with two real package users: a catalog created +by one, in a sticky root-owned directory, is rewritable by the next. + +## Blame the owner, not whoever ran ldconfig (1.12.30) + +freetype is fixed on the user's tree; fontconfig showed the same missing +development symlink, and the box named the wrong package: + + lfs-helper build libpaper --phase all --force + (the file is owned by p_libpaper ...) + +libfontconfig.so.1 is the SONAME link, made by LDCONFIG under whatever +account happened to run it -- p_libpaper's install, in this case. The hint +now follows the link to the real library and asks the MANIFESTS who owns +that file, falling back to file ownership; it names fontconfig. + +### Recommendation recorded for the user: rebuild rather than repair + +The tree carries pre-1.12.x damage: mis-owned files, installs that never +finished (freetype, fontconfig), records that lied. The tools now prevent +each of those, but every package installed during the broken era may carry +some of it, and each one costs a round trip to find. A rebuild from the +LFS-base snapshot costs CPU time and no human attention, and everything +after it runs on tooling where these bugs cannot arise. Advised: restore the +snapshot, `make install` the current tools INTO it first, then +`packagemanager setup --run` and `packagemanager stack sway --run --yes`. +Only valid if the snapshot predates the BLFS installs. + +## The ordinary path must not need the sweep (1.12.29) + +The user asked the right question about 1.12.27/28: does a plain +`packagemanager stack sway --run` handle both problems without a manual +`verify --fix`? + + * mis-owned files: YES -- takeover_files_from_log is in cmd_build's retry + loop, which fires on "Operation not permitted". + * lying records: NO, and that was the real gap. `verify` had the majority + rule since 1.12.10, but the PLANNER used a different one: + _manifest_has_real_paths returned True on ONE surviving path. binutils + (140 of 330 alive) therefore read "installed" to every ordinary install, + and only the manual sweep disagreed. Two rules for one question, which + is this project's oldest disease. + +The planner now applies verify's rule: more than half the manifest must +still exist. Counted over the whole manifest (capped at 5000 paths) -- +sampling the first N is wrong, because pkg.lst is path-sorted and a package +whose /usr/bin survived while /usr/share died would look healthy from the +top. And when a pkg.lst exists it decides: install_last naming a version no +longer overrides a manifest that is mostly gone. + +So `packagemanager stack sway --run` alone now rebuilds a package with a +dead record instead of skipping it; `verify --fix` remains the way to clean +the record itself. + +## Deleting a lie is a repair (1.12.28) + +The lying-record sweep (1.12.10) was report-only, on the reasoning that +"rebuilding is a build decision, not a repair". Half right: rebuilding is +the user's call, but the false RECORD is not a build decision at all -- it is +wrong data, and while it stands the planner skips the package and every +dependent fails somewhere else entirely (xmllint, glib, -lfreetype: three +sessions of chasing symptoms of one lie). + +`verify --fix` now removes pkg.lst and VERSION for a record whose files are +mostly gone, says so, and names the reinstall for when the user wants it. A +bare `verify` still only reports. Pinned in the suite: bare verify deletes +nothing, --fix deletes the liar's record and keeps the honest one. + +## Ownership follows the installer (1.12.27) + +The bottom of the whole ghostscript chain, in one line: + + install: cannot remove '/usr/lib/pkgconfig/freetype2.pc': Operation not permitted + /usr/lib/pkgconfig/freetype2.pc -rw-r--r-- p_cairo:p_cairo + in /usr/lib/pkgconfig drwxrwxr-t root:install + +freetype2's OWN file, owned by cairo -- the over-claiming manifests of +1.12.11 and earlier, made real by a `verify --fix` -- in a sticky (sealed) +directory, where only the owner may replace a file. EPERM is not something +a collector group can grant away, so the retry loop had nothing to try. One +cause, four symptoms: freetype's install never finished -> libfreetype.so +missing -> ghostscript could not link -> four sessions of chasing. + +In this model the package that installs a file OWNS it (that is verify's +ownership rule), and the installer is by definition installing it now. So +takeover_files_from_log hands such files over -- loudly, naming the previous +owner -- and the retry proceeds. Bounded deliberately: only files owned by +another PACKAGE USER, never root's (root's files are not a package's +mistake), and never inside a build tree, /tmp or /sources. + +Tested with real package users: the mis-owned file is taken over, a +root-owned one is not, a build-tree file is not. (Fixture note: package +users need their OWN group -- useradd -U -- or chown p_x:p_x fails and the +test lies.) + +## libfoo.so.N is not libfoo.so (1.12.26) + +The user's tree answered the ghostscript question: + + libfreetype.so.6 -> libfreetype.so.6.20.1 exists (p_libpaper, 00:51) + libfreetype.so.6.20.1 exists (p_cairo, 00:39) + libfreetype.so MISSING + /usr/src/pkgusr/p_freetype/VERSION does not exist + +NOT a lying record: `-lfreetype` needs the DEVELOPMENT symlink libfreetype.so, +which only the package's own `make install` creates. libfreetype.so.6 is the +SONAME link, which ldconfig makes from any library it finds -- that is why it +carries a later timestamp and an unrelated owner. So freetype's install +never finished, and the odd ownership is the old over-claiming bug that +`verify --fix` then acted on. + +Two illnesses, two cures, and 1.12.25 could not tell them apart -- worse, its +presence check globbed libfoo.so* and so declared libfoo.so.6.20.1 "present", +filtering out exactly the failing case. Now it checks for libfoo.so (or .a) +EXACTLY, and the box distinguishes: + * libfoo.so.N present, libfoo.so absent -> the owner's install did not + finish; rebuild it (the owner is named from the file's ownership). + * nothing at all -> a dependency recorded as installed that is not. + +User's cure for this tree: rebuild freetype and fontconfig +(lfs-helper build freetype2 --phase all --force, likewise fontconfig), then +resume. Their records were never written, so the planner will not skip them. + +## The linker's version of the same sentence (1.12.25) + +ghostscript got past configure (1.12.24 worked) and died linking: + /usr/bin/ld: cannot find -lfontconfig + /usr/bin/ld: cannot find -lfreetype +Same species as "Program 'xmllint' not found": a dependency the planner +believes is installed did not leave its library behind. The failure box now +says so -- it extracts the -lNAME list, keeps only the ones genuinely absent +from /usr/lib, /usr/lib64, /lib, /lib64, /usr/local/lib, and prints the cure +(packagemanager search libNAME; lfs-helper verify; purge pkg.lst+VERSION; +reinstall). + +A bug found while testing the check, worth recording because it is a +classic: `ls a* b* c*` returns non-zero when ANY pattern misses, so a +library present in the first directory was reported missing because the +other two lacked it. One glob per directory now, and the suite pins both +halves (a missing library IS named, an installed one is NOT). + +Open on the user's system: freetype and fontconfig are recorded as installed +but their libraries are absent -- ghosts of the same family as glib. The +sweep (verify) plus the printed cure is the path; if freetype turns out to +be installed and only its .so symlink missing, that is a different bug and +worth its own look. + +## Four causes behind one ghostscript failure (1.12.24) + + configure: error: Mixing local libtiff with shared libjpeg not supported + make: *** No rule to make target 'so'. Stop. + install_gs: line 118: gs: command not found + mv: cannot stat '/usr/share/doc/ghostscript/10.06.0' + +**Why configure failed.** The block that prevents it -- +"If you have installed the recommended dependencies on your system, remove +the copies of freetype, lcms2, libjpeg, libpng, and openjpeg: + rm -rf freetype lcms2mt jpeg libpng openjpeg" -- was COMMENTED as +conditional. But these tools install required AND recommended dependencies +by default; that is the contract. A condition predicated on what we +guarantee is satisfied by construction, so _conditional_guard now keeps such +blocks active ("optional" still wins over both). + +**Why everything after it ran anyway.** `unpack_pkg && build_pkg || exit 1` +reads like it stops on failure and does the opposite: POSIX disables set -e +inside a function whose status is being TESTED. The phase bodies' own set -e +(1.12.8) was switched off by the CALL. The dispatcher now runs `set -e` with +sequential calls, and the PM_TEST condition is an if, not an and-list (which +would abort the run when false). + +**Two things that only became dangerous once failures stop the run.** The +book demonstrates results with the package's own not-yet-installed program +("gs -q -dBATCH .../tiger.eps"); and it offers test commands in the build +section. Chasing the wording ("To do this...") is the mistake this file +keeps making, so the rule is structural: a build line whose first word is a +program THIS package installs is commented out. Test blocks are matched by +_TEST_BLOCK_RE as a smaller, separate rule. + +Known cosmetic gap: `make so` stays active in the build while its install +("If you built the shared library...") is skipped -- harmless (libgs.so is +only wanted by asymptote/ImageMagick) but inconsistent. + +## The tool died on its own source (1.12.23) + + # libjpeg: done + Installed. Refreshing mandb in the background... + /usr/bin/lfs-helper: line 8312: mmand: command not found + /usr/bin/lfs-helper: line 8314: syntax error near unexpected token `)' + +Line 8312 is a COMMENT containing the words "a command", and bash read +"mmand". Nothing was wrong with libjpeg -- it had built and installed. The +shipped file parses cleanly; the one on disk was REPLACED while bash was +executing it, and bash reads scripts incrementally by byte offset, so +execution resumed mid-word in the new content. (A `make install` of a new +build in another terminal will do it; so would any tool refresh.) + +Everything in lfs-helper is now one group command -- `{` after the header, +`}` at the end -- so bash parses the WHOLE file before executing any of it; +after that the file on disk may change freely. Functions and variables in a +group are global, exit still exits, the status is the last command's. +Verified working wrapped, and the suite pins the shape (first non-comment +line is `{`, last line closes it, and the tool still runs). + +blfs/packagemanager/lfs are Python, which reads the file whole -- immune by +construction. lfs-sanity.sh is bash but short-running; worth the same +treatment if it ever grows. + +## The same failure, one layer earlier (1.12.22) + +The user asked whether the LFS BASE build hits this too. It does, and +1.12.21 left the gap: /usr/share/info/dir usually does not exist when +init-pkgusr runs -- it is created mid-build by the first package that ships +an info page, owned by THAT package user -- so the second such package +fails, is repaired by the retry loop, and the user watches a failure that +never needed to happen. Chapter 8 hits this on every fresh system. + +cmd_build now shares the indexes BEFORE the build (pkgusr_ready && +share_shared_files, idempotent and cheap), so the file is root:install 664 +before anyone needs to replace it. The auto-fix path stays as the net for a +tree that predates this. + +Proven with real accounts: package A creates the index (p_a:install 644), +package B cannot replace it, sharing takes it into root:install 664, package +B can. That is the LFS chapter-8 sequence exactly. + +## Sealing and sharing were contradictory rules (1.12.21) + +1.12.20 made /usr/share/info/dir root:install 664 and nettle failed at the +same line. The message said which half was missing all along: **Operation +not permitted**, EPERM, not EACCES. install-info does not write the index +in place -- it writes dir.new and RENAMES it over the old file, and in a +STICKY directory only a file's owner may replace it. /usr/share/info is an +install directory, the tree is sealed, so no mode on the file could ever +have helped. + +Two of this project's own rules were in direct contradiction: sealing +(sticky, so package users cannot delete each other's files) and sharing +(every package must replace this file). A directory that holds a shared +index is now excluded from sealing -- in share_shared_files (which clears +the bit: g+w,o-t), in cmd_seal_install_dirs (which skips it and says so), +and in _vfy_install_dirs (which wants 775 there even on a sealed tree, so +the next verify does not put it back). + +Reproduced in the suite before fixing: a real package-user account, a +sticky root-owned index, the rename fails with exactly EPERM; after the +rule it succeeds. That test is the one that would have caught 1.12.20 +being half a fix. + +## A shared index is shared, not owned (1.12.20) + + install-info: Operation not permitted for /usr/share/info/dir + +nettle died there AFTER installing its libraries. /usr/share/info/dir is +the info INDEX: every package that ships an info page rewrites it. It has +been in never_claim_list since early on -- nobody OWNS it, correctly -- but +nothing ever made it WRITABLE by the package users who must update it. Half +a rule: we knew whose it wasn't, and never said whose it was. + +A shared DIRECTORY is root:install and group-writable. A shared FILE is the +same rule one level down: shared_files_list + share_shared_files (root:install +664, idempotent), applied in the three places that matter -- init-pkgusr when +the system is set up, verify --fix on an existing tree, and cmd_build's +auto-fix loop, so a build that hits it repairs and retries instead of dying. + +The list is deliberately short (just the info index today). Anything else +that turns out to be a shared, rewritten-by-everyone file belongs there too +-- and that is the test to apply, not "which package installed it". + +## What runs decides, twice over (1.12.19) + +alsa-lib failed two ways at once, and both were label-reading: + + tar: ../alsa-ucm-conf-1.2.15.3.tar.bz2: Cannot open + install: cannot stat 'doc/doxygen/html/*.*' + +**Downloads follow the commands.** 1.12.8 kept additional downloads whose +label said "required", plus every .patch. alsa-ucm-conf is labelled +"Recommended file:" and is used by a command that is NOT conditional: + make install && tar -C /usr/share/alsa ... -xf ../alsa-ucm-conf-... +Reading the label is guessing; reading the commands is knowing. The +extractor now collects every listed download, and _needed_additional_links +keeps the ones an ACTIVE (uncommented) phased command names -- patches +always. Verified: alsa-lib keeps its tarball, docbook-xsl only its patch, +libxml2 and libuv keep nothing (their extras are commented). Note the trap +found on the way: data["install_cmds"] is the RAW page, where conditional +blocks are still live and vote for their own downloads -- only the phased +text may be consulted. + +**An artifact that was never built is not a failed install.** The book +installs optional extras the package's own `make install` does not produce +(alsa's doxygen html, libuv's sphinx man page). Without the optional tool +the sources do not exist, and install killed a package that had built and +installed correctly. The install WRAPPER now skips when EVERY source +argument is missing, loudly ("an optional artifact that was never built -- +not a failure"); a partial set still goes to the real install, so a +genuinely half-built package is not hidden. One place, every package -- +and it retro-fixes the libuv class too. + +## Book prose is data; our shell options are ours (1.12.18) + + install_texlive-20250308-source: line 25: TEXLIVE_PREFIX: unbound variable + +Line 25 is metadata lifted from the book's Contents section -- +"Installed Directories: $TEXLIVE_PREFIX/bin, $TEXLIVE_PREFIX/lib" -- emitted +inside DOUBLE quotes, so the shell expanded it. Two independent causes, both +fixed: + +1. The generator now emits every book string through bash_str(): single + quoted, with the one escape single quoting needs. A dollar sign in + documentation is not a variable, and a backtick in it is not a command. + (bash_array uses it too.) SCRIPT_VERSION bumped to 5. +2. pm-install sources the script only to read that metadata, and did so + under THIS shell's set -u. Our strictness governing their file is a + category error, and would keep producing failures that are ours, not + theirs: it now saves $-, does set +u +e around the source, and restores. + +Also answered for the user: the ~195 "regenerating" lines are the +generator-version check (1.12.16 -> 1.12.17) doing its job once, not lost +work; the plan is re-resolved per stack ENTRY, and a book entry with 195 +pending dependencies re-plans them until they are installed. Possible +improvement if it grates: cache the resolved plan per entry and offer to +reuse it (the plan cache exists -- "plan cache v1" in --version -- but the +per-entry resolve still runs). + +## The pattern, not the wording (1.12.17) + +libpaper died writing /etc/profile.d/libpaper.sh -- the directory does not +exist, exactly the class 1.12.15 was supposed to have fixed. It did not +fire because 1.12.15 matched MESSAGE WORDINGS (cmake's `cannot make +directory "X"`, coreutils' `cannot create directory 'X'`) and bash words a +failed redirection differently: + install_libpaper: line 150: /etc/profile.d/libpaper.sh: No such file or directory +The same mistake as the verb list in the book guard, made by the same +author two releases later. Every tool words it differently; the pattern is +"No such file or directory" plus an absolute path. + +missing_dirs_from_log now works from that pattern, with two classes and one +rule each: a directory the tool tried to CREATE -> make that path; anything +else that could not be written -> make its PARENT. Both only when the +directory chain really is what is missing (if the parent exists, the error +is about a file, and inventing a directory would turn a clear message into +a mystery), and never inside $SRCROOT, /sources, /tmp or a build tree -- +a path that failed there is the build's own problem. + +Verified on five real line shapes at once: cmake's directory, bash's +redirection, a missing source file in a build tree, a missing file whose +directory exists, a relative path. Only the first two are acted on. + +## A compile error is not ours, but the cure has a home (1.12.16) + +cairo 1.18.4: + error: static declaration of 'ctime_r' follows non-static declaration +Cairo's HAVE_CTIME_R check came out false, so cairo-ps-surface.c defined its +own static ctime_r and collided with glibc's. The book carries no patch -- +this is the package's source against a newer toolchain, and the tools cannot +and should not repair a package's C. + +What IS ours: making sure the cure survives. A hand-edit of the generated +script is lost on --regenerate; machine.conf is the place that persists and +is per-package. Shipped: + [cairo] + c_args = -DHAVE_CTIME_R +merged into cairo's `meson setup` line by apply_machine_opts (verified). +And the failure box now says so: on a real compile error +(file:line:col: error:) it states that this is the package's own source, +not a permission or layout problem, and prints the machine.conf shape for a +per-package build flag. + +Note the honest boundary here (ALWAYS TREAT THE CAUSE): the cause is +upstream cairo, which we cannot fix; what we fixed is that the workaround +had nowhere durable to live and the failure said nothing about it. + +## A directory that is not there cannot be granted (1.12.15) + +cmake installs into /usr/share/vim/vimfiles/indent -- vim's tree, a +subdirectory that has never existed: + + file INSTALL cannot make directory ".../vimfiles/indent": + No such file or directory + +Not a permission problem, so the grant machinery never fired (and could not +have helped): the directory is not unwritable, it is ABSENT, and a package +user may not create it inside another package's tree. The model has always +had the answer for a directory two packages share -- root:install, +group-writable -- but nothing applied it ON DEMAND, so every package that +installs into a new subdirectory of another package's tree hit this. + +cmd_build's auto-fix loop now creates them: missing_dirs_from_log reads +DESTINATION directories out of the log (cmake's `cannot make directory "X"` +and coreutils' `cannot create directory 'X'`), create_missing_dirs_from_log +makes them as root with set_install_dir_owner + g+w, and the loop tries +creating before granting, since a round often needs both. Deliberately NOT +matched: `install: cannot stat 'X'`, which names a SOURCE the build never +produced (libuv's man page) -- creating a directory there would turn a clear +error into a mystery. + +## A condition is a condition (1.12.14) + +libuv died on `install: cannot stat docs/build/man/libuv.1`. The 1.12.6 +guard had caught the block that BUILDS the man page ("If you installed the +optional sphinx-9.1.0 python module, create the man page:") and missed the +one that INSTALLS it ("If you built the man page, install it as the root +user:") -- because the guard enumerated verbs (wish|want|downloaded|have| +installed|...) and "built" was not among them. So the page was never made +and the install of it ran anyway. + +Enumerating verbs is guessing at English; the pattern is the conditional +opening. _COND_GUARD_RE is now ^(if\b|optionally\b|should you\b) -- any +sentence that opens with a condition guards its block. Skipping stays safe +either way: the block is emitted commented with the book's own sentence +above it, one uncomment away. Verified on the user's book: libuv's man +build AND its install are both commented, `make install` stays active, and +libxml2's mandatory root sed on xml2-config is untouched. CACHE_VERSION +bumped to 6. + +(Same lesson as the h3/h4 fix and the settle fix: the list of cases is not +the rule. See ALWAYS TREAT THE CAUSE at the top.) + +## The cause of the over-claiming manifests (1.12.13) + +The user asked what actually caused the 711 wrong-owner claims, and whether +it was one package or a leak. It is a leak, and it is structural. + +cmd_build decides what a package installed BY TIMESTAMP: drop a stamp, run +the build, claim every file newer than the stamp. That rests on one +assumption -- nothing else writes to the tree during a build -- and the +user's run disproved it. make-ca (package 98) was still generating +/etc/ssl/certs and /etc/pki/anchors at 19:45, after its own scan had closed, +while libogg (99) was building: libogg's manifest claimed ~400 +certificates. The files themselves were fine (root:root); the MANIFEST was +wrong, which is why --fix was dangerous. Same mechanism gave p_docutils our +own `make install` files and p_wget /tools. Any writer active during a +build window is attributed to whoever is building -- and this toolchain +supplies such writers itself, detaching mandb and record-install with setsid +precisely so they outlive the install. + +Fix at the cause: the attribution window does not open until the tree is +quiet. _settle_tree_before_stamp waits (bounded, LFS_SETTLE_SECONDS, +default 60) for a NAMED list of writers -- mandb, make-ca, +update-mime-database, gtk-update-icon-cache, gdk-pixbuf-query-loaders, +glib-compile-schemas, fc-cache, p11-kit, trust, list_package -- then stamps. +Named rather than "any activity", which would hang on whatever the user is +doing on purpose; and bounded, so a stuck mandb warns that the manifest may +over-claim instead of stopping the build. + +The 1.12.11 path shields stay: they also cover files written outside any +build window at all. But they are no longer the fix. + +## Sealing is one-way (1.12.12) + +The user asked the right question about the verify log: what CAUSED the +wrong install-dir permissions? Answer: nothing on the tree -- the ~50 +"installdir ... 1775 -> 775" lines were verify proposing to STRIP correct +sticky bits. _install_dirs_are_sealed demanded that all five probed +directories be sticky; the user's tree is partly sealed (sealing +interrupted, or install dirs created after the seal), so it read as "not +sealed at all", and the unsealed-phase rule (775) was then applied to +directories that had been deliberately sealed. A `verify --fix` would have +unsealed half the system. + +Now: ANY sticky install directory means sealed (probe widened to 12, mixed +state warned about). Sealing is a deliberate one-way end-of-build act, so +verify COMPLETES it on stragglers and never takes it back. A never-sealed +mid-build tree has no sticky install dirs and still reads unsealed, which +is what the build needs. + +**Still NOT fixed at the root: the 711 wrong-owner claims (1.12.11).** The +cause is structural -- cmd_build attributes every file newer than its stamp +to the building package, and detached background work (make-ca's cert +regeneration, `make install` of these tools, mandb) runs during someone +else's build. 1.12.11 shielded the paths this actually hit; the general +cure is to stop guessing by timestamp (per-phase DESTDIR staging already +exists for the install phase -- extending it, or ignoring files whose mtime +falls inside a known background task's window, is the next real fix). + +## What the sweep found (1.12.11) + +The first real sweep on the user's system answered both open questions. + +**glib was never the tool's fault.** `blfs order desktop-file-utils` puts +GLib-2.86.4 in the plan, correctly (an earlier `head -8` of mine truncated +it, which is how it looked missing). So glib is judged installed while +pkg-config cannot find it: another ghost, and the planner therefore starts +at desktop-file-utils. The sweep names p_binutils a liar (330 claimed, 190 +missing, 57%) but says nothing about glib -- so glib's "installed" verdict +comes from a validate/library check rather than a pkg.lst, or its record is +just under the 50%% threshold. Next session: teach the sweep to also judge +a package the PLANNER calls installed whose account/record does not exist, +and consider a `packagemanager why-installed <pkg>` that prints which +signal decided it. + +**711 wrong-owner claims, one species.** p_libogg claiming every +/etc/ssl/certs file, p_make-ca the /etc/pki anchors, p_docutils +/etc/pkgusr/stacks and /usr/share/lfs-pkgusr, p_wget /tools. None of them +installed those: make-ca's detached background regeneration and `make +install` of these tools ran DURING someone else's build, and the timestamp +scan sweeps whatever is newer than the stamp. `verify --fix` would have +chowned the system certificate store to libogg. never_claim_list now +covers /etc/ssl/certs, /etc/pki, /etc/pkgusr, /usr/share/lfs-pkgusr and +/tools -- regenerated-on-demand or ours, same class as +/usr/share/info/dir. **The user must NOT run `verify --fix` on a +pre-1.12.11 helper.** + +## The sweep for every lie at once (1.12.10) + +desktop-file-utils: `Dependency "glib-2.0" not found` -- while +shared-mime-info, which requires the same glib, had just built. The book's +dep extraction is CORRECT (verified against the uploaded book: both require +GLib-2.86.4, ordered first), so either glib is recorded-but-absent (another +ghost) or something environmental. Rather than guess one crash at a time, +the deferred detector is built: verify rule 3c, _vfy_lying_records, reads +every pkg.lst under $PKGUSR_ROOT and counts how many claimed paths exist; a +record with >=50%% of its files missing is named a LIAR with the exact cure +(rm pkg.lst VERSION; packagemanager install <pkg> --run). Report-only -- +rebuilding is a build decision, not a repair. Wired into cmd_verify after +the unclaimed pass. + +Diagnostics handed to the user for the glib mystery (their output decides +the next fix): + lfs-helper verify # the sweep names every ghost + pkg-config --exists glib-2.0; echo $? + ls -la /usr/lib/pkgconfig/glib-2.0.pc + cat /usr/src/pkgusr/p_glib2/VERSION 2>/dev/null + +## The lie on disk, and the box that names it (1.12.9) + +shared-mime-info: "Program 'xmllint' not found". xmllint is libxml2's -- +and libxml2 was NEVER really rebuilt: the 1.12.7 recorder gate stops new +lying records, but the pre-fix pkg.lst still sits in p_libxml2's home and +the planner believes records. Worse, docbook "succeeded" under a +pre-set-e script whose xmlcatalog failures were mid-script and ignored, so +its catalog registration is silently missing too. + +Tree remedy (given to the user): + rm -f /usr/src/pkgusr/p_libxml2/pkg.lst /usr/src/pkgusr/p_libxml2/VERSION + packagemanager install libxml2 --run --yes + packagemanager script install DocBook # redo the skipped catalog steps + packagemanager stack sway --run --yes + +Tool side: the failure box now recognises the pattern. On "Program 'x' +not found" where x really is absent from the system, it says a dependency's +record probably LIES (pre-1.12.7 failed build), and prints the cure: +search for the owning package, purge its pkg.lst/VERSION, reinstall. + +Open question for a later session: a one-shot detector for remaining +pre-1.12.7 lying records (a verify pass comparing each record's claimed +programs against the filesystem) -- the failure-box hint covers the +discovery path for now. + +## Required patches exist, phases fail loudly, caches notice (1.12.8) + +docbook-xsl, third stack failure, three causes in one crash: + +**additional_links was empty for EVERY package in the 13.0 book.** The +extractor looked for an <h3>Additional Downloads</h3>; this book edition +uses <h4>. The tag is layout, the title is the contract -- it matches h3 OR +h4 now, by get_text(). And the extractor collects only REQUIRED entries +plus every .patch; optional doc/test tarballs stay uncollected, matching +their commands being emitted commented. CACHE_VERSION bumped to 5 so the +parsed index rebuilds (the suite's guard now checks >= instead of an exact +number that itself went stale on the bump). + +**A failed patch was shrugged off for 1900 lines.** Command blocks joined +without set -e meant `patch: Can't open ...` at line 2 and a "successful" +unpatched build until the optional cp finally tripped. Every generated +phase body opens with set -e now -- bodies do not execute on `source`, so +pm-install's sourcing of the script is untouched. + +**The /tmp script cache did not notice a moved GENERATOR.** "version-keyed += safe" caught a moved book, not a fixed blfs: the user's run reused a +pre-1.12.6 docbook-xsl script whose optional blocks were still active. +resolve_script now compares the cached script's "### generated by blfs X" +header against the live generator's version and regenerates on mismatch. +(Local scripts in package homes stay sacred -- user edits win; a stale +pre-fix HOME copy still needs a manual rm, as with libxml2.) + +User remedy for the current failure: + rm /usr/src/pkgusr/p_docbook-xsl/install_docbook-xsl-nons-1.79.2 + # stage the patch or let _fetch get it, then resume the stack + +## A failed build makes no claim (1.12.7) + +Second --run: docbook died on `xmlcatalog: command not found` -- a program +libxml2 installs, in a plan the book orders correctly (libxml2 first). The +planner had SKIPPED libxml2 as installed, because the previous FAILED +libxml2 run had still called cmd_record_install: pkg.lst with 4 half-built +files, and pkg.lst is what the planner believes. The very next comment in +cmd_build says "A FAILED build must not take ownership of anything" -- the +recorder now obeys the same rule: gated on rc==0. The manifest (what a +step TOUCHED) rightly survives failure; the claim of installation does not. +User remedy on the tree: `lfs-helper build libxml2 --phase all --force` +rebuilds it for real (the planner cannot, it still believes the lie until +the record is rewritten by a successful build). + +The failure box printed p_docbook and p_DocBook in one breath: python's +pkgusr_name did not lowercase, the helper's pkg_owner_name does. Python +now lowercases first, same as the helper -- one casing rule for every name. + +And the user's ask: `stack --run` now CACHES completed entries per stack +file ($LFS_PKGUSR_DIR/progress/stacks/<n>.done, "kind target" lines). A +resume skips them as "done (stack cache)" instead of re-planning a +200-package tree for a stage already finished; --force ignores the cache. + +## The book's conditionals stay conditional (1.12.6) + +First --run: 217-package plan, died at 2/217 in libxml2 -- `ERROR: Program +'doxygen' not found`, plus a missing xmlts test tarball. The user called +it: "in the script we include the docs [but] we don't install the +requirements for it." The book agrees -- both blocks sit under prose +conditionals ("If you wish to build and install the manual pages...", "If +you downloaded the test suite...") that the extractor ignored, taking every +<pre> unconditionally. + +extract_commands_phased now reads the paragraph INTRODUCING each command +block (_conditional_guard: "If you wish/want/would like/downloaded/have/ +prefer/need/did/are going to/installed", "Optionally", "Should you", "If, +instead"). A conditional block is emitted COMMENTED with the book's own +sentence above it -- visible, one uncomment away. Proven against the +user's real 13.0-systemd book (uploaded this session): libxml2's meson +setup and the mandatory xml2-config sed stay active; docs=enabled and the +xmlts tar are commented. Class fix: applies to all 217 pending scripts. + +RESUME TRAP: a failed pre-fix run leaves the OLD script in the package's +home, and resolve_script deliberately prefers local copies. The user must +`rm /usr/src/pkgusr/p_libxml2/install_libxml2-2.15.1` (and likewise for any +other pre-fix leftover) before re-running the stack. + +The suite's 4 remaining bare-checkout fails need the LFS book (not BLFS) -- +still environment-gated. + +## A book entry can carry its own fallback (1.12.5) + +Per the user: elogind must build even where the book lacks it. Generalised +rather than special-cased: a `book` entry carrying url= (plus the usual git +opts) FALLS BACK to a git entry when the dry run judges it NOT IN BOOK -- +it no longer trips the refusal, and shows "not in book -> git fallback". +One stack file now serves a system whose book flavour has the package and +one whose book does not. elogind carries url=github.com/elogind/elogind +ref=v255.17 prog=loginctl with a meson build (-Dcgroup-controller=elogind). +The 1.12.4 flavour fix still matters -- the BOOK build is the better one +(BLFS's configure choices + dependency resolution); the fallback is the +safety net, not the plan. + +## The book flavour follows the system (1.12.4) + +The second judged dry run on the real machine came back with every anchor +green except one: elogind, still NOT IN BOOK. The root cause was a +constant: `DEFAULT_SELECTOR = "stable-systemd"` -- on a SysV system, blfs +defaulted to the wrong flavour unless told otherwise, which is exactly how +the systemd book got onto the 13.0-sysv chroot in the first place (the old +open item, closed here). The default is now decided by the running init +(/run/systemd/system exists -> systemd, else sysv); an explicit +`blfs set-default` always wins. + +For the user's machine: `blfs fetch stable-sysv && blfs set-default +stable-sysv` (or just delete the stored default and let the new rule +decide), then the dry run judges elogind and --run is unblocked. Closing +the old open item: "BLFS must not pick a systemd-flavour default on that +system" -- it no longer can. + +## The machine's options, and the anchors trued up (1.12.3) + +The first judged dry run on the real system named seven wrong anchors. +Fixed from the machine's own `packagemanager search` output: the gstreamer +family is gstreamer10 / gst10-plugins-{base,good} (bad ADDED -- the user +needs it), mpv joins the stack, dejavu-fonts became a `tar` entry (new stack +kind, tarball through the same template; needs url= and pkg=), and +sdl3-image/-ttf became pinned git entries (SDL_image release-3.2.4, +SDL_ttf release-3.2.2, cmake builds, have=/usr/lib/libSDL3_*.so). + +**elogind NOT IN BOOK is a flavour smell, not a missing page**: the system's +default book is BLFS-BOOK-13.0-systemd-nochunks (see the 1.11.56 verify +log), and elogind only exists in the SysV edition. The user must switch the +default book to the plain 13.0 edition BEFORE --run, or every generated +script leans systemd. This is the old open item about BLFS flavour defaults +on the SysV system, now with teeth. + +**Machine option overrides are wired.** /etc/pkgusr/stacks/machine.conf +(PM_MACHINE_CONF overrides the path; ships with the user's mesa flags: +radeonsi, amd, x11+wayland, video-codecs=all, glvnd/libunwind disabled, +llvm on). apply_machine_opts merges each [anchor] key into the generated +script's `meson setup` line as -Dkey=value -- existing keys replaced, new +ones appended -- at the ONE place scripts are born (resolve_script's +generation return). The change makes the script a local edit, which +resolve_script then deliberately keeps across runs. meson only; cmake +merging (llvm options) pending if ever wanted. + +## The dry run asks the book (1.12.2) + +The user read the first real dry run and said the quiet part: "sway and +wlroots are also git repos" -- BLFS carries neither, and the dry run had +printed "book" for every book entry WITHOUT LOOKING. A plan half-judged +would have discovered that one llvm build too late. + +Now the dry run asks: each book entry is looked up via book_version, with +one availability probe first (book_version cannot tell "anchor absent" from +"no book on disk", so without a book every entry reads "unjudged" instead of +a false NOT IN BOOK). Unknown targets warn on a dry run and REFUSE a --run. + +The stack file grew what the book lacks, as pinned git entries with real +build commands -- which forced the parser onto shlex (build= carries +spaces; values are quoted now) and added have=<path> as the doneness signal +for libraries that put nothing on PATH (prog=- disables the PATH check): +seatd 0.9.1 (libseat-logind=elogind), wlroots 0.19.2, sway 1.11, swaybg +v1.2.1, and a terminal the stack had silently omitted -- foot 1.24.1 with +tllist 1.1.0 and fcft 3.3.2. 37 entries now. Pins are best-guess release +tags: a wrong one fails the clone, stops the stack, and is a one-line fix. + +Still pending: machine option overrides (mesa flags), and the book-anchor +names (sdl3*, dejavu-fonts, xwayland, gtk4...) get trued up by the first +dry run on the real machine now that it judges. + +## One command, one environment: `packagemanager stack` (1.12.1) + +The next goal is a working Sway desktop with the user's own WebKitGTK-6.0 +browser (github.com/nimbin2/Browser -- GTK4, so the whole stack is GTK4) and +the Sw* family (SwBr: wayland; SWOv/SwAS: SDL3). Decisions locked with the +user: LLVM yes, Mesa radeonsi+amd (options must become per-machine), Xwayland +yes, ALSA as the floor with pipewire started on demand. + +`packagemanager stack <name> [--run]` replays a declarative plan file, top to +bottom, through the machinery the fresh build proved: `book` entries via the +ordinary install planner (BLFS resolves deps, installed is skipped), `git` +entries via an auto-generated template (family convention `make all && make +PREFIX=/usr install`, doneness = prog= on PATH, --force rebuilds), `check`/ +`cfg` entries as root scripts beside the stack file, run through run-script +--as-root; a failing check STOPS the stack. Dry run by default; malformed +plans are refused with a line number. Stack files are found at the given +path, /etc/pkgusr/stacks/, or stacks/ beside the tool; `make install` ships +stacks/. + +Shipped: stacks/sway.stack (33 entries, kernel check -> seat/session -> +llvm/mesa -> wayland/sway -> gtk4/webkitgtk -> alsa+pipewire -> sdl3 -> the +four nimbin2 repos -> session wiring), stacks/kernel-sway.sh (verifies DRM, +evdev, snd, seccomp, user namespaces, tmpfs, a render node -- builds +nothing), stacks/sway-session.sh (writes /usr/bin/sway-session = +dbus-run-session sway, /usr/bin/pipewire-start for on-demand audio, and a +sway config skeleton for $LFS_HOME_USER with the Sw* binds). + +**Pending for the stack (next session):** +- Machine option overrides (opts=@mesa -> /etc/pkgusr/stacks/machine.conf + [mesa] section merged into the generated script's meson line). The user's + flags: gallium-drivers=radeonsi vulkan-drivers=amd platforms=x11,wayland + video-codecs=all glvnd=disabled libunwind=disabled llvm=enabled. +- Book target names in sway.stack are UNVERIFIED against the real 13.0 book + (sdl3*, dejavu-fonts, xwayland anchors may differ) -- the first dry run + + install on the machine trues them up. +- sway-session.sh appends Sw* binds only when creating a FRESH config. + +## Four tools (1.12.0) -- plan step 7, the endpoint + +packagemanager_install is DELETED. No shim, no fallback, at the user's +explicit request -- and both `make install` and lfs's chroot tool sync now +SWEEP a previously installed copy off the system, because a stale fifth tool +on any PATH is old code waiting to run. + +What survived it lives in `lfs-helper pm-install <account> <script>`: the +front-matter transplanted near-verbatim (the exact bash that carried the +fresh build), behind one command. Name resolution through pkg_owner_name, +the confirm prompt (PM_YES), account creation via the hint's add_package_user +plus the in-process home repair and user sort, tarball staging from /sources +(_pm_stage_source), collector-group membership (_pm_check_groups), the +install_last/diff/script records, and the two special installers -- +_pm_install_collector_dirs for nimgnu_ accounts, _pm_install_app_user for u_ +accounts (pam su rule, shared dir, launcher; the desktop user is +LFS_HOME_USER, default n76310, no longer hardcoded). Then the script goes to +the one loop: cmd_build --script for phased scripts, run_phase_as + +record-install for a rare old-style one. packagemanager's find_pm_install +returns `lfs-helper pm-install`; PM_YES/PM_MODE/PM_NO_AUTO_FIX still work. + +Deliberately dropped: the installBaseScript/vim editing flow, and the +engine's private runner/retry/wrapper/recorder block. + +The suite lost 22 counted results and gained the replacements: tests of the +deleted legacy loop are gone; tests of behavior that MOVED (staging order, +home settlement, name resolution, one recorder, one runner) now point at +pm-install; new checks assert the tool STAYS retired (its presence in tree +or chroot is now the bug) and an end-to-end pm-install run installs, records +under the step name, and keeps install_last. + +## The script owns the source search (1.11.58) + +First real run of the one loop, on a fresh 13.0-sysv bootstrap: wget's own +install tried to DOWNLOAD wget. + + staged /sources/wget-1.25.0.tar.gz -> /usr/src/pkgusr/p_wget + ... + install_wget: line 33: wget: command not found + md5sum: wget-1.25.0.tar.gz: No such file or directory + +Generated scripts open with `BUILD_ROOT="${BUILD_ROOT:-$PWD}"` and looked for +the tarball ONLY in that directory. Under the old engine nothing set +BUILD_ROOT and `su -` started in $HOME -- so "look in ." found the file +packagemanager had staged there, by accident. The one loop exports a real +BUILD_ROOT, and the accident stopped happening. + +The fix is at the chokepoint: the SCRIPT owns the decision of where a source +might already be, so it no longer depends on who runs it or from where. +Before any download it now searches $HOME/<tarball> then /sources/<tarball> +and copies the file in; additional_links get the same rule, and are no +longer re-fetched when already present. Scripts are regenerated from the +book on every packagemanager run, so `make install` of the new blfs repairs +the live flow immediately -- rerun the bootstrap, it skips what is done. + +## One loop (1.11.57) -- plan steps 5 and 6 + +The endpoint item 6 named: `packagemanager_install` no longer runs a build. +Its installPkg keeps the front-matter -- name resolution, confirmation, +account creation, tarball staging, group checks, the script/install_last +records -- and then hands the prepared script to the one loop: + + lfs-helper build <account> --script <home>/install_<account> --phase <P> --force + +cmd_build grew three things to take the hand-off. `--script` builds from a +file that is not in $SCRIPTS. The name argument is normalised through +unprefix_pkg_user, because build-all passes step names and packagemanager +passes accounts, and recording p_wget's build under "p_wget" while lfs +records "wget" is two keys for one package. And `--phase update` is a full +run: stale source cleaned, VERSION/pkgusr-info/mark_done written, no false +"earlier phases never completed" hint -- while staging still never fires for +it (_phase_can_stage allows only a lone install phase, which is also why +step 6 is now true by construction). + +The engine's wrappers/runner/retry/failure-box/recorder block survives only +for nimgnu_/u_ accounts, old-style scripts and systems without lfs-helper. +Step 7 (retiring the file) waits until a real build has proven this. + +Two report fixes from the husk repair's first real run: verify's summary +counted nothing ("repaired 0 path(s)" after merging four husks -- the +`fixed` variable was declared and never fed; _repair_misplaced_home feeds +_vfy_n_fixed now and the summary adds it), and "some files differ ... +compare and remove by hand" now NAMES up to five of them, because a report +that hands work over without saying where sends the user diffing eleven +directory pairs. + +## Root's shell history is not sqlite's (1.11.56) + +The first verify on the repaired 13.0-sysv chroot reported, alongside the +expected strays: + + [62/221] sqlite /root/.bash_history root -> p_sqlite + +The timestamp scan sweeps every new file into the open manifest, and root was +in a shell while sqlite built -- so --fix would have handed root's shell +history to a package user. Same species as /etc/group- ("ncurses did not +install /etc/group-; useradd did, while ncurses was building"). + +Three guards: /root joins never_claim_list (which also protects manifests +ALREADY polluted -- no manifest surgery needed), and both snapshot scans now +skip /root and /home so no new manifest picks such files up. + +The same run reported six files claimed by nobody. Five are the tools' own +stores -- /usr/share/lfs and /usr/share/blfs hold downloaded books, parse +caches and stamps, runtime data like $STATE, now never-claimed. The sixth, +the bash completion file, IS installed by `make install`, so +_adopt_pkgusr_tools claims the completions dir alongside /usr/bin and +/usr/sbin, and `verify --fix` refreshes that manifest before the unclaimed +pass so the report heals in one run. + +## A home that never met its files, and the runner owns the phases (1.11.55) + +Two changes, one found on the user's real 13.0-sysv chroot. + +**The husk homes.** `ls /usr/src` showed `p_wget`, `p_requests`, `p_make-ca` +and seven more sitting beside the roots. The hint's add_package_user takes no +home argument and creates `/usr/src/<account>`; every tool computes +`/usr/src/pkgusr/<account>`. Three tools each held part of the repair, and +the python side's part was `usermod -d` alone -- it repointed the passwd entry +and LEFT THE DIRECTORY BEHIND. Same species as ever: one decision (where a +home is, and how to move it there) held in several places. + +Now there is ONE implementation, `lfs-helper fix-home <name> [--run]`: +repoints the passwd entry (usermod, or the file itself pre-Shadow), then +merges the stray into the real home -- moves what is missing, deletes what is +byte-identical, KEEPS anything that differs and says so. packagemanager and +packagemanager_install both call it (each keeps a reduced local fallback for a +system without lfs-helper), and `verify` grew rule 6b: every package-prefixed +account is checked for a misplaced passwd home and for a husk at +`$SRCROOT/<account>`, repaired under --fix by the same function. + +**Phase tracking moved into the runner (plan step 4).** It was cmd_build's +private bookkeeping -- clear on "all", record on success, infer finished +phases from a failed log -- so an install driven by packagemanager left no +record and could not be resumed. run_phase_as now takes the record key and +does all three, for every caller; cmd_build's copies are deleted, and +run-script derives the key from --user through pkg_owner_name + +unprefix_pkg_user, so both drivers record under the same step name. Gated on +the script defining `install_pkg()`: an old-style script runs everything +whatever argument it gets, so recording a phase for one would claim what did +not happen. + +## Merged: the SysV-book session (1.11.54) + +A parallel session branched from **1.11.26** to solve a real problem: LFS +13.0 dropped the SysV edition. It produced a new tool and four fixes. Since +that branch never saw 1.11.27-1.11.53, everything was PORTED, not +overwritten — the branch's `lfs`, `lfs-helper` and `packagemanager_install` +were older than the ones here. + +**New tool: `lfs-sysvbook` 1.0.0** (own version line; it converts books, it +is not part of the build engine). It grafts the SysV flavour of the last +SysV book (12.4, donor) onto a systemd-only release (target) and stores the +result as a normal book, with wget-list and md5sums generated offline: + + lfs fetch 12.4 && lfs fetch 13.0-systemd + lfs-sysvbook make # -> 13.0-sysv + lfs-sysvbook check --book 13.0-sysv # 12 checks, fails loudly + lfs --book 13.0-sysv run + +Rules are declarative tables at the top of that file, so a later release is +`make --target 13.1-systemd`. Transplanted sections stay pinned at donor +versions; everything else builds at target versions. Verified there: a full +13.0-sysv build reached chapter 8 and BLFS bootstrap. **Booting the finished +13.0-sysv system is still untested.** + +Ported into the current tools: + +- **`lfs`**: the infix tarball pattern (SQLite-3510200 ships as + `sqlite-autoconf-3510200.tar.gz`, a new 13.0 package that broke unpack), + with the unpack doc-filter extended to infix spellings so it cannot grab + `sqlite-doc-*`; the "download the sources" step is done only when EVERY + basename of the current book's wget-list is present, not "some tarball + exists" (leftovers from an older book were skipping the download); and a + fresh-run confirmation that states the book and the whole session config + once before touching anything. +- **`lfs-helper`**: `chown 0:0` instead of `chown root:root` for the wrapper + directory (before 7.6 there is no /etc/passwd in the tree, so names do not + resolve); `/tmp` created early if missing (sort needs it before init-dirs + runs); and a line saying the manifest scan is running, which on a fresh + tree looks like a hang. +- **`packagemanager_install`**: the "could not write to <dir>" box printed a + wrong remedy — swapped arguments, and the wrong command entirely when the + directory is ALREADY shared and the user simply is not in its collector + group. Hit in the wild with p11-kit vs /usr/share/zsh/site-functions. + +One test broke on the merge: it asserted `chown root:root "$WRAPPERS"` +literally, and the numeric-id fix is the same property spelled differently. +Fixed to accept either. Fifth time this session that a test matched an +implementation rather than an intent. + +## "Up to date" is not "built against this system" (1.11.53) + +After glibc changes, a package whose version did not move is still the old +binary. The version is right; what it was built against is not. So +`--reinstall` now reaches the LFS book's up-to-date packages too, via +`lfs update --porcelain --include-current`, and the plan can be the whole +system in book order rather than only the 29 whose numbers moved. Rows where +installed == book print as "(rebuild, same version)" so the plan does not +claim an upgrade it is not making. + +Worth recording honestly: glibc keeps backward ABI compatibility through +symbol versioning, so existing binaries keep working and a full rebuild is +NOT required the way it would be after an ABI break. It is required to know +the system is built against the new toolchain, which is a different +question, and one only the person can decide is worth the time. + +Another test asserted a call's exact spelling (`_lfs_stale_list()`) and +broke the moment the call took an argument. Fourth time this session. + +## Rebuild in book order (1.11.52) + +The stale list came out **alphabetically** — Python, binutils, coreutils, +... glibc — and that is a genuinely dangerous order to rebuild in: each +package is compiled against the ones before it, so Python rebuilt ahead of +glibc links against the old one and is stale again the moment glibc is +replaced. The list is now sorted by where each package appears in the book, +which is chapter 8's own build sequence; anything the book does not name +goes LAST rather than jumping the queue. packagemanager preserves that order +when it folds the LFS packages into its plan. + +Also: the systemctl hint now prints the FILE PATH, not just the package +name. The next thing anyone does with that list is open the file and add the +SysV equivalent, and the script it names is the one the build runs. + +## Said once, and said before the build (1.11.51) + +1.11.50 printed every stale package twice: the "the LFS book answers for +these" report ran BEFORE those packages were folded into the plan, so each +appeared as unplanned and then again as planned. The folding now happens +first and the report covers only what the plan does not. + +New: **a systemd book on a SysV system says which packages need systemctl.** +The shim (1.11.30) turns each call into a visible skip so the build +succeeds -- but "skipped" means the service was not enabled, and only the +person can judge whether that matters for a given package. The plan now +names them with a line count, before the build, instead of leaving the +notices to scroll past during it. Silent on a system that has systemd. + +## One update, both books (1.11.50) + +The plan held two packages -- what the BLFS book carries -- while +twenty-nine LFS-book packages were merely REPORTED, with a command to paste. +That was never the request: "update all, preferring BLFS" is what was asked +for in the first place, and preference means falling back, not stopping. + +`lfs update --porcelain` prints `name<TAB>installed<TAB>book` and nothing +else; packagemanager reads it and folds those packages into the SAME plan, +skipping any the BLFS book already covers -- so BLFS stays preferred, and +the LFS book fills in the rest. On `--run` an LFS entry is built by +`lfs install --reinstall <name>`, which already knows how to fetch the +tarball, generate the script and hand over to lfs-helper; reimplementing +that here would have been a second builder for the same book. + +Parsing the human report was the alternative and was rejected: it breaks the +first time a word changes, which this tool has already been bitten by. + +**Toolchain packages need consent.** glibc, binutils, gcc, libstdcpp and +linux-headers are planned and shown, but skipped by `--run` unless +`--toolchain` is given. Rebuilding those on the running system can leave it +without a working compiler, and that is not the same act as rebuilding +`less`. + +For the third time in this session a new test was inserted after the +`if problems: sys.exit(1)` that guards its block, and passed on the old code +until the red check caught it. Worth a rule: when adding to an existing +python test block, put the assertions ABOVE that guard, and never trust a +new test that has not been seen red. + +## A run's record must not outlive the run (1.11.49) + +**m4 built end to end** — fetched from the wget-list, unpacked, compiled and +installed as p_m4 through the shared runner. First real proof of step 3b and +of the single-package fetch, on the live system. + +It ended with "Collector groups used during this run:" and nine grants for +p_urllib3, make-ca and p11-kit — all real, all made weeks earlier. The +granted-groups file is only cleared by the report, so any build that granted +without reporting (a failure, another path) left its entries for the next +successful build to claim as its own. A build now starts with an empty +record, and the heading says what it means: "Collector groups this build +needed". + +Also fixed: a `DeprecationWarning` from 1.11.48's own code — +`re.split(pat, s, 1)` must pass `maxsplit=1` by keyword on Python 3.13. It +printed on every install, from the tool that had just been shipped to stop +noise. + +## The book ships the URLs; use them (1.11.48) + +`lfs install --reinstall m4` failed with "no M4-1.4.21.tar.* in /sources -- +run: lfs build-system get-sources --run" — while the URL for exactly that +tarball sat in the wget-list this tool already downloads and caches with +every book. Only `get-sources` ever read it, and only in bulk, so updating +one package meant fetching everything or nothing. + +`_wget_list_url_for` looks up a single package's URL, `_have_source_for` +answers whether it is already in the tree's /sources (with the same loose +matching the generated script globs with — a stricter test here would fetch +a file the build then refuses to see), and `cmd_install` fetches what is +missing before handing over to the build. The tree root is passed in rather +than guessed: inside the chroot /sources is /sources, outside it is +$LFS/sources, and a fetch into the wrong one would look like it worked. + +Matching is on the version token, so `Util-linux-2.41.3` finds +`util-linux-2.41.3.tar.xz` — the hyphenated names are where a naive split +goes wrong — and a package not in the list returns nothing rather than a +confident wrong URL. + +The test for this passed on the OLD code at first: it was inserted after the +`if problems: sys.exit(1)` that guards its block, so every failure it found +was collected and never reported. A test that cannot fail, again, and this +time caught by the habit of checking red before green rather than by luck. + +## A heading that promises errors must show some (1.11.47) + +The first real build through the shared runner failed for an ordinary reason +-- the 13.0 book wants M4-1.4.21 and only the 12.4 tarballs are in /sources +-- and the failure report printed: + + --- first errors in the log --- + ------------------------------- + +Nothing between them. The log said `no M4-1.4.21.tar.* ... in /sources -- +run: lfs build-system get-sources --run`, which matched none of the +compiler-shaped patterns (`error:`, `*** [`, `undefined reference`). No +pattern matching is a statement about the PATTERNS, not about the log: it +now says so and shows the tail, which is where a non-compiler failure +usually is. Empty-as-no-findings, fifth appearance. + +Also: `lfs-helper run-script` did not answer `--help`, which every other +command does. It does now. + +Not a bug, worth knowing: moving to a newer book means the new tarballs are +not in /sources. `lfs build-system get-sources --run` on the host fetches +them before any of the 30 stale packages can be rebuilt. + +## One runner (1.11.46) — item 6, step 3b + +The last structural piece of the split. Running the install script existed in +**three** copies, two of which I only found because the test looked: + +- `cmd_build`: `su - "$owner" -c "env ... bash SCRIPT PHASE" | tee` +- `cmd_build`'s permission-RETRY loop: the same three-branch block again — so + a retry could run the build differently from the attempt it was retrying +- `packagemanager_install`: `su - -c "set -o pipefail; PATH=...; bash + ~/install_<user> | tee" <user>` + +The differences were exactly where this project's worst bugs lived: whether +the wrappers are in PATH (without them the book's own `install -vdm755 +/usr/sbin` fails as "cannot change permissions"), and whether the status +comes from the script or from `tee` (a failing build reporting success cost a +day). + +`run_phase_as` is now the one runner, with the three privilege cases named +and kept distinct: as root (no wrappers — root legitimately chowns), as the +owner (wrappers first), and root-then-hand-over for the packages built before +Shadow exists. It returns the SCRIPT's status via PIPESTATUS, never tee's. +`lfs-helper run-script --user U --script S --phase P` exposes it to callers +that are not this script, and packagemanager_install uses that instead of its +own line (keeping the old one only where lfs-helper is absent). + +Proved functionally, not by grep: a script exiting 7 reports 7, a passing one +reports 0, the phase argument arrives, and the su branch really runs as the +target user with the wrapper directory first in PATH. + +With step 3 done, what remains of the plan is step 4 (phase tracking into the +shared runner), step 5 (delegate the rest), step 6 (staging follows), and +step 7 (retire the fifth tool). + +## A kept script says which version it builds (1.11.45) + +`lfs script fetch m4` printed the book's heading — `m4 (8.14. M4-1.4.21)` — +above a script generated from book 12.4 that builds M4-1.4.20, and said +nothing about the difference. Keeping the script is right (it is the +editable file, and regenerating discards edits), but two versions on screen +with no relation stated invites the wrong conclusion. It now says which +version the kept script builds, what the book has, and that `--regenerate` +rebuilds it at the cost of edits. + +## One stamp, read by everyone (1.11.44) — item 6, step 3a + +Step 3 opened better than expected: the two generators are NOT far apart. +Both already emit `unpack_pkg()`, `build_pkg()`, `install_pkg()` and take the +phase as `$1`; the lfs one adds `#### BUILD ####` markers its own driver +greps. What actually differed was the STAMP, and that is what made +packagemanager call every LFS-book script "OLD-format" and offer to migrate +it into a BLFS one. + +Both generators now write one shared line: + + ### script format v<N> origin=<tool> + +`_script_origin` reads it, with a fallback to each tool's older mark so +existing scripts keep working, and `_package_script_outdated` judges only +what `blfs` generated. The counters stay PER GENERATOR (blfs v4, lfs v17) +and are never compared with each other — comparing them would be a number +against an unrelated number. The 1.11.40 containment (recognising the other +tool's dialect) is gone: recognising was the workaround, asking is the fix. + +Verified by generating a real crosschain body and reading its stamp back +with packagemanager's parser, plus all five origin cases (new blfs, new lfs, +legacy blfs, legacy lfs, unstamped). + +**Step 3b, the runner, is deliberately NOT in this release.** The `su - … +bash install_<n> <phase>` invocation, the wrappers, `pipefail`/`tee` and the +exit-code capture are one function's worth of work in the live build path, +and the plan's own warning is "doing it as one commit" — so it gets its own +red-tested release. + +A test broke on the way and earned its comment: it asserted that +`_package_script_outdated` CALLS `_is_lfs_book_script` — the mechanism — and +failed the moment the check became a better one. Assert intent, not +implementation; third time this exact lesson has appeared. + +## A refresh that does not record what it did (1.11.43) + +1.11.42 refreshed all five tools on entry and the chroot still opened with +"These tools differ from the copies on the host", naming all five. The +warning does not compare files: it compares `tool-stamps.json`, written by +install-tools. Refreshing the binaries and leaving the stamps alone made +packagemanager announce that tools were stale seconds after being handed +fresh ones, and advise a sync -- on the path that had just synced them. The +refresh writes the stamps now. + +Still open on the real system: the refresh reported all five as updated on +EVERY entry, which cannot happen if the copies landed -- the md5s would +match on the second run. So the refresh now VERIFIES each copy and names any +that did not change on disk. That turns a silent repeat into a stated +failure with a reason. If it still repeats with no failure named, something +outside the refresh is rewriting the tools in the tree between entries, and +the next step is comparing `md5sum /usr/bin/<tool>` on the host against +`$LFS/usr/bin/<tool>` right after an entry. + +## Entering refreshes everything, and changes nothing (1.11.42) + +`chroot enter` refreshed **lfs-helper alone**. The four python tools inside +stayed older than the host's, which is why every packagemanager run in the +chroot opened with "These tools differ from the copies on the host" and told +the user to sync -- advice printed by the very path that had just synced one +file of five. `_refresh_chroot_tools` now content-compares all five and +carries the completion with them, silent when nothing changed. + +The other half, and the reason this needed care: `install-tools` copied the +host's `config.json` straight over the tree's. So `lfs set-default +13.0-systemd` INSIDE the built system was undone by the next entry, back to +whatever the host had, without a word. The config is MERGED now -- host +values only where the tree has nothing to say, the tree's own book default +kept and reported when it differs. A setting the person changed last must +not be reverted by a command whose job is to copy binaries. + +Also: `lfs update` colours the version you would move to, matching +packagemanager's plan, and guarded by `_tty_out()` like every other colour +here -- escape codes in a piped report are noise in a file. + +The refresh test caught a `NameError` (shutil is imported per-function in +this file, not at module scope) before it could reach a chroot. Worth +noting: a functional test on a temp tree found in a second what a static +grep would have shipped. + +## Ask a question whose answer can be no (1.11.41) + +With the false clean bill gone, the honest report showed two more, both in +`lfs update`: + +- **`_blfs_knows` said yes to everything.** It asked `blfs debug <name>` -- + a DIAGNOSTIC, which prints "resolve: NOT FOUND" and exits 0. So every + string passed to it was "in the BLFS book", and the report ended with + `blfs update gcc-pass1 cfg_fstab init-dirs libstdcpp ...`: LFS build steps + offered as BLFS packages. It now asks `blfs versions`, a fact — the set of + anchors the book actually has, read once per process. +- **The temporary chapter-7 toolchain was offered for reinstall.** + `util-linux-tmp 2.41.1 -> 2.41.3`, `python-tmp`, `gettext-tmp` in the + `lfs install --reinstall ...` line — rebuilding the throwaway stage builds + onto a finished system. Only the STAGE VARIANTS are excluded now + (`-tmp`, `-passN`), from the step tables. + +The first draft of that exclusion took every name in `CHROOT_STEPS_7` and +`CROSSCHAIN_STEPS`, which silently hid `glibc`, `coreutils` and +`linux-headers` — chapter 6 builds those under their own names. Caught by +asserting both directions in the same test (these must be excluded; these +must NOT be), which is worth doing whenever a filter is introduced. + +Two testing notes: the new assertion matched the word "debug" and fired on +the comment explaining the old behaviour — "match invocations, not message +text", earned for the second time in five releases. And a 1.11.35 test broke +because `_lfs_update_report`'s contract changed under its stub; a stub is a +copy of an interface and ages like one. + +## Silence is not a clean bill of health (1.11.40) + +Step 1 worked -- `reconcile-records --run` wrote 80 pkg.lst files and update +went from 96 unjudged to 9 -- and the first good run exposed three things. + +- **A manufactured clean bill.** `packagemanager update` printed "all current + with the LFS book" while `lfs update` reported thirty stale packages. + `_lfs_update_report` returned `[]` both for "everything is current" and for + "the call produced nothing", and the caller printed the same reassuring + line for both. It now returns ("current"|"lines"|"failed", …) and a failure + is stated: *these are UNCHECKED*. Empty-as-pass, found for the fourth time + in this project, this time in the helper written to stop packages + disappearing quietly. +- **The book was re-parsed per package.** `_find_lfs_package` called + `load_soup` for every name, so `lfs update` with 77 packages parsed a 10 MB + HTML file 77 times -- measured at 27s of pure parsing, and long enough for + a caller with a timeout to give up. `load_soup` now caches per path: 0.35s + once instead. The consult also asks `lfs update` with no names now, which + is the same question a person asks by hand. +- **A script from the other book was called OLD-format.** Only blfs stamps + `script format vN`; `lfs` stamps `### lfs-crosschain-script vN`. So every + LFS-book script failed the format test, and the advice attached to it was + worse than the warning: `migrate-scripts` would rewrite an LFS-book script + into a BLFS one. `_is_lfs_book_script` now recognises the other dialect and + leaves it alone. + +That last one is item 6 talking: two generators, two dialects, and a tool +confidently misreading the other's work. It is a containment, not a cure -- +**the cure is plan step 3 (one script format / one runner), which this moves +to the front of the queue**, ahead of step 2. + +## One recorder, two strategies (1.11.39) — item 6, step 1 + +The first step of the written plan, and the one that removes the class of bug +that prompted it. What a package installed was written down twice, by two +drivers, in two places, and each tool read only its own. + +`lfs-helper record-install <pkg>` is now the single writer. It chooses by one +named predicate, `_record_strategy`, which turns on the existing +`ownership_established`: + +- **before ownership** (the build stage): the build-stage manifest stands + alone, because a package's files are not owned by its package user yet and + an owner scan would return nothing; +- **after**: `_write_pkg_lst` does the owner scan, and pkg.lst is the record + everything reads. + +`lfs-helper reconcile-records --run` gives every package built during the +build stage its pkg.lst once ownership exists — without a rebuild. Both +drivers call the one door: lfs-helper's build path after it writes the +manifest, and packagemanager_install in place of its own inline +`list_package > pkg.lst` (which falls back to the old line only where +lfs-helper is absent). + +Two traps met on the way, both caught by testing against a real account +rather than by reading: + +- `SCAN_PRUNE` is empty until `scan_prune_set` runs, so the first version of + the owner scan would have swept /build and /sources into the record — the + same bug that once put 3348 scratch files into a manifest. +- An owner scan that finds nothing must NOT write an empty pkg.lst over a + real one. It returns non-zero, says so, and leaves the manifest standing. + Empty-as-pass has cost this project three separate bugs. + +Verified end to end on a real `p_`-prefixed account: no pkg.lst during the +build stage, a correct one after, the home itself excluded, the empty-scan +guard preserving the old record, and reconcile writing what was missing. + +**Next: step 2, the version writer** — one function for VERSION and +install_last, and 1.11.37's read-both fallback comes out with it. + +## Ask for the account, prefer the installed record (1.11.38) + +1.11.37 taught packagemanager to read `VERSION`, and the next run reported +"0 of 97 current, 97 it cannot judge" — unchanged. The fallback was never +reached. Two bugs, one after the other: + +- **`gather_user` looked the PACKAGE name up in the user database.** + `list_all_users` hands names back unprefixed ('glibc'), the account is + 'p_glibc', so `getpwnam` raised KeyError, `has_user` stayed false, and + `state` came out 'broken' — landing every LFS-built package in the "cannot + judge" bucket with its home, VERSION and manifest sitting right there. Now + through `pkgusr_name()`, which is idempotent, so a name that already + carries the prefix is safe. `_user_exists` takes either spelling too (raw + first, so a real account like 'root' is never reinterpreted). +- **Record precedence, found while fixing the first.** The home also holds + the install script, and that script is regenerated whenever the book moves + on — its `name_version` is what the BOOK has. Reading it before `VERSION` + would compare the book with itself and report everything current, forever, + silently. `lfs update` carries a comment warning about exactly this trap. + Order is now: `install_last` (what ran) → `VERSION` (what is installed) → + the script, last resort only. + +Both proved against a real `p_`-prefixed account with a real VERSION file, +not by grep. Note for item 6: **neither of these is a build-loop bug** — they +are reader bugs, and unifying the loop would not have removed them. What the +unification does remove is the two-records-of-one-fact split underneath. + +## One fact, two files (1.11.37) + +With 1.11.36's accounting switched on, the real number appeared: **0 of 97 +current with the BLFS book, 97 it cannot judge** — on a system where `lfs +update` compares all 97 without trouble and finds 30 stale. Not a reporting +bug this time, the underlying split: both build paths record what they +installed, packagemanager into `install_last`, lfs-helper into `VERSION`, +and packagemanager read only its own. Every package the LFS build produced +was therefore version-less to it. + +`read_install_last` now falls back to `VERSION` (install_last stays +authoritative where it exists). That required a second change: the books +capitalise differently — BLFS anchors are lowercase (`which-2.23`), the LFS +book titles its packages (`Which-2.23`) — so `_same_version` compares +case-insensitively. An exact compare would have invented an update for every +package both books carry, which is exactly the hazard of reading VERSION at +all. + +This is a bridge, not the cure: **open item 6 (one build loop) is what this +keeps arguing for**, and each release that works around the split adds to +the case. Also fixed here: `--run` printed "check them with: lfs update" +where the dry run gave real answers — with no BLFS work to do there is time +to consult, and withholding an answer already in hand helps nobody. + +## Two silent exits, and unknown read as current (1.11.36) + +1.11.35 fixed the REPORT and the contradiction survived: 97 checked, +"nothing to update", 1 package named as outside the book — while `lfs +update glibc` said 2.42 -> 2.43. The number gave it away. 1 of 97 in the +not-in-book bucket meant the other 96 were leaving the classification loop +by two paths that said nothing: + +- `if u.state != "installed": continue` — silent unless the package had been + NAMED. Everything lfs-helper built has no `install_last`, so a + chapter-8 package reads as "not installed" here and simply vanished from + every update-all. +- `outdated = u.version and bv != u.version` — with an UNKNOWN installed + version the whole expression is falsy, so the package was filed as up to + date. Not knowing is not the same as being current; that one predicate + turned missing evidence into a clean bill of health. + +Every package now lands in a named bucket (`no_book`, `no_record`, +`unjudged`) and the report makes the arithmetic visible: "Of 97 checked: N +current with the BLFS book, M it cannot judge" — so a bucket that swallows +things can never again hide behind a total that looks healthy. + +Two testing notes worth keeping. The bucket test caught a SECOND copy of the +bad predicate that turned out to be my own comment quoting the old code — +"match invocations, not message text", the handoff's own lesson, earned +again. And the first version of that test failed on old code with a bare +TypeError instead of a reason; a red that does not say why is only half a +test. + +## The whole system is answered for (1.11.35) + +`lfs update glibc` said 2.42 -> 2.43 while `packagemanager update` said +"Nothing to update" in the same breath. Both were right: the BLFS-matched +subset WAS current — but 1.11.31's not-in-book report printed only on the +non-empty-plan path, and the early "Nothing to update" return skipped it in +exactly the state that needs it most (a mostly-LFS system whose few BLFS +packages are fine). Now one shared `_report_no_book` runs on both paths, the +verdict says what it actually judged ("Nothing to update from the BLFS book +(...) — every package it carries is current"), and the dry run CONSULTS the +LFS book through `lfs update <names>` — one door, lfs owns that comparison, +packagemanager only relays its answer (including the ready-made +`lfs install --reinstall ...` line). + +Also, per request, the script-fetching verb reads the same in every tool: + +- `packagemanager script fetch <pkg>` — BLFS script into the package home's + canonical `install_<name-version>` (the file resolve_script prefers, so + fetch→edit→install is the 1.11.31 conflict flow by construction), chowned + to the package user, nothing run. Not-in-BLFS points at the lfs variant. +- `lfs script fetch <pkg>` — same idea for the LFS book; the generation was + factored out of cmd_install into `_ensure_lfs_script`, the ONE generator + both go through, so fetch edits exactly what install runs. BLFS-first hint + mirrors cmd_install; `--lfs-book` forces. +- `blfs script fetch <pkg>` — `blfs script` always WAS fetch; the leading + verb is accepted so the spelling matches. +- `lfs search <fragment>` — packages and sections of the LFS book, with a + pointer at `blfs search` for the other book. + +## The wrapper asks; the person browses (1.11.34) + +The first run of 1.11.33 on the real system showed three bugs at once, two +of them introduced BY 1.11.33. The book name came from the first [bracketed] +filename of `blfs books` — alphabetical, so 12.4 beat 13.0-systemd and +update reported the wrong book even with the right default set; it was only +ever correct while the default happened to sort first. +`_current_blfs_book` now asks `blfs books --brief` (default + cached list, +no network) and reads the Default line. Second: books' new auto-discovery +ran through `run_blfs`, so every packagemanager invocation probed the site, +twice — --brief is the wrapper's contract, discovery stays with a person at +a terminal. Third (pre-existing, exposed by the forwarded notes): +enumeration probing stale index directories ('oldsvn', 'blfs-book-6.0-html') +fell into the resolve-for-use cached fallback, emitting a note per junk +directory and returning the cached book under a false name — +`discover_dir(listing=True)` raises instead, and the listing skips. +Forwarded notes are also deduplicated per invocation: ninety blfs calls in +one plan must not print the same note ninety times. + +A 1.11.33 test broke here for asserting the filename-folding regex — the +implementation — instead of the property; rewritten to assert the name +never comes from a bracket-scraped filename. Assert intent, not +implementation: one more notch on that lesson. + +## A chosen book is a decision (1.11.33) + +`blfs set-default 13.0-systemd` answered "Default book set" — and +`packagemanager update` kept reading 12.4. Three holes in one flow, all now +closed: `set-default` validated nothing, so an uncached default sent every +lookup to the closest-cached fallback (it now calls `ensure_book` and +fetches the book while the person who chose it is looking, or says plainly +that it could not and what falls back until `blfs fetch` succeeds); blfs DID +warn ("note: no book matched...") but `run_blfs` captures stderr and threw +the note away (notes are forwarded now — a decision being overridden is +never silent); and the book was displayed by filename +(`BLFS-BOOK-12.4-nochunks.html` reads as `12.4` everywhere now). + +Also per request: `blfs books` discovers automatically when a connection +exists — `discover_all` raises when no index was reachable instead of +returning its named-book guesses as a tiny "successful" discovery (the +empty-result-as-pass trap, again), and the offline listing says it is the +cached one. `--discover` remains the explicit refresh that reports failure +loudly. Note `lfs books` always discovered; only blfs hid it behind the +flag. + +## Output that reads as one voice (1.11.32) + +Three paper cuts from the first real `packagemanager update` run. `run_blfs` +set a status line ("reading versions from the book") and left it standing, +so the caller's next print landed on the same line — cleared now in a +finally, at the one door, for every status any blfs call sets. The +"Checking N package(s)" line and the "Nothing to update" verdict name the +book they compared against (`_current_blfs_book`, one parser shared with +the init-mismatch check — the two must name the same book). And `lfs +set-default` stopped lecturing about `build_book` on every invocation; it +says what it did and stops. + +## Update-all already existed; now it is honest (1.11.31) + +`packagemanager update` with no names has been the whole-system update all +along — every installed package user, versions from the BLFS book in one +call, dry run by default, plan printed in exactly the order --run executes, +cached and resumable. What its plan did not deliver, it now does: + +- **Local edits are conflicts, not casualties.** `install_last` is the + read-only record of the script as it last ran; the canonical + `install_<version>` beside it is where edits live. When they differ and + the update would generate a fresh new-version script, + `_pending_edit_conflict` flags it: collected at the END of the dry run in + one block, and `--run` STOPS before the first one — the plan is saved, so + fixing and re-running resumes. Self-clearing: a merged + `install_<new-version>` in the home resolves it (resolve_script prefers + it); `--regenerate` declares the discard deliberately. The make-ca + systemctl edit was this class. +- **Not-in-book packages are named, not dropped.** The `if not bv: continue` + made LFS-book packages vanish from update-all without a word; the dry run + now lists them and points at `lfs update` for that side. Driving the LFS + book through packagemanager's engine is open item 6 (the two build loops), + deliberately not half-built here. +- **The bash completion travels with the sync** — the third forgotten file + after skel-package (never forgotten) and skel-u_xdg (1.11.29): + `_install_completion` runs in install-tools and sync-tools, placing the + script plus the blfs/packagemanager/lfs-helper symlinks. Overwritten, not + kept: stale completion is worse than none. + +## The other book is a choice, not a mistake (1.11.30) + +The SysV BLFS book is no longer maintained, so running the systemd book on a +SysV system stopped being an error to warn about and became a decision to +support. Three pieces, found by the first real end-to-end install +(`packagemanager install which --run` — which passed, closing the last +verification item): + +- **Generated scripts shim a missing systemctl.** `command -v systemctl || + systemctl() { echo "# skipped ..."; }` at the top of every blfs-generated + script: on systemd systems nothing changes, on SysV every book `systemctl` + becomes a visible skip instead of `command not found` aborting the phase. + This retires the recurring hand-edits (the make-ca one included, going + forward). +- **The mismatch warning asks once.** `_warn_init_mismatch` records the + accepted pair as `init_mismatch_ok=systemd-on-sysv` in the packagemanager + config; a different mismatch direction asks again. Undo by removing the + key. +- **Downloads retry.** GNU's ftpmirror 502'd twice and succeeded on the + third manual run via a mirror redirect — exactly what a retry loop does + unattended. All generated-script downloads go through `_fetch`: three + tries, 5s apart, `wget -c` so a retried partial continues instead of + landing beside itself as pkg.1. + +Also closed on the real system this session: the collector-group gid check +came back clean (8 nimgnu_* groups, none below 90000), and p_which reused +the freed uid 10093 — the account allocator working as intended. + +## The XDG profile travels with the sync (1.11.29) + +`make install` placed `/etc/pkgusr/skel-u_xdg/.bash_profile`, but no chroot +is populated by `make install` — the tools arrive through `install-tools` +and `sync-tools`, and that path carried `skel-package` verbatim while this +one file was forgotten. So every synced system was missing it, and every +`--shared` user creation warned. `_install_xdg_skel` now runs in both paths: +the host's copy is preferred verbatim (a filled-in `XDG_RUNTIME_DIR` is the +right answer for the machine it syncs to, and `packagemanager` only +substitutes where the placeholder is still present), the placeholder copy +beside the tools is the fallback, and an existing copy in the tree is never +clobbered — same rule as the Makefile. The regression test proves +keep-if-exists functionally and fails on any tool where either sync path +skips the file. This closes the last real-system suite failure at its cause; +the pending-list `make install` entry below remains only as the immediate +fix for the already-built system. + +## The suite must be honest on the machine it runs on (1.11.28) + +The first run on the real, built system reported 12 failures. Ten of them +were the suite reading the machine instead of testing the code — and two of +those uncovered real tool bugs the bare checkout could never show: + +- **`packagemanager` warned onto stdout.** The config-conflict note ("...is + 'n' here but 'nimgnu' in the lfs config") was printed by `printWarning` to + stdout, so anything capturing a value captured the note with it. Moved to + stderr, matching `lfs-helper`'s `warn()`. The values it polluted were + correct all along. +- **`BLFS_STORE` half-applied.** `_blfs_book_on_host` checked the override + first but fell through to `/usr/share/blfs` when the overridden store was + empty — so an empty override reported the system's books as its own. When + set, the override is now the only place looked. The oldest bug shape in + this project, in one more place. + +The rest were suite fixes: the LFS book is now *found* (beside the tool, then +`$LFS_STORE/books`) instead of a path baked in from whoever ran it last; a +bare tool name resolves off `$PATH` and an unreadable tool stops the run with +one message instead of hundreds of tracebacks; the remove_tools/strip +*default* checks point `LFS_STORE` at an empty directory so the machine's own +interview answers cannot fail them; the XDG-profile test accepts the +installed location `/etc/pkgusr/skel-u_xdg/` as well as beside the tool; and +the empty-sources refusal test SKIPs on a system where bootstrap has nothing +left to install, because exit 0 there is the truth. + +The two remaining real-system failures after the first pass were down to two +after 1.11.28, and the second run explained both: the `/tools is deleted by +default` check existed TWICE in the suite — the isolated copy passed while an +unisolated duplicate in the up-front-decisions block still read the machine's +`remove_tools=yes` answer. One decision held in two places, in the suite +itself; the duplicate is gone, the isolated check is the one door. The XDG +failure was the test telling the truth: the profile really is missing at +`/etc/pkgusr/skel-u_xdg/` on the built system — the tools were copied by +hand, and only `make install` places the skel. Strip's dry run on the real +tree behaved: 1098 ELF files, 3135 MB, 70 owners, nothing touched. + +## Stripping runs as the file's owner (1.11.27) + +Open item 2, closed. `strip` was asked for in the interview and ignored with a +NOT YET IMPLEMENTED notice at the end of the build. Now `lfs-helper strip` +does it, and the design follows from one fact: **strip does not edit a file, +it writes a stripped copy and renames it into place — and the copy belongs to +whoever ran strip.** Run as root, every binary becomes root's and the +ownership record is gone. So the candidates (the book's 8.85 set: `*.so*` +under /usr/lib and /usr/libexec, every ELF file under /usr/bin, /usr/sbin, +/usr/libexec — real ELF checked by magic, not by name) are grouped by owner +read from the filesystem, and each group is stripped **as that owner** via +`su`. The sealed install directories permit exactly this: group-writable for +`install`, sticky, so an owner may replace its own file and nobody else's. + +Libtool `.la` files are removed too (the book's cleanup; `--keep-la` keeps +them) — deleting is a directory operation, not a rewrite, so root does that +part directly. In-use binaries (bash, strip, libc) are safe to strip in place +because a running process keeps its open inode across the rename; the book's +copy-to-/tmp dance guards in-place editing, which GNU strip does not do. + +`build-all` now ends with `_strip_if_asked_for` → `cmd_strip --run` when +`LFS_STRIP=1` — one implementation, one caller. A failed run cannot fail the +build (an unstripped system is complete, only larger); it downgrades to the +old end-of-build notice, which now says what to run +(`lfs-helper strip --run`) instead of what is missing. Per-file failures are +printed as marked lines and counted, never swallowed. Files with an owner the +user database does not know are left alone, and that is said. + +Untested against a full real tree — a smoke test proved bytes drop and the +owner survives; watch the first `strip --run` on the real system with +`lfs-helper verify` afterwards. + +## Toolchain audit: one colour vocabulary (1.11.26) + +Rule, stated by the user and now enforced by a test: **red states a failure, +orange warns, green states success — in every tool.** + +What the audit found and fixed: + +- `lfs` had `fail()/warn()/ok()` and, beside them, **20 bare uncoloured + `sys.stderr.write("! ...")` errors**. All routed through `fail()` now, + rewritten mechanically via the AST (constant and %-format sites) so no + message text changed. +- `blfs` had green for commands and **no red at all** — its errors were the + only uncoloured errors in the set, and an error that does not look like the + other tools' errors reads as less serious. It has the same + `fail()/warn()/ok()` painters now (tty-guarded), and `-V` like every other + tool. +- `lfs-sanity.sh` printed `!!` plain. Problems are red, the clean verdict + green, colour only on a terminal so the saved report stays clean text. +- The engine printed its no-wrappers **warning in red**. Orange now: red is a + statement of failure, this is a statement that failure is coming. Its + `echoGreen "Creating:"` header (information, not success) is blue. + +**Verified while auditing:** `blfs` contains zero install commands — no +useradd, groupadd, su or chown. Install work flows through one path: +packagemanager → packagemanager_install → lfs-helper (owner-name, +pkgusr-home, grant-dir, wrappers). The unification asked about is real. + +**Worklist for the next session** (audit items seen, not yet done): +- lfs-helper's `say`/`warn` sites deserve the same pass lfs got — grep for + plain `echo` of error-shaped text. +- Help-message structure: lfs and packagemanager group commands with + headers; blfs's `--help` is flat argparse. Worth aligning. +- `--verbose` exists in lfs and packagemanager only; decide whether the + bash tools need it or the flag should be documented as python-only. +- Long-running steps mostly announce themselves (` $ ...` convention, + `_run_banner`); `blfs fetch` and `make-ca -g` announce, but `write_pkg_list` + regeneration in the background does not say how to check on it. + +## A collector group belongs in its own range, under one prefix (1.11.25) + +A real tree, otherwise clean, ended with: + + !! group nimgnu_p_openssl (gid 10093) is outside every convention + +Two faults in one name. + +**The gid.** `groupadd <name>` with no `-g` takes the next free system gid, so +the group landed at 10093 -- among the package-user accounts, where nothing +expects a group. `_create_collector_group` allocates from 90000-99000, and +every caller goes through it. A test scans for any remaining bare `groupadd`. + +**The name.** `nimgnu_name("p_openssl")` gave `nimgnu_p_openssl`: two prefixes +on one name, which is the thing "one account, one prefix" exists to prevent -- +and not even what the prompt offered, which showed `nimgnu_openssl`. The +owner's prefix is stripped before the collector's is added. + +**And the summary lied.** That `!!` came from an awk that printed its own line +and never touched `$hits`, so the report ended "no problems found by these +checks". A summary that disagrees with its own body teaches you to skip the +summary. It goes through `hit` now, and a test fails on any awk in +`lfs-sanity.sh` that prints `!!` itself. + +## A collector group grants a directory, never a package (1.11.24) + +The prompt said: + + 1) nimgnu_openssl (everything owned by 'p_openssl' -- fewer groups) + +which is simply untrue, and dangerous in a scheme whose whole point is that a +package cannot reach another package's files. A collector group grants the +directories it is **applied to**, one at a time. Choosing the owner-named +option only means the name can be reused on the next directory of that owner; +by itself it grants nothing extra. + +Reworded, with the guarantee stated outright: "Either way it grants $dir and +nothing else." + +## The file to edit is named last, and it is the one that survives (1.11.24) + +A failed install said + + install file: /tmp/packagemanager/install_files/install_make-ca-1.16.1 + +three messages before the end. Both halves wrong. + +**The path.** That is where a script freshly generated from the book lands. The +engine copies it into the package user's home, and the next run regenerates the +/tmp one — so an edit there is silently discarded, which is the exact failure +this message exists to prevent. It names the copy in the home when there is +one, and says explicitly that the other is regenerated each run. + +**The position.** It is what you do next, and it was buried above the logs and +the re-run command. It is the last thing printed now, in the error colour. A +test asserts nothing follows it. + +## Two registries, one question (1.11.23) + +`packagemanager which-package` is documented in the README and never existed: + + error: argument <command>: invalid choice: 'which-package' + +And `lfs-helper which-package` reads the **build** manifests only, so a package +installed ten minutes earlier by these tools answered: + + Nothing recorded means it was installed before file tracking existed, + or outside these tools entirely. + On disk now: -rwxr-xr-x p_wget:p_wget + +Wrong twice over, with the contradiction printed two lines below it. These +tools installed it, and they did record it — in the package user's own +`pkg.lst`. Chapters 5-9 record into `$STATE/manifests`; everything after +records into `pkg.lst`; neither tool looked at the other's. + +`packagemanager which-package` exists now, reads `pkg.lst`, and cross-checks +the recorded package against who owns the file on disk — those can disagree, +and that disagreement is worth seeing. `lfs-helper which-package` searches +`pkg.lst` before claiming nothing recorded it, and points at the other tool +either way. + +## The trigger must match what counts as done (1.11.22) + +1.11.19 made "stage 5 finished" mean *a bundle exists AND no store is empty*, +and left the trigger as *no bundle exists*. On a tree with the GnuTLS bundle +written and `/etc/ssl/certs` empty, stage 5 was skipped entirely — and then the +check reported the empty store as a failure. Correctly, and without ever having +attempted it. Reported twice, tried never. + +The trigger is `not _have_ca_bundle() or _empty_cert_stores()` now: the same +condition that decides it is done decides whether to run it. A test asserts the +two agree, and fails if the trigger drops the store check. + +When it still cannot fix it, the message says so — "even after granting and +generating again" — and prints the two commands to do it by hand, instead of +describing the problem twice in identical words. + +## A wrapper that exists can still be the wrong wrapper (1.11.21) + +1.11.18 fixed the install wrapper. 1.11.20 hit the identical failure on the +identical line: + + install -vdm755 /usr/sbin + install: cannot change permissions of ‘/usr/sbin’: Operation not permitted + +A snapshot had restored the **old** wrapper. It existed and was executable, and +`_wrapper_dir_from_lfs_helper` checked exactly that — "missing or incomplete". +Its comment claimed this "repairs a tree whose wrappers predate a fix"; the +code could not see "predates". The fixed wrapper on the tools side never +reached the tree. + +`make_wrappers` now stamps `$WRAPPERS/.written-by` with the same fingerprint +`--version` prints — same species, same cure as the stale chroot tools. The +engine compares the stamp against the live `lfs-helper --version` and rewrites +on any difference. The test corrupts the stamp and proves the rewrite happens. + +One trap inside the fix: the test suite lifts `make_wrappers` out with sed and +runs it under `set -u` where neither the version variable nor `_build_id` +exists, so the stamp uses `${VAR:-}` and a `command -v` guard — a stamp that +kills the function it stamps is worse than no stamp. Six tests caught the first +draft. + +**For the user's tree:** their snapshot restores old wrappers every time; with +this, the first `packagemanager` run after `install-tools` rewrites them +without being asked. The manual `lfs-helper make-wrappers --run` advice from +1.11.18 is obsolete. + +## The staging guard made the fix a no-op (1.11.20) + +1.11.14 staged the downloaded tarball into the package user's home before the +engine ran, guarded by "only if the home exists". For a **new** package it +never does -- the engine creates it -- so the guard turned the fix off in +exactly the case it was written for, and the identical error came back: + + install_p_wget: line 44: wget: command not found + md5sum: wget-1.25.0.tar.gz: No such file or directory + +It belongs in `packagemanager_install`, after `addUser`, where the account and +its home exist. The Python copy is gone: one door, and the one that could +never work is not the door. A test asserts the ordering, not just the presence +-- reintroducing the old placement fails it. + +**And the message was wrong about itself.** "ERROR: md5sum check failed" was +printed for two different things: contents that differ from the book, and a +file that is not there at all. The second is what a fresh system hits, and it +is not a checksum problem -- it is a download that never happened. It now +prints expected, got (or `<no file>`), and says which. + +`PM_SKIP_MD5=1` accepts a mismatch on purpose, for when the book's checksum +disagrees with a mirror's tarball. Not a default: a checksum skipped by +accident is worse than none, so it has to be asked for by name. + +## A zero exit is not a finished job (1.11.19) + +Bootstrap reported all five stages complete on a system whose OpenSSL trust +store was empty: + + install: cannot create regular file '/etc/ssl/certdata.txt': Permission denied + p11-kit: couldn't create file: /etc/ssl/certs/...pem: Unknown error 13 + Failed!!! + Extracting GNUTLS server auth certificates to: ...Done! + CA certificates found (/etc/pki/tls/certs/ca-bundle.crt) -- re-enabled + +`/etc/ssl/certs` is `p_openssl:p_openssl`, so `p_make-ca` could not write it. +make-ca wrote the stores it could, **exited 0**, and `_have_ca_bundle` found +the GnuTLS bundle and said yes. Certificate checking was then turned back on +against a store that is empty — and `/etc/ssl/certs` is OpenSSL's default +CApath, which is what `wget` itself reads. + +Three separate failures of the same kind: a success signal that was not +checked against the actual result. + +- **The errno is a number.** p11-kit prints "Unknown error 13", so `_PERM_HINT` + — looking for the word "denied" — skipped the line and no directory was + extracted. 13 is EACCES. Both its wording and `couldn't create file:` are + matched now. +- **Nothing granted or retried.** Stage 5 ran make-ca once, outside the + recovery every install goes through. `_run_make_ca` captures the output, + grants what the errors name, and runs it again once. +- **Existence is not population.** `_empty_cert_stores` checks that + `/etc/ssl/certs` actually holds certificates, and verification is only + restored when it does. An empty store is now a stage-5 failure. + +## The install wrapper must see bundled short options (1.11.18) + +**This was the actual cause of the make-ca failure**, under three rounds of +wrong diagnosis. The wrapper detected directory mode with + + for a in "$@"; do case "$a" in -d|--directory) want_dirs=1 ;; esac; done + +which matches `-d` only as a whole argument. make-ca's Makefile writes it +bundled: + + install -vdm755 /usr/sbin + +one token. The rule never fired, the real `install(1)` ran, and it died on the +exact message this wrapper's own comment quotes. The comment was written for +the unbundled spelling and the code only ever handled that one. + +Why no amount of granting helped: `install -d` on an **existing** directory +still applies the mode, and chmod requires ownership, not write permission. +`/usr/sbin` is `root:install`, `p_make-ca` was already in `install`, and +`_user_can_write` correctly said yes. Writing was never the problem. + +The wrapper now expands a short cluster into canonical form before any rule +looks at it — `-vdm755` becomes `-v -d -m 755` — so one spelling reaches all of +them. `m o g t S` take a value: the rest of the token if there is one, else the +next argument. + +**A second bug fell out of it.** The setuid guard matches `-m` followed by a +four-digit mode, so `install -vm4755` slipped past it entirely and set the bit. +It is refused in both spellings now. + +The test lifts the wrapper out of `lfs-helper` exactly as it is written and +runs it: bundled and unbundled `-d`, `--directory`, a directory that does not +exist yet, an ordinary file install, ownership stripping, and setuid in both +spellings. + +**Diagnostic note for next time.** Three sessions chased permissions because +the error said "Operation not permitted". The absence of the new "no +package-user wrappers" warning was the evidence that the wrappers were present +and running — an absent warning is evidence, and it was not read as any. + +## Granting cannot fix what is not a group problem (1.11.17) + +With 1.11.16 the grant fired, found the right directory, and then: + + /usr/sbin is already group install -- adding 'make-ca' to it + usermod: user 'make-ca' does not exist + +Two faults in three lines. + +**The name.** `auto_grant_from_log` was called with the package name. Third +appearance of that species in this file, after `_run_pm_install_once` and +`cmd_pip`. It takes `pkgusr_name(name)` now. + +**The diagnosis.** `/usr/sbin` is `root:install` and `p_make-ca` is in +`install` — the grant was for a group it already had. The actual failure is +`install -vdm755 /usr/sbin` CHMODing a directory root owns, which no package +user may do. That is what the **wrappers** are for, not the group. Granting a +group someone already holds looks like progress and is not, so it now says so +and points at `lfs-helper make-wrappers --run`. + +**And the underlying cause.** `packagemanager_install` falls back to +`_wrapdir=""` when it can establish no wrappers, then builds without them +silently — so a book command the wrapper is designed to absorb fails as a +permission error, and every diagnosis chases the permission. It says so loudly +now. Whether the wrappers were actually missing on the real system is still +unconfirmed; that message is what will answer it. + +## The auto-grant must recognise the message it is given (1.11.16) + +The recovery lfs-helper does during the base build exists in packagemanager +too: a package that must write into another package's directory joins that +directory's collector group, and the install is retried. It never fired on the +real failure. + + install: cannot change permissions of ‘/usr/sbin’: Operation not permitted + +Two reasons, both in `_PERM_PATTERNS`. coreutils quotes paths in the locale's +style — U+2018/U+2019 under UTF-8 — and every pattern matched ASCII +apostrophes only. And `change permissions` was not among the verbs, though it +is the one `install(1)` uses when the directory exists and only its mode cannot +be set, which is the commonest way a package meets a directory another package +owns. + +So no path was extracted, `auto_grant_from_log` returned nothing, the retry +loop stopped on the first round, and packagemanager printed the manual command +as though it had no opinion — while the engine's own diagnostic, which uses +different code, had already named the directory correctly. + +**The advice named the wrong file.** A package user's home holds +`install_<name>-<version>` (edit this) and `install_<user>` (a copy the engine +executes, overwritten every run). The error lines name the copy — +`install_p_make-ca: line 96` — while the advice named the original. Correct, +and it reads as a mistake because it contradicts the output above it. The +advice now says which is which. + +## --run means run, not print (1.11.15) + +Stages 3-5 were printed as commands to type. The reasoning was that each has to +be seen to work before the next is worth trying — but `bootstrap --run` is +someone saying "get this system ready", and answering with three commands to +type refuses the request while looking like an answer. + +`_bootstrap_finish` does them: `blfs fetch` for the book, `install make-ca +--recursive` so BLFS resolves its dependencies, and `make-ca -g --force` for +the bundle. Then `restore_wget_verification`, the moment there is something to +verify with. + +The part that actually mattered is kept: a stage whose dependency failed is +**skipped and said so**, never attempted and blamed for a failure that belongs +to something else. Without a book there is nothing to look make-ca up in; +without make-ca there is no bundle to generate. + +The dry run still prints the same list, as what `--run` would do. + +## The tarball is staged before the build, not only in new scripts (1.11.14) + +The `/sources` lookup added to the generated script in 1.11.11 only reaches +scripts generated after it. A **local** script is preferred over the book -- +it holds your edits -- so the one written by the previous run won, had no +lookup, and did: + + install_p_wget: line 44: wget: command not found + md5sum: wget-1.25.0.tar.gz: No such file or directory + +reaching for a download tool in order to install that download tool. + +`_stage_source_from_sources` reads `pkg=` out of whatever script is about to +run and copies the file from `/sources` into the package user's home first. It +sits in `_run_pm_install_once`, so every caller gets it, and it works for +scripts that already exist -- which the in-script version cannot. Copied, never +moved. + +Both remain: the script keeps its own lookup for when it is run by hand. + +## The account's home is settled where the account is made (1.11.13) + +`add_package_user` takes no home argument: it uses `/usr/src/<name>`, while +every other tool here puts package users in `/usr/src/pkgusr/<name>`. So the +account was created, a home was created, and the install ran in a third place +that existed in neither: + + tee: /usr/src/p_wget/log/...: No such file or directory + bash: /usr/src/p_wget/install_p_wget: No such file or directory + logs: /usr/src/pkgusr/p_wget/log + +The last line is packagemanager reporting the correct path in the same breath. +Two halves of one toolchain disagreeing, not a missing directory. + +`addUser` now reconciles it at the one point where the account comes into +existence: `usermod -d` to the computed home, and the half-populated directory +the hint made is **moved**, not abandoned — `.bash_profile`, `.bashrc` and +`.project` are already in it, and a passwd entry pointing at an empty directory +is the same bug wearing a different path. + +`installPkg` also re-reads the home after creating the user. It computed +`install_user_dir` at the top of the function, describing an account that did +not exist yet. + +**1.11.9 fixed the same fault in packagemanager's own `create_package_user`.** +It was never fixed in the engine, which is the path an actual BLFS install +takes. One rule, two implementations — item 6 again. + +## The parsers come before the thing that needs them (1.11.12) + +Installing the download tool means looking it up in the BLFS book, and reading +the book needs requests and beautifulsoup4. Stage 1 installed the tool and +stage 2 the parsers, so on a fresh system the first run could not read the book +it was holding: + + $ install wget --run + wget: not in the BLFS book. + searching the book for wget Nothing similar found either. + +It succeeded only on a **second** run, once stage 2 had put the parsers in +place — which reads as flakiness rather than as an order. Both earlier logs in +this project show it: the run that found `Wget-1.25.0` was a run where the +modules were already installed. + +The wheels need neither the book nor a network, so they are stage 1 now and +nothing is lost. When they fail, stage 2 says the lookup cannot work and does +not attempt it — a missing parser reported as a missing package is how this +hid in the first place. + +## The install engine is handed an account, not a package (1.11.11) + +Argument 1 of `packagemanager_install` is used for exactly one thing: the user +to create, own the files and build as. The bare package name was passed: + + $ /usr/bin/packagemanager_install wget /tmp/.../install_Wget-1.25.0 + id wget || addUser wget + uid=10089(wget) ... bash: /usr/src/wget/install_wget: No such file + +An account literally called `wget`, living in `/usr/src/wget` — no prefix, no +subdirectory, colliding with any real account of that name. A fresh snapshot +proved the install created it: `id wget` said no such user beforehand. + +Fixed at both ends. `_run_pm_install_once` passes `pkgusr_name(name)`, and +`installPkg` resolves through `lfs-helper owner-name` regardless of what it is +handed, because the script is also run by hand and by older callers. Both are +idempotent, so an already-resolved name passes through unchanged. + +## An already-downloaded tarball is used, not re-fetched (1.11.11) + +`unpack_pkg` looked for `$pkg` only in the build directory, so the first +install on a fresh system reached for the network — to download the very +package it was installing, over a connection it cannot verify, because make-ca +needs that same download tool. `get-sources` had already put the tarball in +`/sources` before the reboot. + +It now looks in `$LFS_SOURCES_DIR` and `/sources` first and **copies**, never +moves: `/sources` is the record of what was downloaded and no build may consume +it. The download still happens when nothing is staged. + +This is the circle the last session hit. With the tarball staged, the first +install needs no network at all. + +## One key, one meaning: pkgusr_home (1.11.10) + +A real install put its logs here: + + logs: /usr/src/pkgusr/pkgusr/p_wget/log + +`lfs` writes `pkgusr_home` as the FULL package-user root -- `LAYOUT_DEFAULTS` +says `/usr/src/pkgusr` -- and `_base_dir_default` read the same key as the BASE +and appended `PKGUSR_SUBDIR` itself. One key, two readings, and the tools only +disagreed on a tree that lfs had configured, which is every real one. + +Fixed on the READING side: a value already ending in the subdirectory yields +its parent, so both tools agree without either changing what it writes. A base +that does not end in it is still taken literally, which is what a hand-written +config means. + +**Still open from the same log**, not yet diagnosed: the account was created as +`wget`, not `p_wget` (`uid=10089(wget)`), and `packagemanager_install` used +`/usr/src/wget` for its home. The doubled path above is a separate fault from +the missing prefix; fixing the first does not explain the second. Needs +`/etc/pkgusr/packagemanager.conf` from the tree to go further. + +## The BLFS book has to be fetched where there is a network (1.11.9) + +A real bootstrap run walked into the circle: + + wget: there is no BLFS book on this system to look it up in. + Could not install: wget + +The built system installs a download tool by looking it up in the BLFS book, +and fetches the book with the download tool it cannot install. The host has a +network and `install-tools` already copies a cached BLFS book into the tree -- +what was missing was **asking**, at a moment when the answer can still be acted +on. + +`fetch the BLFS book (for the built system)` is a checklist step now, walked +through by `lfs run` like the LFS book choice. Declining is recorded in +`blfs_book`, so a resume does not ask again. It sits **before** `download the +sources` on purpose: `get-sources` asks `packagemanager bootstrap --sources`, +which asks the book where the download tool's tarball is, so with no book that +URL is silently absent from the download list and the gap only shows up after +the reboot. `get-sources` also says so when it finds nothing. + +The warning in `bootstrap_sources` names no package, deliberately: which +packages the built system needs is packagemanager's list, and a test forbids a +copy of it in `lfs`. It caught the first draft of that message. + +## A home that is recorded must be the home that exists (1.11.9) + +The same run reported both of these about one account: + + $ chown -R -h p_urllib3:p_urllib3 /usr/src/pkgusr/p_urllib3 + home directory created: /usr/src/p_urllib3 + +`add_package_user` takes no home argument, so the hint's script used its own +default while `init_package_user_home` created and chowned the one this tool +computes. The `useradd` fallback had always passed `-d`; the preferred path +never did. `su - p_urllib3` lands in a directory that does not exist. + +`_fix_recorded_home` compares the passwd entry against the computed home after +creation and corrects it with `usermod -d`, saying so. + +## Do not ask for what the system already knows (1.11.8) + +`packagemanager setup` in a fresh chroot opened with a wizard: + + Collector-group prefix [sysgroup]: + +on a system built by these tools, where `lfs build-system session` had asked +for that exact value, written it into config.json, and copied it into the +chroot. Being asked for a setting the machine is holding teaches you the +answer does not matter. It matters a great deal: a collector prefix that +disagrees with the built system gives two parallel sets of groups over the +same directories, and nothing reports it. + +`_adopt_existing_settings` answers from three sources, in order of authority: +the lfs config (the interview's own answers), the environment `lfs` exports +into the chroot, and the system itself — collector groups live at gid 90000+ +and every one carries the prefix, so on a built system the answer is in +/etc/group whether or not any config survived. The wizard now runs only for +what none of them answer, and prints where each adopted value came from. + +`_SHARED_WITH_LFS` was `collector_prefix` alone; `pkgusr_prefix`, +`cfguser_prefix` and `main_user` are collected by the same interview and were +being asked for twice. `install-tools` writes all of them into +`packagemanager.conf` now — bare, without the trailing underscore, because +lfs's accessors return `p_` ready to concatenate while packagemanager stores +the word and adds the underscore itself. That only worked because it rstrips +on read: an accident, not an agreement. + +**A default is not an opinion.** The conflict warning compared the lfs config +against whatever was in `_config`, and a built-in default counted, so a system +with no `packagemanager.conf` at all was told its settings disagreed: + + note: collector_prefix is 'sysgroup' here but 'nimgnu' in the lfs config + +Nothing had said sysgroup. `load_config` now tracks which keys came from the +file, and only those can conflict. + +## `setup` is `bootstrap`, and it walks you through (1.11.7) + +**The name.** Every other command is a verb naming what it does to packages -- +install, update, remove, verify. `setup` named neither the thing nor the +action, and the word was already taken: `packagemanager user setup` applies the +shared-account treatment to an existing application user. Two unrelated +commands, one word. `setup` remains as a hidden alias, because an installed +`lfs` calls `packagemanager setup --sources` from `get-sources` and a rename +that breaks the tool which downloads the sources costs a reboot to discover. +`get-sources` asks for `bootstrap` first and falls back. + +**The walkthrough.** Every check already existed -- wget, the book, make-ca, +the bundle, the wgetrc weakening -- as separate housekeeping that fired +whenever it happened to fire. What was missing was the ORDER, on a system where +all of them are unsatisfied at once and nothing says which to do first. +`bootstrap` is now five numbered stages: + + ok 1. a download tool /usr/bin/wget + -- 2. book-parsing modules 0 of 7 installed + -- 3. the BLFS book no book on disk + -- 4. make-ca not installed + -- 5. CA certificates no bundle generated yet + +Stages 1 and 2 need no network -- they come from what `get-sources` already +put in /sources -- so `--run` does those. Stages 3-5 each need a working +download that the previous stage buys, and each has to be SEEN to work before +the next is worth trying, so they are printed with the exact command and never +run. Running it again shows where you are. + +## A native gcc means opposite things on either side of the build (1.11.7) + +Snapshotting a finished system said: + + toolchain: BROKEN: a native gcc (x86_64-pc-linux-gnu) is installed over + ! this snapshot is of a tree whose toolchain is already broken. + +on a tree with nothing wrong with it. Chapter 6 installs only under `$LFS_TGT`, +so mid-build a native gcc on top of the cross one is the toolchain being +clobbered. Chapter 8 **replaces** the cross toolchain with exactly that native +gcc: it is the finished state, the thing the build is for. + +`_build_is_finished` asks the same question `lfs-sanity.sh` does -- every step +in `progress/steporder` present in `progress/steps-built` -- and both the +snapshot label and `verify`'s WARNING now read the native gcc against it. Same +species as the sticky-bit check, which had to learn the same distinction. + +## The kernel and the bootloader are yours (1.11.6) + +Both are decisions about the whole machine rather than about a package, and +getting either wrong costs the system doing the building. So the build does +neither, and ends by **saying** so — `_say_what_is_yours_to_finish`, beside the +login and strip warnings — rather than leaving it to be discovered after a +reboot. + +`bootloader` was already `none` by default and stays that way. The prompt now +says what that means: nothing is written to any ESP, boot sector or partition +table, and the machine keeps booting exactly as it does today. rEFInd is opt-in +by name, and the closing note prints how to ask for it. + +## $LFS may never be the running system (1.11.5) + +Every path this tool writes is `$LFS/something`. Nothing checked that the tree +was not the host's own, so a config holding `/`, an `export LFS=/` in the wrong +shell, or a typo in the interview was enough for +`build-system restart --run` to delete the top-level directories of the machine +doing the building, and for `_verify_lfs_ownership` to run `chown -R lfs /usr` +on it. + +`_is_host_path` is the rule; `_refuse_host_tree` is the hard stop. It is called +from `require_lfs_for_root`, the single door every command resolves `$LFS` +through, and from `_verify_lfs_ownership`, which resolved `$LFS` itself and was +therefore the one destructive path with no check at all. The interview refuses +the same paths at the prompt, where you can just type another one. + +**Both spellings are checked.** On a merged-`/usr` system `/bin` is a symlink +to `usr/bin`, so `realpath("/bin")` is `/usr/bin` and a check on the resolved +path alone let `$LFS=/bin` through. The first version of the test caught this. + +**Inside a system tree counts.** `/usr/src/lfs` is not a build tree, it is a +directory in the system's own `/usr`. `/home` and `/srv` are deliberately not +treated that way: `/home/you/lfs` is an ordinary place to build, and a check +that refuses `$LFS/usr` refuses every build there is. + +What was already safe, and stays so: `restart` refuses while the chroot's bind +mounts are up, because deleting through `/dev` would reach the host. GRUB +scripts are not generated at all unless GRUB is the chosen bootloader, so +nothing that could run `grub-install /dev/sda` is left lying around. The rEFInd +step never formats, partitions or writes a raw device; it adds files to an +already-mounted ESP, backs the config up, adds a menu entry beside your +existing bootloader, and refuses to overwrite an existing rEFInd without +`REFIND_OVERWRITE=1`. + +## A suppressed failure is a bug waiting to be found (1.11.4) + +`soft <what> -- <cmd ...>` runs a command that may fail, and **says so** when +it does, without stopping the build. Every ownership and permission call whose +failure changes what the system looks like goes through it: the wrappers' +own root:root, the install directories' group-writable bit, the staging tree's +inherited owner and mode, `passwd`/`group` backups, the `lfs` build user, and +the mode `verify --fix` counts as fixed. + +Fifteen call sites. `2>/dev/null || true` is not banned — `mkdir -p` on a +directory that exists is fine — but an ownership call that keeps it must carry +a comment on the line immediately above saying **SUPPRESSED DELIBERATELY** or +**benign:**, and a test counts the ones that do not. + +Any comment was not enough as a rule. The surrounding prose explains what a +call does, not why its failure may be thrown away, so a reintroduced +suppression inherits whatever comment happened to be above it — the first +version of the test passed a mutation that put one straight back. + +The two deliberate ones are `mkstate` and `ensure_pkgusr_roots`: both run +before book 7.6, when the `install` group does not exist and the call cannot +succeed. Reporting them would print a failure on every invocation for the +first third of a build, and that is how a report becomes noise. + +## The build says what it did not do (1.11.4) + +`strip` is collected by the interview, carried into the chroot environment as +`LFS_STRIP`, and acted on by nothing. `_warn_if_strip_was_asked_for` is now the +build's second-to-last word, beside `_warn_if_no_login` — the place where the +things you still have to do yourself are collected. It fires only if you asked. + +Still not implemented, and the reason stands: stripping rewrites every binary, +and here a file's owner is the record of which package installed it, so it has +to run as that owner, package by package. + +## One door per decision (1.11.3) + +Item 6 does not need a rewrite. `grant_dir_to_user` already showed the shape: +**ask `lfs-helper` when it is on the system, keep the local copy as the +fallback for when it is not.** Three more decisions now go through it. + +**The wrappers.** `packagemanager_install` shipped its own 480-line copy and +wrote it to `/tmp/pkgusr-wrappers.$$` on every run. The wrappers *are* the rule +about what a package user may do — chown skipped, chgrp skipped, `install -d` +on an existing directory allowed to succeed — so that was a second answer to +the question, and `/etc/pkgusr/bash_profile` pointed at neither copy: it names +`/usr/lib/pkgusr`, which is where lfs-helper writes them. Now +`_wrapper_dir_from_lfs_helper` asks, and writes them through +`lfs-helper make-wrappers` if they are missing or incomplete — which also +repairs a tree whose wrappers predate a fix. The local copy stays, for a system +with no lfs-helper. + +**Only a temporary directory is deleted.** Both call sites ended with +`rm -rf "$_wrapdir"`. Pointed at the shared directory that removes the wrappers +from every package user on the system, so the cleanup is now conditional on +having built a temporary one. + +**The account home.** `pkgusr_home_for` re-derived the layout — root per kind, +prefix, `cfg_` special case. Correct today, wrong the moment the layout is +configured differently, which it can be. It asks `lfs-helper pkgusr-home` now. + +**site-packages.** `pip_install_module` did its own `groupadd` + `chgrp -R` + +`chmod -R g+w`: a third implementation, and the only one that never asked what +group the directory already carried. It goes through `lfs-helper grant-dir`. +That is the debt 1.11.2 left. + +Two new commands make the doors reachable: `lfs-helper wrapper-dir` and +`lfs-helper make-wrappers --run`. + +**The dispatch repeated six entries** — `verify`, `install-as`, `pkgusr-info`, +`pkgusr-home`, `owner-name`, `clean-state` — and `case` takes the first match, +so the second set was unreachable code that looked like a second decision. A +test now fails on any repeat. + +What is left of item 6 is the build loop itself: `cmd_build` and +`packagemanager_install` still each drive unpack/build/install/configure. That +is where staging (item 4) lives, and it is the part that needs a real build to +prove. + +## packagemanager setup installs, not just declares (1.11.2) + +`--run` was a stub. It now does the bootstrap half of the old final step, and +`--sources` is unchanged, so `get-sources` still asks it the same question. + +- **wget** comes from `packagemanager install wget --run`, invoked as a + command. A build plan, a script, a package user and a manifest — there is no + "just the wget part" of that which is not a second implementation of it. +- **The wheels** go through `pip_install_module`, extracted from `cmd_pip` so + both commands run the same code. `--no-deps`, in the order `_SETUP_WHEELS` + lists them, `--no-index --find-links /sources`: a fresh system has no CA + certificates, so a pip index lookup is a hang rather than an error. +- **Idempotent.** It reads `<dist>-<version>.dist-info` in site-packages — + what pip itself writes, so the check is an identity rather than a guess — + and skips what is there. `--force` redoes it. +- **It refuses** when the wheels are not on disk, naming each one and saying + they came from `get-sources` before the reboot. + +**A bug fixed on the way in.** `cmd_pip` took `_sanitise_user_name(mod)` and +used it everywhere. `create_package_user` normalises internally, so the account +it created was `p_requests` while `usermod` and `su` were handed `requests`: + + usermod: user 'requests' does not exist + +Nobody hit it because nothing called `packagemanager pip` on a real system. +Same species as `chown: invalid user: 'wget:wget'` — half the code asking for +the name, half assembling it. + +Still owed: site-packages access is granted by `chgrp -R`/`chmod -R g+w` inside +`pip_install_module` rather than by `lfs-helper grant-dir`. Correct group, +right result, wrong door — fold it in with item 6. + +## last-step is cleaned out of the tree (1.11.1) + +`last_build_step.sh` stopped existing in 1.10.0, but three things still spoke +as if it did: + +- **`make install` copied it into `/etc/lfs`** and aborted, because the file is + not in the repo. Installing was broken and nothing said so. +- **`build-all` sealed the install directories early** when the step name was + `last-step` — dead since the final step became `init-accounts`, which runs as + root and installs nothing. The seal after the loop was already doing the work. +- **Three tests looked for the file beside `lfs`** and skipped silently when it + was absent, which is every run since 1.10.0. + +The name stays in `_is_not_a_package`'s match: a tree generated before 1.10.0 +still carries it in `steporder` and in its manifests, and the rule must still +say no. It is marked as legacy there. + +`make clean` exists now, and `make install` no longer hides a failed test +install behind `2>/dev/null || true` — the first of item 10. + +## The build's last word is whether you can log in (1.11.0) + +`init-accounts` asks for a root password, but it is skippable — a scripted run +has no terminal, and an interactive one can be answered with a blank line. So +`_warn_if_no_login` is the **last thing `build-all` does**, because it is the +last moment it can still be fixed. After the reboot the chroot is gone and the +way back is booting the host and mounting the tree by hand. + +It checks the **result**, not that the step ran: `/etc/shadow` for a real root +password, or any account between uid 1000 and 9999 with one. + + !! NOBODY CAN LOG INTO THIS SYSTEM. + Fix it NOW, while you are still in here: + lfs-helper init-accounts --force + +`lfs-helper check-login` runs it on its own. + +## wget belongs to packagemanager setup (1.11.0) + +`packagemanager setup --sources` prints every URL setup needs, and +`lfs build-system get-sources` asks it. **One list, owned by the tool that +installs from it.** + +The first version of this had `bootstrap_packages` in `lfs`'s own config, +resolving wget from the BLFS book separately from the tool that installs it — +a second thing to go stale, which is the mistake this project keeps making. +That key is gone. + +`setup` itself is a stub: `--sources` works, `--run` says it is not implemented +and exits non-zero. That is deliberate — `get-sources` needs the list *now*, so +a fresh build still downloads what the finished system will need. + +## A command that works in silence looks like one that hung (1.10.3) + +`lfs-helper verify` ran for five minutes printing nothing, on a tree it had +already verified clean, and was reported as a hang. It was not. Two faults: + +**A fork per path, twice over.** `stat -c %U` for each path, and +`is_never_claimed` running `< <(never_claim_list)` — a process substitution — +for each path too. Across 66265 manifest paths that is ~130000 processes. + +- `_never_claim_load` reads the list once into `_NEVER_CLAIM`; matching is then + pure shell. +- Owners come from **one batched `stat` per manifest** via `xargs -0`. + Measured on 8000 paths: **12.6s → 0.15s**. + +**No output until every pass had finished.** Each pass now announces itself +before it runs, and the long one prints `[n/m] <package>` live. + +Two traps worth remembering, both of which produced code that was *correct and +slow* — the hardest kind of wrong to notice: + +- **`stat` does not expand escapes.** `stat -c '%U\t%n'` prints a literal + backslash-t, so every line came back as one field, the map stayed empty, and + the loop fell through to the per-path `stat` it was written to replace. Use a + real tab (`$'\t'`). +- **I first pasted the batching into `cmd_build`**, not the verify pass — both + contain `for man in ...` over manifests. The tests caught it. + +## Scratch, and the one list that names it (1.10.2) + +`scan_prune_paths` / `scan_prune_set` name everything no ownership scan should +walk: `/dev /proc /sys /run /tmp`, `$LFS/sources`, `$BUILD_ROOT`, `$STATE`, and +**every package's unpacked source tree** (`<home>/$PKGUSR_BUILD_SUBDIR`). + +There were five such lists and they had already diverged. Build trees moved from +`/build` into each package user's home in 1.7.6; the snapshot scans followed and +the ownership scans did not. So `lfs-helper verify` began walking every unpacked +source tree in the system — minutes of silence — and a finished tree filled the +sanity report with tcl's own documentation: + + !! files with no owner (first 40): + /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm + +A tarball can carry any uid it likes. Unpacked sources are not installed, +nothing owns them, and asking who does has no answer. + +**`SCAN_PRUNE` is an ARRAY, not a string spliced through `eval`.** The patterns +contain `*`, and `eval` lets the shell expand them against the real filesystem +before `find` ever sees them — `/usr/src/pkgusr/*/src/*` becomes a handful of +literal directory names and the prune silently matches almost nothing. That is +what the first version of this fix did, and it looked like it worked. + +`lfs-sanity.sh` prunes the same paths, or it reports what `verify` does not. + +Also: a config step that writes **root-owned** files (`cfg_clock` → `/etc/adjtime`) +is not an orphaned manifest. It installs no software, so it has no account, and +there is nothing adoption could ever want. Eleven such findings on a perfect +tree. + +## The tree remembers which version generated it (1.10.1) + +`gen-chroot-scripts` writes `progress/generated-by` with the version, build id +and date, and deletes scripts for steps no longer in the order. +`_warn_if_scripts_are_stale` compares that against `lfs-helper`'s own version +before `build-all` runs anything. + +**The step order and the scripts live in the TREE; the tools live outside it.** +Nothing connected the two, so a tree generated before 1.10.0 kept running +`last-step` — a step 1.10.0 had removed entirely — under an `lfs-helper` that no +longer knew what it was. The chroot *tools* have carried a build id for exactly +this reason since 1.6; the generated scripts did not. + +Removing a step from the tools is therefore **two** changes: the code, and a +regeneration in every existing tree. The stamp is what makes the second one +visible instead of silent. + +## last_build_step.sh is gone (1.10.0) + +It was a script created once on the HOST and then owned by the person, run last +inside the chroot. Wrong home for everything it held: + +- a fix to it reached nobody who already had a copy — that is how + `chown: invalid user: 'wget:wget'` survived three releases; +- bootstrapping the tooling is `packagemanager setup`'s job; +- setting the root password is not a matter of taste. Book 8.5 says to do it. + +Split three ways: + +| What | Where now | +|---|---| +| root password, login account | **`init-accounts`**, a built-in step in `lfs-helper`, ordered last | +| wget, the Python wheels, the wgetrc weakening | **`packagemanager setup`** — not written yet | +| the `SOURCES` array | **`bootstrap_packages`**, resolved from the BLFS book | + +`init-accounts` could not move to `packagemanager setup`: setup runs on the +**booted** system, and you need to log in to run it. + +`bootstrap_packages` defaults to `wget` and its URL comes from `blfs sources`, +which runs on the host with its own book store. One place knows where wget +comes from, and it is the same place `packagemanager` will ask when it installs +it. Set it empty to turn it off. + +**Losing the "add your own steps here" hook is the real cost.** If it comes +back it should be a directory of drop-in scripts, not a template copied once +that then drifts from the tool that generated it. + +## packagemanager setup: what it owes + +Three guarantees left the suite with `last_build_step.sh` and belong to `setup`. +They are requirements, not history: + +1. **Each module installs as its OWN package user, from a WHEEL.** A modern + sdist needs its build backend, and with no index reachable pip cannot fetch + one: `BackendUnavailable: Cannot import 'hatchling.build'`. +2. **site-packages access goes through the normal collector-group rules** — + join the group that owns it, or ask which group should. Handing + site-packages to the `install` group would mean "anyone may install here", + which is what collector groups exist to avoid. +3. **Never rebuild an account name or home by hand** — `lfs-helper owner-name` + and `lfs-helper pkgusr-home`. This is the bug that started all of it. + +Also: wget needs `check_certificate = off` in `/etc/wgetrc` with the +`lfs-temporary-no-verify` marker, because a fresh system has no CA certificates +until BLFS `make-ca` — and you need wget to fetch make-ca. + +## Things that belong to nobody + +`never_claim_list` — files no package may own, however many write to them: + +- `/usr/share/info/dir` — rewritten by every package that ships an info page +- `/etc/ld.so.cache`, `/var/cache/ldconfig/aux-cache` — regenerated by ldconfig +- **the user database** and Shadow's backups (`passwd-`, `group-`, `.pwd.lock`) + — every `cmd_add_user` during a build touches them, so whichever package + happened to be building swept them up. ncurses did not install `/etc/group-`; + useradd did, while ncurses was building. A package owning the file that says + which packages exist is the one ownership mistake with no way back. +- the wrapper directory — `root:root` **by design**, so a package user cannot + rewrite the rule that constrains it + +- **mail spools** (`/var/mail/<name>`) — Shadow creates one for every account + it makes, so they arrive as a side effect of `useradd` + +Recording any of them produces a conflict no repair can settle: the next +package takes the file straight back. + +**A repair must not create the next run's finding.** Two passes disagreed about +`/var/mail/lfs` and each undid the other — the build-user pass gave it to root, +the unclaimed pass then reported that root holds a file no manifest claims. +`_vfy_build_user_leftovers` consults `is_never_claimed` for exactly this reason, +and there is a test on the property, not just on the one path. + +## The wrappers + +`make_wrappers` ships `chown`, `chgrp`, `chmod`, `cp`, `install`, `mkdir` in +`/usr/lib/pkgusr`, so a **package user** cannot change ownership. They are +ordinary files on `$PATH`, and if that directory is ever ahead of `/usr/bin` +for **root**, root's own privileged calls resolve to them too — and they report +success while doing nothing. + +So every privileged ownership call goes through `real_chown` / `real_chgrp` / +`real_chmod`, which resolve the binary by absolute path via `_real_tool`. +**Ownership is root's business; no `$PATH` may come between the decision and +the filesystem.** `make_wrappers` itself deliberately uses plain `chmod` — it +is setting the mode on files it just wrote, and must stay self-contained. + +## Mechanisms that are not obvious from the code + +**Staged installs.** `make install DESTDIR=<stage>`, then each file is copied +beside its destination and `rename()`d over it. Never `cp -a` over a live +system: that truncates in-use shared libraries and kills running processes. +Only for `gcc glibc binutils bash coreutils`, and only for a lone +`--phase install` — see open item 4. + +**Earlier stages.** A package is built more than once (ncurses in chapter 6 as +`lfs`, `python-tmp` in chapter 7 as root, then the real one as its package +user). Before building X, `_claim_earlier_stages` hands X every path recorded +under X, X-tmp, X-pass1, X-pass2 — matching case-insensitively, because book +titles are capitalised and tarball names are not. + +**Adoption.** Chapter 5-6 packages get their users from the manifests. This +cannot run before book 7.6 creates `/etc/passwd`; it happens at the ownership +epoch (`init-ownership`), which is the one place that decides. Walks manifests in build order so uids read as the +build order. `adopted.list` records `name size` per package so it runs once. + +**verify.** Ownership is derived state: manifests + install-dir list + +collector map determine what the tree should look like. `lfs-helper verify +[--fix]` reconciles actual against expected. + +**Build ids.** Each tool prints an md5 of its own contents. Compare host and +chroot — a mismatch means the chroot is running old code, which wasted a lot of +debugging time. + +**Section resolution.** `_resolve_section` matches a book section by id **or** +title. Some sub-sections carry ids the book toolchain generated +(`idm139921653990176` = 8.5.2.1 Adding nsswitch.conf) which change on every +rebuild of the book. An ambiguous title fragment is refused, not guessed at. + +**Sections that ship a file.** 9.6.8 prints `/etc/sysconfig/rc.site` as a +`<pre class="auto">` block with no commands at all. `_auto_file_block` extracts +it into a quoted heredoc. Every line is commented; install it anyway — it +documents the boot-script knobs where the boot scripts look. + +**Steps that are not packages.** `last-step`, `init-*` and `refind` never get +a user. `cfg_*` is decided by READING the generated script: a versioned book +section produces a phased build with `pkg_glob`, prose produces a plain root +script. Book 9.2 is `LFS-Bootscripts-20250827`, a real package. + + +## Sanity script + +`lfs-sanity.sh` — read-only, run inside the chroot as root, writes one file: + +```sh +./lfs-sanity.sh # -> /tmp/lfs-sanity.txt +``` + +Eight sections: tool build ids, shared-directory modes (flags early sealing), +uid/gid collisions, unprefixed accounts, home ownership, files with no owner, +groups, and root-owned counts. Each says what a hit MEANS, so a report reads on +its own. + +Section 1 **asserts** the owner and group it prints, not just the mode — a tree +ran 34 packages with every install directory at `root:root 775` while the +report said nothing, because that column was decoration. + +Section 3 flags an account wearing **two** prefixes (`p_cfg_*`, `p_u_*`, +`p_p_*`). Nothing creates those any more, so one appearing means a tool is +still applying the package prefix regardless of kind. `tmp_` and `init_` are no +longer allowed either: there is no such kind of account. + + +## Watch on the next build + +(Written before the 1.9-era rebuild; that build and a 1.11.6 build have both +completed since, and items here that they exercised are confirmed good. Kept +because a FRESH tree on a new machine walks the same order.) + +1. **`ls /usr/src`** should show `pkgusr/ cfg/ lfs-pkgusr/` and nothing loose. +2. **`init-ownership` appears in `lfs-helper list`** as step 3, and turns + `built` right after `init-files`. If it says `pending` afterwards, the + checkpoint ran but did not mark it. +3. **`# 27 install directories are now root:install`** — not `0`. Zero is now + an error, but see it green rather than trusting it. +4. **`ls -ald /usr/src/pkgusr/p_*` after the first few packages** — each home + owned by its own user, not `root:root`. +5. **`/usr/src/pkgusr/p_gettext/src`** exists and is a directory, not a + symlink. +6. **The first config step** (`cfg_nsswitch`) comes out as `cfg_nsswitch` + under `/usr/src/cfg`, with no `p_` on it. +7. **A collector-group prompt** offers `<prefix>_perl`, never `<prefix>_p_perl`. +8. **`lfs-helper verify`** at the end: expect nothing, or a stale manifest + entry for a file two packages both install (`/usr/bin/hostname`). + +Then `bash lfs-sanity.sh` and read the `!!` lines. + +## Open items + +**Start here (as of 1.12.0).** The plan "one build loop, four tools" is +COMPLETE: a full fresh build proved the merged loop, and the fifth tool is +retired. What remains is smaller: boot-test the fresh system, prove +`packagemanager update` (--phase update) on one real package, the slice-noise +sweep (item 11), and a verify + snapshot baseline of the finished system. Alongside it: + +- **SysV branch, from the merged session**: booting a finished 13.0-sysv + system is untested (the build reached chapter 8 + BLFS bootstrap). And the + BLFS default-flavour issue below is now doubly relevant — BLFS must not + pick a systemd-flavour default for a SysV system. +- **The live system has ~29 stale packages** against 13.0-systemd, in book + order, with glibc/binutils behind `--toolchain`. `packagemanager update + --reinstall` plans the whole system. Nothing large has been rebuilt yet; + m4 is the only package that has gone through the merged runner. +- **Slice noise (item 11)** still prints `soft: command not found` and + `unbound variable` from sliced functions; every affected test passes + anyway, which is the trap this project has been bitten by repeatedly. + +1. ~~`packagemanager setup`~~ — **done in 1.11.2.** What is left of it is the + grant-dir door, folded into item 6. + +2. ~~`strip` is asked for but does nothing~~ — **done in 1.11.27** (see the + section above). What remains is watching the first `--run` on a real tree: + run `lfs-helper verify` right after it. + +3. **Steps 88-99 have never completed.** Steps 1-87 are well proven — the + sources/build split ran across all of them. The last twelve are all `cfg_*` + steps, and the fixes to them are unit-tested but have never run against the + real book. + +4. **Staging is limited to `--phase install`,** so `build-all` does not stage + gcc or glibc. `--phase all` runs the book's post-install commands in the + same script, and those touch live paths (`mv /usr/bin/chroot /usr/sbin`), + which fails with DESTDIR set. Only bites on rebuilds of a *running* system. + Deliberately deferred to item 6 — fixing it separately means restructuring + the build loop twice. + +5. **Root password** — the one decision the interview cannot hold; a plaintext + config would be worse. Handled as far as it can be: `init-accounts` prompts, + and `_warn_if_no_login` checks the *result* as build-all's last word. A + non-interactive run therefore finishes with a system nobody can log into, + loudly. Nothing further is planned. + +6. **The build loop is still two implementations — and the endpoint is four + tools, not five.** Everything else went through one door in 1.11.3 — + wrappers, account home, collector grants — and what remains is the loop + itself: `cmd_build` and `packagemanager_install` each drive + unpack/build/install/configure. Item 4 (staging) lives inside it. Needs a + real build to prove, so it is not a desk change. + + `packagemanager_install` is absorbed by the same work: its 990 lines are + almost all duplicates of lfs-helper functions, and its one distinctive + part (`u_` application users) is a third implementation of what + `packagemanager user create` already does in python. + + **There is now a written plan: "Plan: one build loop, four tools" near the + end of this file.** Read that before touching either driver. + + The original framing, still true of the loop: Nearly every bug this project has had was the same species: one + decision held in several places, free to disagree — `cfg_bootscripts`, + `adopted.list`, the package-user prefix, the `/sources` permissions, the + account's kind. `install-as` is the entry point that makes unifying them + possible. Highest value, highest risk. + +7. **`migrate-layout`** — layout is configurable but only takes effect on a + fresh build. Moving an existing system's users is not written, and the + naming split widened the gap again: an existing tree holds + `p_cfg_bootscripts` accounts and directories. The tools *read* those + correctly — `pkgusr_kind` strips the package prefix before it looks, so a + legacy name resolves to the right kind and root — but nothing renames the + account or moves the directory. A fresh build is still the answer. + +8. ~~`README.md` is stale~~ — done, and kept current. Build ids are + deliberately not in it; they are in "Current state" above, which is the + file that gets handed over. + +9. **`lfs-sanity.sh` still reads the old state path on an old tree.** It + takes `LFS_PKGUSR_DIR`, so it is right on anything built by 1.9.0, but it + has no fallback for a tree built before the move. Harmless given that no + tree is migrated; worth deleting the ambiguity if migration is ever written. + +10. ~~The `2>/dev/null || true` sweep~~ — **done in 1.11.4** for ownership and + permission calls, with `soft` and a test on the marker. `packagemanager_install` + still has seven, all benign (chmod on a scratch directory, `disown` on a + backgrounded mandb); they have not been reviewed one by one. + +11. **Slice noise: tests that shout but still pass.** A real-system run + prints dozens of `soft: command not found`, `WRAPPERS: unbound variable`, + `_NEVER_CLAIM: unbound variable`, `pkgusr_prefix_for: command not found` + from sliced functions run without their dependencies. Every affected test + currently passes anyway — which is exactly the "empty result is not a + pass" trap this project has been bitten by twice. The sweep: each slice + should stub or slice its dependencies, and the suite should treat an + unbound-variable death inside a slice as a failure of the slice, not + silence. About ten distinct sites. + + +## Useful commands + +```sh +lfs build-system verify # what state is this tree in? +lfs build-system run # continue from wherever it stopped +lfs snapshot save <n> --run # records the toolchain state in metadata +lfs-helper verify --fix # reconcile ownership +lfs-helper which-package <p> # which package installed this +lfs-helper install-as <n> -- <cmd> # install anything as its own package user +lfs-helper pkgusr-info # where each package came from +``` + + +## Tests + +```sh +bash test_lfs_crosschain.sh ./lfs # 644 counted results, 640 pass bare +``` + +Counts are kept in files, not shell variables, so subshell increments survive. +A few Python blocks print several PASS lines but count once. + +"Nothing can fail silently" was not true: a test whose helper died before its +first assertion reported PASS. Use `_slice_fn` to lift a function out of a +script — never `sed -n "/^fn() {/,/^}/p"`, which cannot handle a one-liner. + +**Slice the whole function, never a fixed window.** Several tests used +`grep -B 14` or `sed -n '/marker/,+6p'` around a message. Adding a branch above +the marker pushed what they were asserting out of the window, and they failed +without anything being wrong. Use `_slice_fn`. + +**Do not define a function inside another one.** `_package_already_owns_files` +was first written inside `cmd_build`. bash accepts it -- and it is then defined +only after `cmd_build` has run once -- but every `sed -n '/^cmd_build() {/,/^}/p'` +in the suite stopped at the inner closing brace, silently truncating what those +tests examined. + +**Match invocations, not message text.** Several tests assert that a function +no longer calls something. The comment explaining *why* it no longer calls it +names the thing, so a naive grep matches the explanation and the test can never +pass. Strip comments first, or anchor on the call syntax. + +Run it from a directory holding ALL five scripts, plus `lfs-sanity.sh` and +`skel-u_xdg/.bash_profile` — about 100 tests silently skip if the others are +not beside `lfs`. + + +## Working style that worked + +- Diagnose from evidence in the output, not from the most recent theory. +- Every bug fixed gets a regression test with a comment naming what broke. +- Verify a new test **fails on the old code**. A test that cannot fail is + worse than no test. One nearly shipped because its slice boundary was wrong + and it reported four false failures; another shipped and sat green for + months, its four assertions never reached, because `sed -n "/^fn() {/,/^}/p"` + cannot slice a one-line function and the over-captured text died under + `set -u` before the first check. If a test has never been seen red, it has + not been tested. Use `_slice_fn`. +- **An empty result is not a pass.** Both of those failed the same way: a + helper was missing, the output was empty, and empty read as "no findings". + Assert the helper exists before trusting that it found nothing. +- Assert intent, not implementation. Three tests broke on the rename because + they grepped for literal paths rather than the property that mattered. +- Fix causes, not symptoms. Remove workarounds once the cause is fixed. +- Say plainly when something is uncertain or untested. +- **Before any restructuring, diff the command list before and after.** A + deleted command passes every test that does not mention it — that is how + `adopt-existing`, `sort-users` and `seal-install-dirs` silently disappeared. + + +## Plan: one build loop, four tools (item 6) + +Written after the "0 of 97 judged" bug (1.11.36/37) made the cost concrete, +then retargeted once the inventory showed where it actually ends: **the +endpoint is four tools, not five.** Nothing below is implemented yet. + +### Why now + +The split is no longer theoretical. `cmd_build` and `packagemanager_install` +each drive unpack/build/install/configure, and they diverge where it counts: + +| | lfs-helper `cmd_build` | `packagemanager_install` | +|---|---|---| +| phase tracking, resumable | yes | no | +| staging / DESTDIR (item 4) | yes | effectively no | +| install record | `$MANIFESTS/<n>.files` | `~/pkg.lst` | +| version record | `VERSION` | `install_last` | + +The last two rows produced the bug: two records of one fact, and each tool +blind to the other's. 1.11.37 bridged the *reading* side. This plan removes +the split. + +### Where it ends: packagemanager_install is absorbed + +990 lines, and almost none of it is uniquely its own: + +| what it does | already exists in | +|---|---| +| build wrappers | `lfs-helper cmd_make_wrappers` | +| create the package user | `lfs-helper cmd_add_user` | +| collector groups | `lfs-helper`'s allocator | +| grant a directory, retry | `lfs-helper cmd_grant_dir` | +| diagnose a failed install | `lfs-helper` | +| stage source from /sources | `lfs-helper` | +| `u_` app users, PAM `su_u_user` | `packagemanager user create` (python) | + +That last row is the sharpest: its one distinctive feature is a THIRD +implementation of application-user setup. And `packagemanager`'s own header +has carried the question since the beginning — *"keep packagemanager_install +(bash) as the privileged engine, or fold it in?"* It is answered here: fold +it in. Steps 3-5 below already hollow it out; what remains afterwards is a +shim, so the plan says so up front rather than leaving a fifth tool standing +because nobody named the endpoint. + +Two things this must not lose: + +- **The privilege boundary.** It is the root-side engine: it runs as root and + does the `su -` into the package user. That boundary moves to `lfs-helper` + (already root-only, `need_root`) deliberately — it must not dissolve into + the python tool, which runs unprivileged for most commands. +- **The entry point.** `packagemanager_install <user> <script>` is documented, + printed in this project's own hints, and people run it by hand. It survives + as a COMPATIBILITY SHIM that delegates to `lfs-helper build`, for at least + one full release, with a notice. Only then is it removed. + +### The one thing to get right first: the install record + +Two recorders exist because they answer the same question by different means, +and each is the only one that works in its own window: + +- **Scan by timestamp** (`find -newer $stamp`) — the BUILD-STAGE recorder. + Early in the LFS build a package's files are not yet owned by its package + user (ownership is established later), so an ownership-based listing returns + nothing. This variant must stay. It is also the only one that catches a + config step that rewrites an existing file without installing a new one. +- **Scan by owner** (`list_package`, i.e. `find -user`) — the DEFAULT + everywhere after the build stage. Authoritative once ownership holds, and + self-correcting: it reflects the tree as it is now, not as some past run + recorded it. + +So: **one recorder function, two strategies, one explicit predicate.** + + record_install <name> + if ownership_established_for <name> -> owner scan (pkg.lst) + else -> timestamp scan (manifest) + then: reconcile, so both files exist and agree + +`ownership_established_for` must be ONE named function, not an inline test +repeated in two drivers — that is the shape of every bug this project has +had. Candidate source of truth: whatever `establish-ownership` records, plus +the existing `pkgusr_ready`. + +Rules the reconcile step must satisfy: + +- After the build stage, `pkg.lst` is the record that gets read; the manifest + is kept as the build-stage artifact and for cleanup (it holds *touched* + files, which the owner scan cannot know). +- A package built in the build stage and later owned must end up with a + `pkg.lst` without being rebuilt — the reconcile runs at `establish- + ownership` time for every package, once. +- Neither file is ever the only copy of a fact the other needs. + +### Same treatment for the version record + +`VERSION` and `install_last` both say what was installed. One writer, both +files, or one file both read — decided when the loop is merged, not before. +1.11.37's read-both fallback comes OUT once this lands (fix causes, remove +workarounds). + +### Steps, each shippable and testable on its own + +1. **Extract the recorder.** `record_install` + `ownership_established_for`, + both drivers call it. No behaviour change intended; the test asserts both + files exist and agree after a build in each stage. *This step alone fixes + the class of bug that started this.* +2. **Extract the version writer.** One function, called from both. Remove the + 1.11.37 read-fallback in the same release, with its test inverted. +3. **Extract the runner.** The `su - … bash install_<n> <phase>` invocation, + wrappers, `pipefail`/`tee`, and the exit-code capture — one function, in + lfs-helper, where root already lives. Both drivers spell this differently + today; that is where "no wrappers" bugs come from. +4. ~~Move phase tracking into the shared runner~~ -- **done in 1.11.55.** + run_phase_as takes the record key; cmd_build's private clear/record/infer + copies are deleted; `run-script` derives the key from --user (--name + overrides, --name '' disables). Old-style scripts are never tracked: + `bash script unpack` on one runs EVERYTHING, so a record of "unpack" + would lie. A packagemanager install that dies mid-phase now leaves a + record `lfs-helper build <n> --phase <next>` can resume from. +5. ~~Delegate~~ -- **done in 1.11.57.** cmd_build takes `--script <path>` + (the loop, for a script that is not in $SCRIPTS), normalises its name + argument through unprefix_pkg_user so records land under the step name + whichever form the caller passed, and treats `--phase update` as a full + run (stale-source clean, finalise, no false "earlier phases" hint; + staging still never fires for it). The engine's installPkg hands its + prepared script over for every phased script when lfs-helper is + installed; its private loop survives only for nimgnu_/u_ accounts, + old-style scripts, and lfs-helper-less systems. The engine's after-care + (mandb refresh) stays in the front-matter; record-install/VERSION are the + loop's. +6. ~~Staging applies to both~~ -- **true by construction since 1.11.57**: + there is one loop, and _phase_can_stage rules it (lone install phase + only, so all/update never stage). The existing staging tests pass; the + real-system proof rides on the next full build. +7. ~~Retire the fifth tool~~ -- **done in 1.12.0.** See the changelog + entry; the endpoint item 6 named is reached. + +### How it gets proved + +Not by desk-checking. The system now has ~30 genuinely stale packages, which +is the test bed this always needed: + + lfs-helper snapshot create pre-item6 # rollback, first + lfs-helper verify # baseline, keep the output + # pick ONE low-risk package with no dependents mid-tree: less, xz or m4 + packagemanager update less --run + lfs-helper verify # must differ only for 'less' + # then the same package by the other door, to prove they agree: + lfs install --reinstall m4 + +Only after those agree: a batch of five, then the rest. Never glibc, +binutils or gcc until the loop has proven itself on the small ones. + +### What would make this go wrong + +- Doing it as one commit. Seven steps, seven releases, each red-tested. +- Changing the record format while merging the writers. Merge first, change + formats never if avoidable. +- Deleting the fifth tool before the shim release. Someone's notes say + `packagemanager_install p_foo /path/script`; that has to keep working long + enough to be told otherwise. +- Letting the privilege boundary blur. Root-side work stays in lfs-helper, + named and `need_root`-guarded, not sprinkled into the python tool. +- Trusting an empty scan. If `record_install` finds nothing, that is a + failure to report, not a package that installed nothing — the project has + been bitten by empty-as-pass at least three times. + + +## Known limits + +x86_64 only. Kernel and bootloader are left to the user. diff --git a/HANDOFF-sysv-session.md b/HANDOFF-sysv-session.md new file mode 100644 index 0000000..05e1fbb --- /dev/null +++ b/HANDOFF-sysv-session.md @@ -0,0 +1,86 @@ +# Handoff: SysV-on-13.0 session + +Base for everything here was the 1.11.26 files.zip + HANDOFF.md. +Read the DRIFT section before overwriting anything. + +## New tool: lfs-sysvbook 1.0.0 (build 62ebe64) + +LFS 13.0 dropped the SysV edition. This tool grafts the SysV flavour of the +last SysV book (12.4, donor) onto a systemd-only release (target) and stores +the result as a normal book in the lfs store, with wget-list + md5sums +generated offline from the books' own chapter 3. + + lfs fetch 12.4 && lfs fetch 13.0-systemd + lfs-sysvbook make # -> 13.0-sysv + lfs-sysvbook check --book 13.0-sysv # 12 checks, fails loudly + lfs --book 13.0-sysv run + +What make does: replaces Systemd section with 12.4's "Udev from Systemd", +removes D-Bus, inserts Sysklogd + SysVinit after E2fsprogs, swaps chapter 9 +and the fstab wholesale, filters systemd-* accounts out of 7.6's heredocs, +applies the three SysV configure-flag edits (Man-DB, Procps-ng, Util-linux +ch8), relabels /etc/lfs-release and the kernel image to <out>. Transplanted +sections stay pinned at donor versions; everything else builds at target +versions. Section NUMBERS on transplanted pages keep donor numbering -- +cosmetic, build order is by document position. Rules are declarative tables +at the top of the file for future migrations (make --target 13.1-systemd). + +Verified: a full 13.0-sysv build ran into chapter 8 + BLFS bootstrap on the +real system. + +## lfs 1.11.26 -> 1.11.29 (build 42ca92a) + +1. "download the sources" pipeline step: done = every basename of the + CURRENT book's wget-list present in $LFS/sources, not "any tarball + exists". Leftovers from an older book no longer skip the download step. + Shows "download the sources (N missing)". Falls back to the old + heuristic only when no wget-list is cached. +2. Fresh-run confirmation in cmd_bs_run (both `build-system run` and the + `lfs run` alias): mounted tree with no layout yet + tty -> prints the + book and the whole session config once, asks [Y/n/e(dit)]. n stops with + pointers, e runs _prompt_session_config(force=True). Never asks on + resume or non-interactively. +3. _pkg_glob_for: last-resort infix pattern name-*version (SQLite-3510200 + ships as sqlite-autoconf-3510200.tar.gz -- new package in 13.0, broke + unpack). Unpack doc-filter extended with infix spellings + (*-doc-*|*-docs-*|*-html-*|*-man-pages-*|*-tests-*) so the new pattern + cannot grab sqlite-doc-* or systemd-man-pages-*. Unit-tested: sqlite, + systemd, tcl, expect, python all pick the right tarball. + After installing: regenerate scripts (gen-chroot-scripts --run + --overwrite) so the fix reaches the chroot. + +## packagemanager_install 1.11.26 -> 1.11.27 (build d6b1854) + +The "could not write to <dir>" failure box printed a wrong remedy +(add-dir-to-sysgroup with swapped args, prefixed name for script install). +Now: reads the blocking dir's group owner; if it is already a collector +group it prints `packagemanager sysgroup add <group> <user>`; only an +unshared dir gets add-dir-to-sysgroup (correct order: group first) plus the +membership step. `script install` line uses the bare package name. +Hit in the wild: p11-kit vs /usr/share/zsh/site-functions (nimgnu_zsh). + +## Makefile / README.md + +Makefile: lfs-sysvbook added to TOOLS (install/uninstall/check covered). +README.md: tool table row + short "SysV on LFS 13.0+" section. + +## DRIFT -- read before installing + +The user's installed lfs-helper reports 1.11.53; the files.zip copy is +1.11.26. Do NOT overwrite it from this set. Three pending lfs-helper fixes +are in lfs-helper-1.11.53.patchnotes.txt (progress line before the manifest +scan, chown 0:0 instead of root:root pre-7.6, mkdir /tmp early); apply them +to the CURRENT lfs-helper. The scripts-vs-tools version-mismatch warning in +the chroot is this drift, advisory only; it goes away once the version +lines agree again. +Also check `packagemanager_install --version` before overwriting: if it is +newer than 1.11.26, port the remedy fix instead. packagemanager and blfs +were not touched this session. + +## Open items + +- lfs-helper fixes above: pending, need the current 1.11.53 file. +- blfs set-default flavour issue from the old HANDOFF: unresolved, now + doubly relevant (BLFS must not pick a systemd-flavour default). +- Not tested: booting the finished 13.0-sysv system (build reached BLFS + bootstrap; boot untested). diff --git a/HANDOFF.md b/HANDOFF.md index 63269ee..beafb07 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,1123 +1,228 @@ -# lfs-pkgusr — context for a new chat - -Five scripts that build and maintain an LFS/BLFS system where **every package -is owned by its own unprivileged user**. - -Attach: the five scripts (`lfs`, `lfs-helper`, `packagemanager`, -`packagemanager_install`, `blfs`), plus `test_lfs_crosschain.sh`, -`lfs-sanity.sh` and this file. - -**New here? "Start here" below says what to read and in what order.** - -## Current state - -All five tools at **1.11.6**. Build ids — check these match what is installed: - - lfs f1f6164 - lfs-helper 3e99aa1 - packagemanager df7bb26 - blfs c479327 - packagemanager_install 1d6c685 - -589 regression tests, 584 pass in a bare checkout. - -**This is a build candidate.** Everything since 1.11.0 is desk work — no -change has met a real build. The remaining open items need one to make -progress, so the next move is a full run, not more code. The five failures are -environment-only: four need the book HTML beside the tool, one needs -`skel-u_xdg/.bash_profile`. - -**A full 104/104 build has completed on 1.10.1** and verified clean: 66265 -paths owned by the right package, every account prefixed, every home owned by -its user, install directories sealed, `check-toolchain` usable. - -**A full build on 1.9.3 verified clean: 66265 paths owned by the right -package, nothing else outstanding.** The one failure is `the XDG profile is not -shipped`, which looks for `skel-u_xdg/.bash_profile` next to the tool — -environment-only, it passes where the repo is laid out properly. - -**A full 104/104 build has completed** on the 1.8.2 code. Everything since is -either a fix for something that build revealed, or the 1.9.0 cleanup. - -**A fresh build is required and no tree is migrated.** Account names, home -directories, the build scratch and the state directory have all moved. - - make install - lfs build-system restart --run - lfs build-system session --reconfigure # several new questions - lfs run - # inside the chroot: - lfs-helper --version # MUST read 7a7b240 - lfs-helper build-all - -Afterwards `lfs-helper verify` is the real check — it reconciles ownership -across the whole tree. Then `check-toolchain`. - -Kernel and bootloader are still yours to finish. - -## Start here - -If you are picking this up cold, read in this order: - -1. **The core idea** — one paragraph, and everything else follows from it. -2. **Where everything lives** — the layout changed in 1.9.0. -3. **The ownership epoch** — the single moment the scheme turns on. -4. **One bug, twelve times** — every failure this project has had was the same - shape. Knowing the shape is worth more than knowing the twelve. -5. **Open items** — item 1 is the next job. - -## The core idea - -Each package gets a user that owns the files it installs. - -- Every file traces to the package that installed it. -- Removing a package is removing that user's files. -- A broken build cannot overwrite another package's files. - -Directories several packages install into are shared via **collector groups** -(`<prefix>_<owner>`, e.g. `nimgnu_python`). The install group (`install`, gid -9999) means "anyone may install here" and is only for genuinely shared dirs — -never a package's own tree, and never `/root` or `/home`. - -## The five scripts - -| Script | Runs | Purpose | -|---|---|---| -| `lfs` | host, Python | Drives the build: partition, sources, chapters 5-6, chroot, snapshots | -| `lfs-helper` | inside the chroot, **bash** | Builds each package as its user. Bash because there is no Python in the chroot yet | -| `packagemanager` | built system, Python | Installs/updates packages, application users, config | -| `packagemanager_install` | built system, bash | The install engine `packagemanager` calls | -| `blfs` | built system, Python | Parses the BLFS book, generates install scripts, resolves dependencies | - - -## Where everything lives - -Two trees under `/usr/src`: the accounts, and what the build knows about them. - -``` -/usr/src/pkgusr/p_gcc a package's account -/usr/src/pkgusr/p_gcc/src where gcc unpacks and builds -/usr/src/cfg/cfg_bootscripts a config step's account -/usr/src/u_firefox an application user -/usr/src/lfs-pkgusr/ what the build knows about all of them -``` - -`SRCROOT` (`/usr/src`) is the parent and stays whole-tree, **deliberately**: -every ownership pass excludes it as one path (`-not -path "$SRCROOT/*"`). -Splitting that exclusion would silently un-exclude half the accounts. - -That is also why the state directory moved here in 1.9.0. It was -`/.lfs-pkgusr` — a hidden directory at the root of the filesystem, sitting -beside `/boot` and `/etc` as if it were part of the system being built. It is -not; it is this toolchain's working notes *about* that system. Under -`/usr/src` the scans exclude one path instead of two. - -The state directory is sorted, so opening it tells you what is in it: - -| | | -|---|---| -| `scripts/` | one shell script per build step, generated from the book | -| `manifests/` | what each package installed (`.files`, `.dirs`) | -| `logs/` | one log per step, plus `verify.log` | -| `progress/` | `steps-built`, `steporder`, `adopted.list`, tree snapshots, the ownership marker | -| `groups/` | collector groups and the grants that created them | -| `config/` | `installdirs.lst`, `env` — settings you may edit | -| `stage/` `steps/` `tmp/` | transient, only during a build | - -Both tools name it once: `STATE` in `lfs-helper`, `PKGUSR_DIR` + -`pkgusr_state()` in `lfs`. They read each other's manifests, so a disagreement -here is a silent one. - -**Definition order matters.** `SRCROOT` must be set before anything derived -from it. Under `set -u` a forward reference is not a subtle bug — the tool -will not print its own version. There is a test on the ordering. - -## The build scratch is the package's - -A package unpacks and builds in `<its home>/src` — not in a shared `/build` -that every package writes into at once. - -The shared scratch was a permanent ownership fight. Every build's touched-file -scan swept it in, so each manifest claimed every other package's sources and -chowned them; the next package chowned them back. A real `verify` reported -3363 wrong-owner and 3348 of them were `/build`: - - /build/bash-5.3.tar.gz is p_shadow want p_acl - /build/bash-5.3.tar.gz is p_zstd want p_acl (after "repairing" it) - -Under the package user's home it is settled by construction: the user creates -the tree so the user owns it, `$SRCROOT` is already excluded from every scan so -it can never reach a manifest, and removing a package removes its build tree. -Steps with no account keep the shared `/build` — they run as root, which owns -everything there anyway. - -`~/src`, not `~/build`: `~/build` is already the hint's build helper script, -symlinked into every package user's home. `pkgusr_skel_links` names the -skeleton once and a test checks the build subdir is not one of them. - -## Names and places - -Three kinds of account share one passwd file, so each carries a prefix. -Without one, a package called `man` or `news` collides with a real system -account and nothing says which accounts belong to the build. - -| Thing | Looks like | Lives in | -|---|---|---| -| package user | `p_gcc` | `/usr/src/pkgusr/p_gcc` | -| config-step user | `cfg_bootscripts` | `/usr/src/cfg/cfg_bootscripts` | -| application user | `u_firefox` | `/usr/src/u_firefox` | -| collector group | `<prefix>_python` | — | - -**One account, one prefix — its own.** - -| Range | Used for | -|---|---| -| 9998 | `lfs` build user *(inside the chroot; the host's is unpinned)* | -| 9999 | `install` group | -| 10000+ | Package users, in build order (binutils first) | -| 90000+ | Collector groups | - -**Prefixing is idempotent, and that is load-bearing.** It is applied to names -a person typed (`gcc`), to directory names read back from disk (`p_gcc`), and -to values that already went through it. Because a second pass cannot produce -`p_p_gcc`, every call site can apply it without first knowing which kind of -name it holds — which is what made 24 call sites a mechanical change instead -of 24 judgement calls. - -**One chokepoint per tool.** Never build a home by hand: - -- `lfs-helper` — `pkgusr_kind()`, `pkgusr_prefix_for()`, `pkgusr_root_for()`, - `pkgusr_home_for()`, `pkgusr_roots()`, `unprefix_pkg_user()`, - `step_stage_dir()` -- `packagemanager` — `pkgusr_kind()`, `pkgusr_name()`, `unprefix_pkgusr()`, - `pkgusr_root()`, `pkgusr_roots()`, `pkgusr_home()` -- `lfs` — `pkgusr_prefix()`, `cfguser_prefix()`, `LAYOUT_DEFAULTS`, `layout()` - -`pkgusr_kind()` is the top of that chain: it decides the kind once and the -prefix and the root are both answered from it, so a name cannot be given one -kind's prefix and placed in another kind's root. - -`lfs-helper pkgusr-home <name>` exposes the rule to shell scripts, so nothing -else re-derives it. `packagemanager_install` and `lfs-completion.bash` go -through a chokepoint now; a test fails if either builds `/usr/src/$something` -by hand. - -**The chokepoint must resolve to the ACCOUNT home, not just the root.** -Callers pass a mixture and always will: `build_pkg` has the step name -(`man-pages`, `util-linux-tmp`), the staging code has the owner -(`p_man-pages`). `pkg_owner_name` maps one to the other — prefix, lowercase, -fold `-tmp`/`-passN` onto the real package — and it is idempotent. Routing by -kind without that mapping shipped once and broke the build at step 9: - - # created package user p_man-pages ... - line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory - -Chapter 7 did not catch it — root steps `mkdir -p` their own directory first, -so it surfaced at the first real package user. - -**A step that is not a package has no home.** `pkgusr_home_for` answers for -accounts; `step_stage_dir` answers for steps, and routes `init-*`, `last-step`, -`refind` and prose `cfg_*` sections to `$STATE/steps/<name>` because they never -get an account. Asking `pkgusr_home_for` for their home is what left -`/usr/src/pkgusr/p_init-dirs` behind. - -Note `SRCROOT` in `lfs-helper` is the **parent** (`/usr/src`), deliberately. -Tree scans exclude it as one path (`-not -path "$SRCROOT/*"`); splitting it -there would silently un-exclude half the accounts from every ownership pass. - - -## Two directories, two jobs - -| Directory | Job | Permissions | -|---|---|---| -| `$LFS/sources` | downloaded tarballs, never written by a build | `root:root 1777` | -| `<package home>/src` | where that package unpacks and builds | owned by its user | -| `$LFS/build` | scratch for steps with no account | `root:root 1777` | - -Book 3.1 settles the ownership — `chmod a+wt`, then -`chown root:root $LFS/sources/*` — and gives the reason: a host uid means -nothing in the built system. Here it is sharper than the book's case. The host -`lfs` account is created by book 4.3's own `useradd`, which pins no uid, so it -lands wherever the host had a gap — 10753 on a Debian host. Package users start -at 10000, so it does not merely show as a number, it **collides with a real -package user** and tarballs report as owned by something unrelated. - -`/sources` and `/build` are in `never_adopt_list` and NOT in -`install_dirs_list`. Being in both made `/sources` `root:install 775`, -dropping the `o+w` that lets the unprivileged build user unpack at all. - -**The symlink farm.** `unpack_pkg` links every downloaded file into -`BUILD_ROOT`, because the book reaches sibling tarballs through `..`: - - cd .. - tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 - -`cd ..` leaves the build subdirectory, so `..` is then `BUILD_ROOT`. Linking -everything rather than a known list means any future book works without a -second thing to keep in sync. - - -## The ownership epoch - -**Before book 7.6 there is no `/etc/passwd`, so ownership is not wrong — it is -meaningless.** No name resolves, not even root's; the prompt in the chroot -reads `I have no name!`. Every ownership question asked before that point gets -a confident wrong answer: - - chown root:install fails on every directory -> "# 0 install directories" - find -nouser matches every file in the tree -> 11802 "orphans", incl. / and /proc - -There is one gate: - -| | | -|---|---| -| `ownership_established()` | is it done? (`progress/ownership-established`) | -| `ownership_possible()` | can it be? (passwd + chown/chgrp/chmod) | -| `establish_ownership()` | do it | -| `_ownership_checkpoint()` | the one call site, after every step | - -`init-ownership` is a **step you can see**, inserted straight after -`init-files` (book 7.6). `lfs-helper` performs it itself — there is no book -script — and marks it done, so `list` and `next` agree with reality: - - 3. init-ownership built - -It does everything in order: install group and install directories, the build -user's leftovers handed back to root, package users for everything already -built, adoption, our own tools, then orphans. Idempotent, and the marker is -durable, so a resumed or restored build crosses it once. -`lfs-helper establish-ownership` runs it by hand. - -Everything that needs a user database refuses without one: `_vfy_orphans`, -`_has_orphaned_files`, `_vfy_build_user_leftovers`, `_vfy_install_dirs`. - -Install directories are set and read back **by number** (`0:$INSTALL_GID`) — -`root` need not be a name. - -## The two chowns that can destroy a tree - -Book 4.3 gives the tree to the build user; book 7.2 gives it back. Both are -recursive, and each is catastrophic on the wrong side of chapter 7. - -- **4.3 on a built tree** hands the finished system to the host's build - account. `_chown_tree_to_lfs` refuses when `_handover_is_safe` is false. - - There were **four doors to that chown** and only one had the guard: - `_chown_tree_to_lfs` (book 4.3), `session --run` re-applying it by hand, - the pre-build ownership check, and `fix-ownership --run`. All four now ask. - A test walks every `chown -R lfs` in the file and fails on any whose - enclosing function does not consult `_handover_is_safe` — the property, not - the four paths, because a fifth door would otherwise be silent too. -- **7.2 on a built tree** strips every package user in one pass. Refused for - the same reason. - -Refusing both left a tree with no way forward, so there is a third, precise -form: `_handover_build_user_only` selects on the **build user's uid**, taking -back exactly what 4.3 took and leaving every package user's files alone. Safe -at any stage. `chroot prepare` runs it instead of refusing, and `chroot enter` -runs it before deciding to stop. - -`chown -h root`, user only — 4.3 chowns the user, so the inverse does too; -taking `root:root` would strip `install` off every shared directory. And `-h`, -because `/bin`, `/lib` and `/sbin` are symlinks. - -`_handover_is_safe` asks the **tree's own passwd** which accounts are package -users (`_pkg_users_in_tree`), never a uid range. Book 4.3's `useradd` pins no -uid, so the host's `lfs` came out **10753** on one machine — inside the package -range — while the chroot's own `lfs` is 9998. Across that boundary only the -number means anything. - -## Saying things - -Four colours, one meaning each, identical in both tools — they print into the -same terminal one after the other, and a log where the same colour means two -things is worse than no colour at all. - -| | | -|---|---| -| green | something finished, and finished correctly | -| yellow | something needs your attention | -| red | something failed | -| dim | detail you can skip: provenance, and what to do next | - -Prose gets none. Colour on every other line stops carrying information. - -`lfs-helper`: `say` (prose) `ok` `warn` (stderr) `note` (stdout) `detail` -`hint` `fail` `die`. `lfs`: `warn` `note` `ok` `fail` `hint`. - -**Use the helper, never the variable.** A colour opened in one `echo` and -closed in another leaves the terminal painted if anything between them writes — -the failure banner was 22 lines inside a single open/close. The only direct use -left is the status column in `list`, where the colour *is* the information. - -## One bug, twelve times - -Every failure this project has had was the same shape: **one decision held in -more than one place, free to disagree.** They are worth reading as one list, -because the next one will look like them. - -| What disagreed | How it showed up | -|---|---| -| Package prefix applied regardless of kind | `p_cfg_bootscripts` — two prefixes, in the config root, claiming to be a package | -| Two wrapper directories | `make_wrappers` wrote `$STATE/wrappers`; the profile said `/usr/lib/pkgusr`, which nothing created | -| Two copy routines for the tools | `install_helper` claimed ownership, `_install_pkgusr_helpers` did not — eight files owned by nobody | -| `chgrp` vs `chown root:install` | `init-pkgusr` set the group, `verify` demanded owner+group; `/usr/share` sat at `lfs:install` | -| Install-dir rule in three places | `cmd_verify` carried inline copies without the guards the helpers had grown | -| **`cmd_verify` defined twice** | bash keeps the last; the tidy one had never run | -| Scratch scanned by two rules | the snapshot scans excluded `/build`, the build's own scan did not — 3348 false findings | -| Build tree named `build` | collided with `~/build`, the hint's helper script | -| Wrapper `chgrp` shadowing root's | reported 46 successes, changed nothing, and `verify --fix` could not repair it either | -| Handover guarded by one inode | `_owner_is_root` checks `$LFS/usr`; the pass it guards covers eight trees | -| uid range used to identify the build user | host `lfs` was 10753, inside the package range | -| Adoption reached from two directions | an up-front pass and a mid-loop "late" pass with its own flag | -| Ownership scans kept their own exclusion lists | build trees moved into the package homes in 1.7.6, the snapshot scans followed and the ownership scans did not — `verify` ran for minutes walking every unpacked source tree | -| Generated scripts had no version stamp | the step order lives in the tree, the tools outside it — a tree generated by an older version went on running `last-step` under a lfs-helper that had removed it | -| Collector export written and read in two places | 1.9.0 sorted the state dir; the writer stayed at the top level, the reader moved into `groups/` | -| `pkgusr-home` was exposed, `owner-name` was not | a script could ask where a package lives but not what its user is called, so `last_build_step.sh` assumed — `chown: invalid user: 'wget:wget'` at step 104 of 105 | -| `add-user` named the account raw, the home prefixed | `add-user wget` gave account `wget` in `/usr/src/pkgusr/p_wget` — every caller but `last_build_step.sh` passed an already-prefixed name, so it never showed | -| Tree snapshot written and read in two places | same move, same shape: `lfs` kept `.snapshot`, `lfs-helper` read `progress/tree-snapshot` — the chapter 6→7 handover snapshot was lost | - -**Half a chokepoint is not a chokepoint.** `pkgusr_home_for` was reachable from -outside as `lfs-helper pkgusr-home`; `pkg_owner_name` was not. So a shell script -could ask where a package lives but had to *guess* what its user is called — -right by accident until `add-user` started applying the prefix, then wrong at -step 104 of a 105-step build. `lfs-helper owner-name <package>` is the other -half, and a test checks the two agree. - -The fix is always the same: **name the decision once, and make everything ask -that one thing.** `pkgusr_kind()`, `build_root_for()`, `never_claim_list()`, -`ownership_possible()`, `pkgusr_skel_links()`, `_real_tool()` all exist for -this reason. - -Two rules follow from it, and both have tests: - -- **No function may be defined twice.** A duplicate is invisible — both parse, - one silently wins. -- **Every `cmd_*` a tool calls must be defined in that tool.** `verify --fix` - called `cmd_fix_orphans`, which never existed, and died mid-repair after - chowning 3473 paths. - -## The collector export - -A previous build's group decisions, so a rebuild does not ask -"which group should share this directory?" for every package again. - - lfs-helper export-groups # write one out - collector_import_file = <path> # session setting on the host - -`lfs` copies it into `groups/collector-groups.import`; `lfs-helper` reads it -from there in `choose_collector_group`, **before** asking. Those two paths are -the whole mechanism, and nothing else connects them — when the state directory -was sorted in 1.9.0 the writer stayed at the top level while the reader moved, -and a rebuild with a perfectly good export asked about `/usr/lib/bfd-plugins` -anyway. Silent, because a question that gets asked looks exactly like a -question that was never answered. There is a test that the two paths match. - -It is re-copied on **every chroot entry**, so editing it on the host reaches -the next build without regenerating the scripts. It only prints when the -contents changed. - -`session` reports what is in the file (`N directory decision(s), M group(s)`) -rather than echoing a path — a file that was configured, present and never read -looked identical to one that was working. - -## One rule for /usr/src - -Everything under `/usr/src` is `root:install 1775`: `/usr/src` itself, the -account roots, and the state directory. - -The account roots were `root:root 0755` — defensible, since only root creates -an account home, but it made one subtree with two rules and a special case to -remember. **The sticky bit is what makes group-writable safe and is not -optional:** without it any member of `install` could delete or rename another -package's home. With it, only the owner can, so group write grants only the -ability to *create*, which nothing unprivileged does (`add-user` is root-only). - -Sharing the mode is not sharing the meaning. The account roots and the state -directory are still in `never_adopt_list`, so no package may ever take one as -its own, and `install_dirs_list` does not contain them. - -**The state directory is in `never_claim_list`.** It lives under `/usr/src`, -which IS an install directory, so `verify` walked its own manifests, scripts -and logs and reported each as a root-owned file no package claims. A clean -104/104 build came back with `448 root-owned file(s) that NO manifest claims` -and all 448 were this directory — including the two temp files the scan had -just written for itself. - -## Reading a sanity report - -Three of its findings used to be about the report, not the tree. All fixed, but -worth knowing the shape: - -- **The sticky bit means opposite things on either side of the finish line.** - During the build it is a bug — package users cannot replace the temporary - system's files. After `seal-install-dirs` it is the point, and its *absence* - is the bug. The report works out `BUILD_FINISHED` from `progress/steporder` - vs `progress/steps-built` and judges accordingly. -- **An empty manifest is not an orphan.** A prose config step — `cfg_clock`, - `cfg_hosts`, `cfg_locale` — writes into files another package owns, so it - records nothing and legitimately has no account. Fourteen of those were - reported on a perfect tree. -- **A step can install under a name that is not its own.** `libstdcpp`'s files - belong to `p_gcc`, because the book builds it out of the gcc tree. When the - name lookup fails, the report now asks the filesystem who owns the files - before calling it a finding. - -`packagemanager_install` has `--version` now, so section 0 can read all five -tools. A tool you cannot ask "which build are you?" is the one that quietly -goes stale. - -## comm needs one collation - -The snapshot diff is `comm -13 <(before) <(after)`. **comm compares line by -line and trusts both sides to be sorted the same way. It cannot detect -otherwise; it returns the wrong answer.** Two things broke that: - -- the two snapshots may have been written under different locales — a package - user's profile sets `LC_ALL=POSIX`, root's shell may not; -- `load_snapshot` rewrites only the lines carrying the host mount prefix, so a - file holding both forms comes back sorted under neither. - -`snapshot`, `snapshot_dirs` and `load_snapshot` all pin `LC_ALL=C sort`, and -`load_snapshot` re-sorts after the rewrite. The visible symptom is a package -installing forty files and being told it installed none. - -**A re-run over its own work is not a silent failure.** The zero-files check -counts what *this run* wrote, which is the wrong question the second time a -package is built: many installers skip a file that is already up to date -- -perl's ExtUtils::Install prints "Installing <path>" for every target and copies -only the stale ones. So the run writes nothing, the snapshot diff sees nothing -new, no mtime beats the stamp, and the count is 0 for the best possible reason. - -`xml-parser` hit exactly that: forty files on disk, every one owned by -`p_xml-parser`, and the step failed as a no-op. `_package_already_owns_files` -asks the tree instead of the counters. A package owning nothing is still a -failure. - -**"installed NO files" now shows its working.** There are exactly three ways to -reach zero and they need different fixes: nothing was written, it was written -somewhere excluded from tracking, or it was written over files another package -owns and dropped by the ownership filter. The message reports the snapshot-diff -count, the touched count, and which of the three it is. - -## A bootloader is opt-in - -`bootloader` defaults to **none**. It used to default to `refind`, on the -reasoning that rEFInd only ever adds files to an already-mounted ESP — true, -and still the safe choice, but it stopped a finished build to ask about a -bootloader on a machine that already had one. The machine booted in order to -build LFS at all. A bootloader is a decision about the whole machine, not a -package. - - lfs config bootloader refind - lfs config esp /boot/efi - lfs build-system gen-chroot-scripts --run --overwrite - -Skipping it is not silent — `gen-chroot-scripts` says so and prints those three -lines, because someone who *does* need a bootloader would otherwise reach a -finished, unbootable system without ever having been offered one. - -GRUB is unchanged and still withheld unless explicitly chosen: its book -instructions run `grub-install /dev/sda`, which writes a disk's boot sector. - -## A command that works in silence looks like one that hung (1.10.3) - -`lfs-helper verify` ran for five minutes printing nothing, on a tree it had -already verified clean, and was reported as a hang. It was not. Two faults: - -**A fork per path, twice over.** `stat -c %U` for each path, and -`is_never_claimed` running `< <(never_claim_list)` — a process substitution — -for each path too. Across 66265 manifest paths that is ~130000 processes. - -- `_never_claim_load` reads the list once into `_NEVER_CLAIM`; matching is then - pure shell. -- Owners come from **one batched `stat` per manifest** via `xargs -0`. - Measured on 8000 paths: **12.6s → 0.15s**. - -**No output until every pass had finished.** Each pass now announces itself -before it runs, and the long one prints `[n/m] <package>` live. - -Two traps worth remembering, both of which produced code that was *correct and -slow* — the hardest kind of wrong to notice: - -- **`stat` does not expand escapes.** `stat -c '%U\t%n'` prints a literal - backslash-t, so every line came back as one field, the map stayed empty, and - the loop fell through to the per-path `stat` it was written to replace. Use a - real tab (`$'\t'`). -- **I first pasted the batching into `cmd_build`**, not the verify pass — both - contain `for man in ...` over manifests. The tests caught it. - -## Scratch, and the one list that names it (1.10.2) - -`scan_prune_paths` / `scan_prune_set` name everything no ownership scan should -walk: `/dev /proc /sys /run /tmp`, `$LFS/sources`, `$BUILD_ROOT`, `$STATE`, and -**every package's unpacked source tree** (`<home>/$PKGUSR_BUILD_SUBDIR`). - -There were five such lists and they had already diverged. Build trees moved from -`/build` into each package user's home in 1.7.6; the snapshot scans followed and -the ownership scans did not. So `lfs-helper verify` began walking every unpacked -source tree in the system — minutes of silence — and a finished tree filled the -sanity report with tcl's own documentation: - - !! files with no owner (first 40): - /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm - -A tarball can carry any uid it likes. Unpacked sources are not installed, -nothing owns them, and asking who does has no answer. - -**`SCAN_PRUNE` is an ARRAY, not a string spliced through `eval`.** The patterns -contain `*`, and `eval` lets the shell expand them against the real filesystem -before `find` ever sees them — `/usr/src/pkgusr/*/src/*` becomes a handful of -literal directory names and the prune silently matches almost nothing. That is -what the first version of this fix did, and it looked like it worked. - -`lfs-sanity.sh` prunes the same paths, or it reports what `verify` does not. - -Also: a config step that writes **root-owned** files (`cfg_clock` → `/etc/adjtime`) -is not an orphaned manifest. It installs no software, so it has no account, and -there is nothing adoption could ever want. Eleven such findings on a perfect -tree. - -## The tree remembers which version generated it (1.10.1) - -`gen-chroot-scripts` writes `progress/generated-by` with the version, build id -and date, and deletes scripts for steps no longer in the order. -`_warn_if_scripts_are_stale` compares that against `lfs-helper`'s own version -before `build-all` runs anything. - -**The step order and the scripts live in the TREE; the tools live outside it.** -Nothing connected the two, so a tree generated before 1.10.0 kept running -`last-step` — a step 1.10.0 had removed entirely — under an `lfs-helper` that no -longer knew what it was. The chroot *tools* have carried a build id for exactly -this reason since 1.6; the generated scripts did not. - -Removing a step from the tools is therefore **two** changes: the code, and a -regeneration in every existing tree. The stamp is what makes the second one -visible instead of silent. - -## last_build_step.sh is gone (1.10.0) - -It was a script created once on the HOST and then owned by the person, run last -inside the chroot. Wrong home for everything it held: - -- a fix to it reached nobody who already had a copy — that is how - `chown: invalid user: 'wget:wget'` survived three releases; -- bootstrapping the tooling is `packagemanager setup`'s job; -- setting the root password is not a matter of taste. Book 8.5 says to do it. - -Split three ways: - -| What | Where now | -|---|---| -| root password, login account | **`init-accounts`**, a built-in step in `lfs-helper`, ordered last | -| wget, the Python wheels, the wgetrc weakening | **`packagemanager setup`** — not written yet | -| the `SOURCES` array | **`bootstrap_packages`**, resolved from the BLFS book | - -`init-accounts` could not move to `packagemanager setup`: setup runs on the -**booted** system, and you need to log in to run it. - -`bootstrap_packages` defaults to `wget` and its URL comes from `blfs sources`, -which runs on the host with its own book store. One place knows where wget -comes from, and it is the same place `packagemanager` will ask when it installs -it. Set it empty to turn it off. - -**Losing the "add your own steps here" hook is the real cost.** If it comes -back it should be a directory of drop-in scripts, not a template copied once -that then drifts from the tool that generated it. - -## The kernel and the bootloader are yours (1.11.6) - -Both are decisions about the whole machine rather than about a package, and -getting either wrong costs the system doing the building. So the build does -neither, and ends by **saying** so — `_say_what_is_yours_to_finish`, beside the -login and strip warnings — rather than leaving it to be discovered after a -reboot. - -`bootloader` was already `none` by default and stays that way. The prompt now -says what that means: nothing is written to any ESP, boot sector or partition -table, and the machine keeps booting exactly as it does today. rEFInd is opt-in -by name, and the closing note prints how to ask for it. - -## $LFS may never be the running system (1.11.5) - -Every path this tool writes is `$LFS/something`. Nothing checked that the tree -was not the host's own, so a config holding `/`, an `export LFS=/` in the wrong -shell, or a typo in the interview was enough for -`build-system restart --run` to delete the top-level directories of the machine -doing the building, and for `_verify_lfs_ownership` to run `chown -R lfs /usr` -on it. - -`_is_host_path` is the rule; `_refuse_host_tree` is the hard stop. It is called -from `require_lfs_for_root`, the single door every command resolves `$LFS` -through, and from `_verify_lfs_ownership`, which resolved `$LFS` itself and was -therefore the one destructive path with no check at all. The interview refuses -the same paths at the prompt, where you can just type another one. - -**Both spellings are checked.** On a merged-`/usr` system `/bin` is a symlink -to `usr/bin`, so `realpath("/bin")` is `/usr/bin` and a check on the resolved -path alone let `$LFS=/bin` through. The first version of the test caught this. - -**Inside a system tree counts.** `/usr/src/lfs` is not a build tree, it is a -directory in the system's own `/usr`. `/home` and `/srv` are deliberately not -treated that way: `/home/you/lfs` is an ordinary place to build, and a check -that refuses `$LFS/usr` refuses every build there is. - -What was already safe, and stays so: `restart` refuses while the chroot's bind -mounts are up, because deleting through `/dev` would reach the host. GRUB -scripts are not generated at all unless GRUB is the chosen bootloader, so -nothing that could run `grub-install /dev/sda` is left lying around. The rEFInd -step never formats, partitions or writes a raw device; it adds files to an -already-mounted ESP, backs the config up, adds a menu entry beside your -existing bootloader, and refuses to overwrite an existing rEFInd without -`REFIND_OVERWRITE=1`. - -## A suppressed failure is a bug waiting to be found (1.11.4) - -`soft <what> -- <cmd ...>` runs a command that may fail, and **says so** when -it does, without stopping the build. Every ownership and permission call whose -failure changes what the system looks like goes through it: the wrappers' -own root:root, the install directories' group-writable bit, the staging tree's -inherited owner and mode, `passwd`/`group` backups, the `lfs` build user, and -the mode `verify --fix` counts as fixed. - -Fifteen call sites. `2>/dev/null || true` is not banned — `mkdir -p` on a -directory that exists is fine — but an ownership call that keeps it must carry -a comment on the line immediately above saying **SUPPRESSED DELIBERATELY** or -**benign:**, and a test counts the ones that do not. - -Any comment was not enough as a rule. The surrounding prose explains what a -call does, not why its failure may be thrown away, so a reintroduced -suppression inherits whatever comment happened to be above it — the first -version of the test passed a mutation that put one straight back. - -The two deliberate ones are `mkstate` and `ensure_pkgusr_roots`: both run -before book 7.6, when the `install` group does not exist and the call cannot -succeed. Reporting them would print a failure on every invocation for the -first third of a build, and that is how a report becomes noise. - -## The build says what it did not do (1.11.4) - -`strip` is collected by the interview, carried into the chroot environment as -`LFS_STRIP`, and acted on by nothing. `_warn_if_strip_was_asked_for` is now the -build's second-to-last word, beside `_warn_if_no_login` — the place where the -things you still have to do yourself are collected. It fires only if you asked. - -Still not implemented, and the reason stands: stripping rewrites every binary, -and here a file's owner is the record of which package installed it, so it has -to run as that owner, package by package. - -## One door per decision (1.11.3) - -Item 6 does not need a rewrite. `grant_dir_to_user` already showed the shape: -**ask `lfs-helper` when it is on the system, keep the local copy as the -fallback for when it is not.** Three more decisions now go through it. - -**The wrappers.** `packagemanager_install` shipped its own 480-line copy and -wrote it to `/tmp/pkgusr-wrappers.$$` on every run. The wrappers *are* the rule -about what a package user may do — chown skipped, chgrp skipped, `install -d` -on an existing directory allowed to succeed — so that was a second answer to -the question, and `/etc/pkgusr/bash_profile` pointed at neither copy: it names -`/usr/lib/pkgusr`, which is where lfs-helper writes them. Now -`_wrapper_dir_from_lfs_helper` asks, and writes them through -`lfs-helper make-wrappers` if they are missing or incomplete — which also -repairs a tree whose wrappers predate a fix. The local copy stays, for a system -with no lfs-helper. - -**Only a temporary directory is deleted.** Both call sites ended with -`rm -rf "$_wrapdir"`. Pointed at the shared directory that removes the wrappers -from every package user on the system, so the cleanup is now conditional on -having built a temporary one. - -**The account home.** `pkgusr_home_for` re-derived the layout — root per kind, -prefix, `cfg_` special case. Correct today, wrong the moment the layout is -configured differently, which it can be. It asks `lfs-helper pkgusr-home` now. - -**site-packages.** `pip_install_module` did its own `groupadd` + `chgrp -R` + -`chmod -R g+w`: a third implementation, and the only one that never asked what -group the directory already carried. It goes through `lfs-helper grant-dir`. -That is the debt 1.11.2 left. - -Two new commands make the doors reachable: `lfs-helper wrapper-dir` and -`lfs-helper make-wrappers --run`. - -**The dispatch repeated six entries** — `verify`, `install-as`, `pkgusr-info`, -`pkgusr-home`, `owner-name`, `clean-state` — and `case` takes the first match, -so the second set was unreachable code that looked like a second decision. A -test now fails on any repeat. - -What is left of item 6 is the build loop itself: `cmd_build` and -`packagemanager_install` still each drive unpack/build/install/configure. That -is where staging (item 4) lives, and it is the part that needs a real build to -prove. - -## packagemanager setup installs, not just declares (1.11.2) - -`--run` was a stub. It now does the bootstrap half of the old final step, and -`--sources` is unchanged, so `get-sources` still asks it the same question. - -- **wget** comes from `packagemanager install wget --run`, invoked as a - command. A build plan, a script, a package user and a manifest — there is no - "just the wget part" of that which is not a second implementation of it. -- **The wheels** go through `pip_install_module`, extracted from `cmd_pip` so - both commands run the same code. `--no-deps`, in the order `_SETUP_WHEELS` - lists them, `--no-index --find-links /sources`: a fresh system has no CA - certificates, so a pip index lookup is a hang rather than an error. -- **Idempotent.** It reads `<dist>-<version>.dist-info` in site-packages — - what pip itself writes, so the check is an identity rather than a guess — - and skips what is there. `--force` redoes it. -- **It refuses** when the wheels are not on disk, naming each one and saying - they came from `get-sources` before the reboot. - -**A bug fixed on the way in.** `cmd_pip` took `_sanitise_user_name(mod)` and -used it everywhere. `create_package_user` normalises internally, so the account -it created was `p_requests` while `usermod` and `su` were handed `requests`: - - usermod: user 'requests' does not exist - -Nobody hit it because nothing called `packagemanager pip` on a real system. -Same species as `chown: invalid user: 'wget:wget'` — half the code asking for -the name, half assembling it. - -Still owed: site-packages access is granted by `chgrp -R`/`chmod -R g+w` inside -`pip_install_module` rather than by `lfs-helper grant-dir`. Correct group, -right result, wrong door — fold it in with item 6. - -## last-step is cleaned out of the tree (1.11.1) - -`last_build_step.sh` stopped existing in 1.10.0, but three things still spoke -as if it did: - -- **`make install` copied it into `/etc/lfs`** and aborted, because the file is - not in the repo. Installing was broken and nothing said so. -- **`build-all` sealed the install directories early** when the step name was - `last-step` — dead since the final step became `init-accounts`, which runs as - root and installs nothing. The seal after the loop was already doing the work. -- **Three tests looked for the file beside `lfs`** and skipped silently when it - was absent, which is every run since 1.10.0. - -The name stays in `_is_not_a_package`'s match: a tree generated before 1.10.0 -still carries it in `steporder` and in its manifests, and the rule must still -say no. It is marked as legacy there. - -`make clean` exists now, and `make install` no longer hides a failed test -install behind `2>/dev/null || true` — the first of item 10. - -## The build's last word is whether you can log in (1.11.0) - -`init-accounts` asks for a root password, but it is skippable — a scripted run -has no terminal, and an interactive one can be answered with a blank line. So -`_warn_if_no_login` is the **last thing `build-all` does**, because it is the -last moment it can still be fixed. After the reboot the chroot is gone and the -way back is booting the host and mounting the tree by hand. - -It checks the **result**, not that the step ran: `/etc/shadow` for a real root -password, or any account between uid 1000 and 9999 with one. - - !! NOBODY CAN LOG INTO THIS SYSTEM. - Fix it NOW, while you are still in here: - lfs-helper init-accounts --force - -`lfs-helper check-login` runs it on its own. - -## wget belongs to packagemanager setup (1.11.0) - -`packagemanager setup --sources` prints every URL setup needs, and -`lfs build-system get-sources` asks it. **One list, owned by the tool that -installs from it.** - -The first version of this had `bootstrap_packages` in `lfs`'s own config, -resolving wget from the BLFS book separately from the tool that installs it — -a second thing to go stale, which is the mistake this project keeps making. -That key is gone. - -`setup` itself is a stub: `--sources` works, `--run` says it is not implemented -and exits non-zero. That is deliberate — `get-sources` needs the list *now*, so -a fresh build still downloads what the finished system will need. - -## packagemanager setup: what it owes - -Three guarantees left the suite with `last_build_step.sh` and belong to `setup`. -They are requirements, not history: - -1. **Each module installs as its OWN package user, from a WHEEL.** A modern - sdist needs its build backend, and with no index reachable pip cannot fetch - one: `BackendUnavailable: Cannot import 'hatchling.build'`. -2. **site-packages access goes through the normal collector-group rules** — - join the group that owns it, or ask which group should. Handing - site-packages to the `install` group would mean "anyone may install here", - which is what collector groups exist to avoid. -3. **Never rebuild an account name or home by hand** — `lfs-helper owner-name` - and `lfs-helper pkgusr-home`. This is the bug that started all of it. - -Also: wget needs `check_certificate = off` in `/etc/wgetrc` with the -`lfs-temporary-no-verify` marker, because a fresh system has no CA certificates -until BLFS `make-ca` — and you need wget to fetch make-ca. - -## Things that belong to nobody - -`never_claim_list` — files no package may own, however many write to them: - -- `/usr/share/info/dir` — rewritten by every package that ships an info page -- `/etc/ld.so.cache`, `/var/cache/ldconfig/aux-cache` — regenerated by ldconfig -- **the user database** and Shadow's backups (`passwd-`, `group-`, `.pwd.lock`) - — every `cmd_add_user` during a build touches them, so whichever package - happened to be building swept them up. ncurses did not install `/etc/group-`; - useradd did, while ncurses was building. A package owning the file that says - which packages exist is the one ownership mistake with no way back. -- the wrapper directory — `root:root` **by design**, so a package user cannot - rewrite the rule that constrains it - -- **mail spools** (`/var/mail/<name>`) — Shadow creates one for every account - it makes, so they arrive as a side effect of `useradd` - -Recording any of them produces a conflict no repair can settle: the next -package takes the file straight back. - -**A repair must not create the next run's finding.** Two passes disagreed about -`/var/mail/lfs` and each undid the other — the build-user pass gave it to root, -the unclaimed pass then reported that root holds a file no manifest claims. -`_vfy_build_user_leftovers` consults `is_never_claimed` for exactly this reason, -and there is a test on the property, not just on the one path. - -## The wrappers - -`make_wrappers` ships `chown`, `chgrp`, `chmod`, `cp`, `install`, `mkdir` in -`/usr/lib/pkgusr`, so a **package user** cannot change ownership. They are -ordinary files on `$PATH`, and if that directory is ever ahead of `/usr/bin` -for **root**, root's own privileged calls resolve to them too — and they report -success while doing nothing. - -So every privileged ownership call goes through `real_chown` / `real_chgrp` / -`real_chmod`, which resolve the binary by absolute path via `_real_tool`. -**Ownership is root's business; no `$PATH` may come between the decision and -the filesystem.** `make_wrappers` itself deliberately uses plain `chmod` — it -is setting the mode on files it just wrote, and must stay self-contained. - -## Mechanisms that are not obvious from the code - -**Staged installs.** `make install DESTDIR=<stage>`, then each file is copied -beside its destination and `rename()`d over it. Never `cp -a` over a live -system: that truncates in-use shared libraries and kills running processes. -Only for `gcc glibc binutils bash coreutils`, and only for a lone -`--phase install` — see open item 4. - -**Earlier stages.** A package is built more than once (ncurses in chapter 6 as -`lfs`, `python-tmp` in chapter 7 as root, then the real one as its package -user). Before building X, `_claim_earlier_stages` hands X every path recorded -under X, X-tmp, X-pass1, X-pass2 — matching case-insensitively, because book -titles are capitalised and tarball names are not. - -**Adoption.** Chapter 5-6 packages get their users from the manifests. This -cannot run before book 7.6 creates `/etc/passwd`; it happens at the ownership -epoch (`init-ownership`), which is the one place that decides. Walks manifests in build order so uids read as the -build order. `adopted.list` records `name size` per package so it runs once. - -**verify.** Ownership is derived state: manifests + install-dir list + -collector map determine what the tree should look like. `lfs-helper verify -[--fix]` reconciles actual against expected. - -**Build ids.** Each tool prints an md5 of its own contents. Compare host and -chroot — a mismatch means the chroot is running old code, which wasted a lot of -debugging time. - -**Section resolution.** `_resolve_section` matches a book section by id **or** -title. Some sub-sections carry ids the book toolchain generated -(`idm139921653990176` = 8.5.2.1 Adding nsswitch.conf) which change on every -rebuild of the book. An ambiguous title fragment is refused, not guessed at. - -**Sections that ship a file.** 9.6.8 prints `/etc/sysconfig/rc.site` as a -`<pre class="auto">` block with no commands at all. `_auto_file_block` extracts -it into a quoted heredoc. Every line is commented; install it anyway — it -documents the boot-script knobs where the boot scripts look. - -**Steps that are not packages.** `last-step`, `init-*` and `refind` never get -a user. `cfg_*` is decided by READING the generated script: a versioned book -section produces a phased build with `pkg_glob`, prose produces a plain root -script. Book 9.2 is `LFS-Bootscripts-20250827`, a real package. - - -## Sanity script - -`lfs-sanity.sh` — read-only, run inside the chroot as root, writes one file: - -```sh -./lfs-sanity.sh # -> /tmp/lfs-sanity.txt -``` - -Eight sections: tool build ids, shared-directory modes (flags early sealing), -uid/gid collisions, unprefixed accounts, home ownership, files with no owner, -groups, and root-owned counts. Each says what a hit MEANS, so a report reads on -its own. - -Section 1 **asserts** the owner and group it prints, not just the mode — a tree -ran 34 packages with every install directory at `root:root 775` while the -report said nothing, because that column was decoration. - -Section 3 flags an account wearing **two** prefixes (`p_cfg_*`, `p_u_*`, -`p_p_*`). Nothing creates those any more, so one appearing means a tool is -still applying the package prefix regardless of kind. `tmp_` and `init_` are no -longer allowed either: there is no such kind of account. - - -## Watch on the next build - -Nothing below has run against the real book on 1.9.0. In rough order of when -you would see it: - -1. **`ls /usr/src`** should show `pkgusr/ cfg/ lfs-pkgusr/` and nothing loose. -2. **`init-ownership` appears in `lfs-helper list`** as step 3, and turns - `built` right after `init-files`. If it says `pending` afterwards, the - checkpoint ran but did not mark it. -3. **`# 27 install directories are now root:install`** — not `0`. Zero is now - an error, but see it green rather than trusting it. -4. **`ls -ald /usr/src/pkgusr/p_*` after the first few packages** — each home - owned by its own user, not `root:root`. -5. **`/usr/src/pkgusr/p_gettext/src`** exists and is a directory, not a - symlink. -6. **The first config step** (`cfg_nsswitch`) comes out as `cfg_nsswitch` - under `/usr/src/cfg`, with no `p_` on it. -7. **A collector-group prompt** offers `<prefix>_perl`, never `<prefix>_p_perl`. -8. **`lfs-helper verify`** at the end: expect nothing, or a stale manifest - entry for a file two packages both install (`/usr/bin/hostname`). - -Then `bash lfs-sanity.sh` and read the `!!` lines. - -## Open items - -1. ~~`packagemanager setup`~~ — **done in 1.11.2.** What is left of it is the - grant-dir door, folded into item 6. - -2. **`strip` is asked for but does nothing** — and since 1.11.4 the build says - so at the end, so it is no longer silent. Implementing it: The answer is collected and the - prompt says NOT YET IMPLEMENTED. Not a book copy-paste: stripping rewrites - each binary, and here a file's owner is the record of which package - installed it, so it must run as that owner or the record drifts. Book 8.85 - also removes libtool `.la` files, which can make later BLFS builds link - against stale paths. - -3. **Steps 88-99 have never completed.** Steps 1-87 are well proven — the - sources/build split ran across all of them. The last twelve are all `cfg_*` - steps, and the fixes to them are unit-tested but have never run against the - real book. - -4. **Staging is limited to `--phase install`,** so `build-all` does not stage - gcc or glibc. `--phase all` runs the book's post-install commands in the - same script, and those touch live paths (`mv /usr/bin/chroot /usr/sbin`), - which fails with DESTDIR set. Only bites on rebuilds of a *running* system. - Deliberately deferred to item 6 — fixing it separately means restructuring - the build loop twice. - -5. **Root password** — the one decision the interview cannot hold; a plaintext - config would be worse. Handled as far as it can be: `init-accounts` prompts, - and `_warn_if_no_login` checks the *result* as build-all's last word. A - non-interactive run therefore finishes with a system nobody can log into, - loudly. Nothing further is planned. - -6. **The build loop is still two implementations.** Everything else went - through one door in 1.11.3 — wrappers, account home, collector grants — and - what remains is the loop itself: `cmd_build` and `packagemanager_install` - each drive unpack/build/install/configure. Item 4 (staging) lives inside it. - Needs a real build to prove, so it is not a desk change. - - The original framing, still true of the loop: Nearly every bug this project has had was the same species: one - decision held in several places, free to disagree — `cfg_bootscripts`, - `adopted.list`, the package-user prefix, the `/sources` permissions, the - account's kind. `install-as` is the entry point that makes unifying them - possible. Highest value, highest risk. - -7. **`migrate-layout`** — layout is configurable but only takes effect on a - fresh build. Moving an existing system's users is not written, and the - naming split widened the gap again: an existing tree holds - `p_cfg_bootscripts` accounts and directories. The tools *read* those - correctly — `pkgusr_kind` strips the package prefix before it looks, so a - legacy name resolves to the right kind and root — but nothing renames the - account or moves the directory. A fresh build is still the answer. - -8. ~~`README.md` is stale~~ — done, and kept current. Build ids are - deliberately not in it; they are in "Current state" above, which is the - file that gets handed over. - -9. **`lfs-sanity.sh` still reads the old state path on an old tree.** It - takes `LFS_PKGUSR_DIR`, so it is right on anything built by 1.9.0, but it - has no fallback for a tree built before the move. Harmless given that no - tree is migrated; worth deleting the ambiguity if migration is ever written. - -10. ~~The `2>/dev/null || true` sweep~~ — **done in 1.11.4** for ownership and - permission calls, with `soft` and a test on the marker. `packagemanager_install` - still has seven, all benign (chmod on a scratch directory, `disown` on a - backgrounded mandb); they have not been reviewed one by one. - - -## Useful commands - -```sh -lfs build-system verify # what state is this tree in? -lfs build-system run # continue from wherever it stopped -lfs snapshot save <n> --run # records the toolchain state in metadata -lfs-helper verify --fix # reconcile ownership -lfs-helper which-package <p> # which package installed this -lfs-helper install-as <n> -- <cmd> # install anything as its own package user -lfs-helper pkgusr-info # where each package came from -``` - - -## Tests - -```sh -bash test_lfs_crosschain.sh ./lfs # 577 tests, 576 pass -``` - -Counts are kept in files, not shell variables, so subshell increments survive. -A few Python blocks print several PASS lines but count once. - -"Nothing can fail silently" was not true: a test whose helper died before its -first assertion reported PASS. Use `_slice_fn` to lift a function out of a -script — never `sed -n "/^fn() {/,/^}/p"`, which cannot handle a one-liner. - -**Slice the whole function, never a fixed window.** Several tests used -`grep -B 14` or `sed -n '/marker/,+6p'` around a message. Adding a branch above -the marker pushed what they were asserting out of the window, and they failed -without anything being wrong. Use `_slice_fn`. - -**Do not define a function inside another one.** `_package_already_owns_files` -was first written inside `cmd_build`. bash accepts it -- and it is then defined -only after `cmd_build` has run once -- but every `sed -n '/^cmd_build() {/,/^}/p'` -in the suite stopped at the inner closing brace, silently truncating what those -tests examined. - -**Match invocations, not message text.** Several tests assert that a function -no longer calls something. The comment explaining *why* it no longer calls it -names the thing, so a naive grep matches the explanation and the test can never -pass. Strip comments first, or anchor on the call syntax. - -Run it from a directory holding ALL five scripts, plus `lfs-sanity.sh` and -`skel-u_xdg/.bash_profile` — about 100 tests silently skip if the others are -not beside `lfs`. - - -## Working style that worked - -- Diagnose from evidence in the output, not from the most recent theory. -- Every bug fixed gets a regression test with a comment naming what broke. -- Verify a new test **fails on the old code**. A test that cannot fail is - worse than no test. One nearly shipped because its slice boundary was wrong - and it reported four false failures; another shipped and sat green for - months, its four assertions never reached, because `sed -n "/^fn() {/,/^}/p"` - cannot slice a one-line function and the over-captured text died under - `set -u` before the first check. If a test has never been seen red, it has - not been tested. Use `_slice_fn`. -- **An empty result is not a pass.** Both of those failed the same way: a - helper was missing, the output was empty, and empty read as "no findings". - Assert the helper exists before trusting that it found nothing. -- Assert intent, not implementation. Three tests broke on the rename because - they grepped for literal paths rather than the property that mattered. -- Fix causes, not symptoms. Remove workarounds once the cause is fixed. -- Say plainly when something is uncertain or untested. -- **Before any restructuring, diff the command list before and after.** A - deleted command passes every test that does not mention it — that is how - `adopt-existing`, `sort-users` and `seal-install-dirs` silently disappeared. - - -## Known limits - -x86_64 only. Kernel and bootloader are left to the user. +# lfs-pkgusr — handoff + +Tools for building Linux From Scratch and Beyond LFS as **package users**: one +account per package, so every file on the system has an owner, and +`packagemanager verify` can say what is installed and what is not. + +Written for a **SysV** system (no systemd) built from the **systemd** BLFS +book, because BLFS stopped publishing a SysV edition after 12.4. + + all five tools at 1.14.146 + lfs c0858da + lfs-helper bf6756c + packagemanager 367a690 + blfs 877e4cb + lfs-phases ba72201 + lfs-kernel 1aa3396 (no version line of its own) + lfs-sysvbook 62ebe64 (1.0.0; untouched since it was written) + +**`BUILD.md` is the one document for building a whole system**, host to +desktop, with what to configure before each step. README.md points at it. +Keep it true: when a command or a config key changes, it changes there too. + +`test_lfs_crosschain.sh` is the regression suite. It no longer finishes inside +one command slot in the authoring sandbox; run it whole on a real machine, or +one block at a time (each starts `# ---- <title>`). One check fails by +environment only: **rc.site** needs a SysV LFS book on disk. + +The full per-release history — 144 entries, every bug and what it taught — is +`HANDOFF-history.md`. It is worth grepping when something looks familiar; it +is too long to read. + +--- + +## The state of the machine + +A ThinkPad-class AMD Renoir laptop, SysV init, 93 package accounts, collector +prefix `nimgnu`, package prefix `p`. + +**Built and verified: 319 packages**, `packagemanager verify` clean. That +includes the whole sway stack (wlroots, sway, swaybg, foot, seatd, elogind, +webkitgtk, pipewire, mesa, gtk3/4, gstreamer), the network stack (wpa_supplicant, +libnl, iptables-nft with libnftnl, net-tools), services (dbus, alsa, bluetooth, +acpid, fcron, blfs-bootscripts) and bash-completion. + +**Kernel 7.2.4 built, installed and verified** by `lfs-kernel`: image on the +ESP, modules with a depmod index, headers, v4l2loopback, and 15 embedded +firmware blobs. `lfs-kernel verify` reports 32 checks green. + +**The machine is still running 7.2.0.** 7.2.4 is on the ESP as +`vmlinuz_new`, beside the working `vmlinuz`, and has never been booted. +Nothing is recorded as known-good yet, which is correct and deliberate. + +**Nothing in the desktop has ever run.** The last sway attempt failed in the +chroot for want of `/run/udev`, which is the expected result there. + +--- + +## What to do next, in order + +1. **Point rEFInd at `/boot/efi/EFI/NimGnu/vmlinuz_new`** as a second entry. + Keep the existing `vmlinuz` entry: it is the way back. +2. **Reboot into 7.2.4.** Then `uname -r`, `lfs-kernel verify` (last-good + should read "it is running now"), and `dmesg | grep -i firmware` — that + last one is the definitive `FIRMWARE_ONLY` list for this hardware. +3. **Start sway** as the desktop user, on a tty: `sway` (an alias for their + own `sway_start`). The seat comes from `/usr/bin/seatd-<user>`, a setuid + wrapper compiled for one username; there is no seatd daemon and no `seat` + group. If it fails, `stacks/sway-session.sh`'s launcher checks the three + things that were each a separate bug: the wrapper, `/run/udev`, and + whether this tty is the active console. +4. **Bring up wifi.** `wifi` is in `/bin`, `wpa_supplicant`/`wpa_cli` in + `/usr/sbin`. Still missing: `/etc/sysconfig/ifconfig.wifi0` (with + `SERVICE=wpa`) and the per-SSID files under + `/etc/sysconfig/wpa_supplicant/`. Note their `wifi` script calls + `ifup $IFNAME wifi0` with `IFNAME` unset while `INTERFACE` is set and + unused — one of the two should go. +5. **make-ca's weekly job.** The systemd book enables a timer, which the shim + skips. fcron exists now, so add the 12.4 book's cron entry by hand: + + cat > /etc/cron.weekly/update-pki.sh << "EOF" + #!/bin/bash + /usr/sbin/make-ca -g + EOF + chmod 754 /etc/cron.weekly/update-pki.sh + +Smaller, none blocking: `dbus` was built before `[dbus] systemd = disabled` +went into machine.conf (rebuild or accept); a stray `.service` from +at-spi2-core sits in `/usr/lib/systemd/user`; root cannot log in after boot +and the cause is unknown — wants `ls -l /etc/shadow`, `getent shadow root`, +and whether `/bin/bash` is in `/etc/shells`. + +--- + +## The tools + + lfs the LFS build: interview, chroot, chapters 5-8 + lfs-helper the in-chroot driver (bash; no Python in the chroot) + packagemanager BLFS packages, stacks, accounts, verify + blfs turns book pages into install scripts + lfs-phases the phase runner every generated script sources + lfs-kernel build/install/verify a kernel and its firmware + lfs-sysvbook graft the 12.4 SysV flavour onto a systemd-only LFS + book -> 13.0-sysv (LFS only; BLFS has no counterpart + yet, see deferred). Its own notes: + HANDOFF-sysv-session.md (old; its DRIFT section is + history now) + +Everything installs with `make install` (default prefix `/usr/bin`; `make +clean` and `make uninstall` exist). `lfs run` refreshes the tools, the stack +files and `kernel.conf` into the chroot; the list of tools comes from the +Makefile's `TOOLS` line, so a new tool needs no second edit. + +### Things worth knowing before changing anything + +- **Config files under `/etc/pkgusr/` are never overwritten once edited.** + `make install` records what it shipped in `stacks/.shipped/`; if your copy + still matches that record it is updated, otherwise the new one lands as + `.new` and `packagemanager stack <name> --take-new` takes it. +- **`--only <entry>`** runs one stack entry, ignoring the done-cache. +- **`packagemanager reclaim <pkg> --run`** gives a package back files another + package took from it. +- **`lfs-kernel`** is configured by `/etc/pkgusr/kernel.conf` alone: + `VERSION` (`latest` asks kernel.org), `ESP`+`KERNEL_NAME` (which decides + test vs real kernel), `JOBS`, `CONFIG_FROM`, `CONFIG_REFRESH`, + `FIRMWARE_FROM`/`FIRMWARE_ONLY`, `HEADERS`, `V4L2LOOPBACK`. `lfs-kernel + list` shows what kernel.org offers; `lfs-kernel verify` checks the result. + +--- + +## Rules this project learned the hard way + +These are the ones that cost releases. They are worth applying to new code +without waiting for the failure that proves them again. + +**Confirm your own effects.** A zero exit is not proof: a phase whose log +says "make: *** Error 1" failed whatever it returned; a write must be read +back (`usermod -d` returned 0 and changed nothing, three times); a step that +installed nothing must not be cached as done; a manifest must exist where +`verify` looks, not merely be written. Five separate bugs, one rule. + +**A message that names a thing must be checked against the thing.** Paths, +package names, flags, commands — all machine-checkable, and all were wrong at +some point. The suite now asserts that every flag the failure box suggests is +one the command accepts. If a message contains a path, build it with the code +that builds that path. + +**Prose is not code.** A book's example, a NOTE mentioning `cmake`, a shell +diagnostic saying `not: command not found` — five filters and one *injector* +were caught treating text about a command as the command. Anything scanning +generated scripts skips comments unless it has a reason not to. + +**Derive lists, never repeat them.** Adding one file meant editing four +places that enumerated files (Makefile, chroot refresh, `.shipped`, the tool +list). When adding something, grep for a sibling's name first: every place it +appears is a place the new thing belongs. + +**Two settings must not answer one question.** `pkgusr_home` and +`pkgusr_subdir` could each be read as "where homes live", and two programs +read them differently, so every account had two homes and two tools took +turns correcting each other. + +**A package user owns files, not identity.** Ownership, setuid bits, system +accounts and anything on a vfat mount are root's. Do those explicitly, once, +and record what was refused — a silent refusal becomes a mystery hours later +in a different phase. + +**Handing over files is for genuine overlap.** A package installing another +package's whole output is duplicating, not overlapping; stop the bundling +(`wrap_mode=nodownload`) rather than transferring ownership one file per +retry round. + +**Report what you found, not what you expected.** Four releases went into +guessing linux-firmware's directory layout; one `git ls-files` answered it. +When a tool needs a fact about someone else's data, the first version should +go and read it. + +**Say it before it takes a while.** A tool silent for two minutes is +indistinguishable from one that has hung. + +**A check must be cheaper and quieter than the thing it checks.** `verify` +was resolving versions over the network and would have created accounts. If +a check needs the network, the accounts and the config to be right, it cannot +tell you whether they are. + +**Never record a guess as a fact.** `last-good` named a kernel that had never +booted, because an empty file looked like something to fill in. + +--- + +## This session (1.14.146) + +- `BUILD.md` written; README tightened (kernel caveat was stale: it said + nothing builds a kernel, `lfs-kernel` does). +- `lfs-helper`: the end-of-build message pointed at book 10.3 by hand; + now points at `kernel.conf` + `lfs-kernel`. +- `stacks/sway-session.sh` hardcoded one username as the desktop user while + `mytools.sh` read `/etc/pkgusr/desktop-user` -- two settings, one + question. Both now take `LFS_DESKTOP_USER` or that file, and both write + it. Installed copies of the stack file are unedited, so `make install` + updates them in place. +- Not run: the regression suite (sandbox slot). `bash -n` / `py_compile` + and `--version` only. + +## Deferred tool work + +None of this blocks the machine. + +- The regression suite exceeds one execution slot; make it splittable + (`--from N`, or per-tool) before it grows further. +- Three functions move stack files (Makefile, chroot refresh, `--take-new`); + they should be one, and the whole `.new` mechanism deserves rethinking + around a single source of truth. +- Should the CONFIGURE phase run as root? Every BLFS "Configuring" section is + written for root. Files it creates would become root-owned, which the file + tracking does not expect. Decide at the start of a session, not the end. +- `needs=` is set for sway only; the other git entries' dependencies are + discoverable only by building them. Each such failure deserves a line in + the stack, not a package installed by hand. +- `--check-refs` cannot see tar URLs. A `have=`-style check for tarballs + would close the gap libnftnl exposed. +- The "busy tree" test is flaky: it races a `sleep` in the process table and + should use a private marker. +- The rc.site test should say it needs a SysV book rather than failing. +- The whole-book invariants (954 scripts, 0 syntax errors, 0 empty install + phases) have caught or confirmed nearly every filter change. Extend them to + build phases, listing the known-empty pages by name. +- A BLFS-SysV counterpart tool: apply 12.4's init differences to 13.x pages. + `blfs diff-init` is the survey that would drive it. diff --git a/Makefile b/Makefile index b395a75..5d7850f 100644 --- a/Makefile +++ b/Makefile @@ -11,7 +11,7 @@ # make install DESTDIR=/mnt/lfs PREFIX=/usr # /usr, not /usr/local: these tools ARE the system's package management -- the -# chroot invokes them by name, and packagemanager runs packagemanager_install +# chroot invokes them by name, and packagemanager runs lfs-helper pm-install # off $PATH, which does not include /usr/local everywhere. PREFIX ?= /usr DESTDIR ?= @@ -20,13 +20,25 @@ SHAREDIR := $(DESTDIR)$(PREFIX)/share/lfs-pkgusr COMPDIR := $(DESTDIR)$(PREFIX)/share/bash-completion/completions SYSCONFDIR ?= $(DESTDIR)/etc -# The four tools, plus the bash engine packagemanager calls. -TOOLS := lfs lfs-helper packagemanager packagemanager_install blfs +# The four tools, the book merger, and the phase runner every generated +# install script sources (it goes in bin so `. lfs-phases` finds it on PATH, +# on the host, in the chroot and on the built system alike). +TOOLS := lfs lfs-helper packagemanager blfs lfs-sysvbook lfs-phases lfs-kernel COMPLETION := lfs-completion.bash TESTS := test_lfs_crosschain.sh INSTALL ?= install +# Directories are created with mkdir -p, NEVER `install -d`. +# +# `install -d` applies its mode to a directory that ALREADY EXISTS, so +# installing the tools re-moded /usr/bin from 775 to 755 and silently broke +# the package-user system: package users are in group install and need +# group-write on the install directories. The next build then failed with +# permission errors that had nothing to do with the package. +# mkdir -p leaves an existing directory exactly as it is. +MKDIR ?= mkdir -p + .PHONY: all install uninstall check clean help all: help @@ -41,7 +53,13 @@ help: install: @echo "==> tools -> $(BINDIR)" - $(INSTALL) -d $(BINDIR) + $(MKDIR) $(BINDIR) + @# packagemanager_install is retired (1.12.0, folded into lfs-helper + @# pm-install) -- sweep a previously installed copy off the system. + @if [ -e "$(BINDIR)/packagemanager_install" ]; then \ + rm -f "$(BINDIR)/packagemanager_install" \ + && echo " removed retired packagemanager_install"; \ + fi @for t in $(TOOLS); do \ if [ -f "$$t" ]; then \ $(INSTALL) -m 755 "$$t" "$(BINDIR)/$$t" && echo " $$t"; \ @@ -49,14 +67,92 @@ install: echo " !! missing: $$t" >&2; exit 1; \ fi; \ done + @echo "==> stacks -> $(DESTDIR)/etc/pkgusr/stacks" + $(MKDIR) $(DESTDIR)/etc/pkgusr/stacks + @# A STACK FILE IS THE USER'S. This overwrote it unconditionally, so + @# `make install` silently undid every edit -- and the edits are the + @# point of the file: which packages to avoid, which refs to pin. A + @# user who had just removed `avoid ruby` would have had it back, and + @# the build would have failed the same way again with no explanation. + @# Install it when it is missing; otherwise keep theirs and leave the + @# new one alongside for comparison. + @# ...but only a file the user has ACTUALLY edited. Keeping every + @# installed stack forever meant a shipped fix -- new entries, a + @# corrected have= -- waited in a .new file for a manual merge, even + @# when the user had never touched the file. `packagemanager stack + @# services` then ran three entries while the new one had six, and + @# nothing said why. So record what we shipped: if the installed copy + @# still matches the last shipped copy, it is ours to update. + @$(MKDIR) $(DESTDIR)/etc/pkgusr/stacks/.shipped + @for f in stacks/*; do \ + [ -f "$$f" ] || continue; \ + b="$$(basename $$f)"; \ + d="$(DESTDIR)/etc/pkgusr/stacks/$$b"; \ + r="$(DESTDIR)/etc/pkgusr/stacks/.shipped/$$b"; \ + if [ ! -e "$$d" ]; then \ + $(INSTALL) -m 644 "$$f" "$$d" && cp -f "$$f" "$$r" \ + && echo " $$f"; \ + elif cmp -s "$$f" "$$d"; then \ + cp -f "$$f" "$$r"; echo " $$b (unchanged)"; \ + elif [ -f "$$r" ] && cmp -s "$$r" "$$d"; then \ + $(INSTALL) -m 644 "$$f" "$$d" && cp -f "$$f" "$$r" \ + && rm -f "$$d.new" \ + && echo " $$b (updated -- you had not edited it)"; \ + else \ + $(INSTALL) -m 644 "$$f" "$$d.new" \ + && echo " $$b -> KEPT YOURS; new version saved as $$d.new"; \ + fi; \ + done; true + @# kernel.conf is a file to EDIT BEFORE building, so it is installed + @# rather than written on first run -- and, like a stack file, an + @# edited one is kept and the new one lands beside it. + @if [ -f kernel.conf ]; then \ + d="$(DESTDIR)/etc/pkgusr/kernel.conf"; \ + if [ ! -e "$$d" ] || cmp -s kernel.conf "$$d"; then \ + $(INSTALL) -m 644 kernel.conf "$$d" && echo " kernel.conf"; \ + else \ + $(INSTALL) -m 644 kernel.conf "$$d.new" \ + && echo " kernel.conf -> KEPT YOURS ($$d.new)"; \ + fi; \ + fi + @# A STACK ENTRY MAY HAVE A DIRECTORY. The loop above takes files + @# only, so stacks/mytools/install_mytools -- the package-user script + @# the mytools step runs -- was never installed, and the step would + @# have failed with "missing .../mytools/install_mytools". Copy the + @# subdirectories too, without clobbering a script the user edited. + @for d in stacks/*/; do \ + [ -d "$$d" ] || continue; \ + b="$$(basename $$d)"; \ + $(MKDIR) "$(DESTDIR)/etc/pkgusr/stacks/$$b"; \ + for f in "$$d"*; do \ + [ -f "$$f" ] || continue; \ + t="$(DESTDIR)/etc/pkgusr/stacks/$$b/$$(basename $$f)"; \ + if [ ! -e "$$t" ] || cmp -s "$$f" "$$t"; then \ + $(INSTALL) -m 755 "$$f" "$$t"; \ + else \ + $(INSTALL) -m 755 "$$f" "$$t.new" \ + && echo " $$b/$$(basename $$f) -> KEPT YOURS ($$t.new)"; \ + fi; \ + done; \ + echo " $$b/ ($$(ls -1 $$d | wc -l) file(s))"; \ + done; true + @chmod 755 $(DESTDIR)/etc/pkgusr/stacks/*.sh 2>/dev/null || true @echo "==> bash completion -> $(COMPDIR)" - $(INSTALL) -d $(COMPDIR) - $(INSTALL) -m 644 $(COMPLETION) $(COMPDIR)/lfs - @for t in blfs packagemanager lfs-helper; do \ - ln -sf lfs "$(COMPDIR)/$$t"; \ - done + @# Optional, and a missing one must not take the tools down with it: + @# `install: cannot stat 'lfs-completion.bash'` aborted the whole target + @# after the tools were already in place, so the run looked failed while + @# most of it had succeeded. + @if [ -f "$(COMPLETION)" ]; then \ + $(MKDIR) $(COMPDIR); \ + $(INSTALL) -m 644 $(COMPLETION) $(COMPDIR)/lfs; \ + for t in blfs packagemanager lfs-helper; do \ + ln -sf lfs "$(COMPDIR)/$$t"; \ + done; \ + else \ + echo " skipped: $(COMPLETION) is not in this checkout"; \ + fi @echo "==> XDG profile for shared users -> $(SYSCONFDIR)/pkgusr/skel-u_xdg" - $(INSTALL) -d $(SYSCONFDIR)/pkgusr/skel-u_xdg + $(MKDIR) $(SYSCONFDIR)/pkgusr/skel-u_xdg @if [ -f "$(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile" ]; then \ echo " kept your existing .bash_profile"; \ else \ @@ -65,7 +161,7 @@ install: echo " installed (XDG_RUNTIME_DIR is filled in on first use)"; \ fi @echo "==> tests -> $(SHAREDIR)" - $(INSTALL) -d $(SHAREDIR) + $(MKDIR) $(SHAREDIR) # The tests are optional -- say so when they are absent, rather than hiding a # real failure. `2>/dev/null || true` hid both cases alike. @if [ -f "$(TESTS)" ]; then \ diff --git a/README.md b/README.md index 86b60d5..4c79fc7 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Works **offline** from a downloaded copy of the books. --- -## The four tools +## The tools | Tool | Runs | Does | |---|---|---| @@ -33,6 +33,9 @@ Works **offline** from a downloaded copy of the books. | `lfs-helper` | inside the chroot | Builds each package as its user (bash, no Python needed) | | `packagemanager` | built system | Installs and updates packages | | `blfs` | built system | Reads the BLFS book, generates install scripts | +| `lfs-sysvbook` | host | Grafts SysV init onto a systemd-only LFS book | +| `lfs-phases` | everywhere | The runner every generated install script sources | +| `lfs-kernel` | booted system or chroot | Build, install and verify a kernel, its firmware and its headers | --- @@ -46,9 +49,11 @@ make uninstall ``` `/usr` is the default: the chroot invokes these by name, and -`packagemanager` runs `packagemanager_install` off `$PATH`. +`packagemanager` runs `lfs-helper pm-install` for installs. -Installs the four tools, `packagemanager_install`, and the completion script. +Installs the tools, the stack files and `kernel.conf` (never over an edited +copy — a changed one lands beside yours as `.new`), and the completion +script. **Requires:** Python 3.9+, bash, coreutils, tar. Book parsing needs `beautifulsoup4` and `requests`. @@ -57,14 +62,20 @@ Installs the four tools, `packagemanager_install`, and the completion script. ## Build a system +**Read [BUILD.md](BUILD.md).** It walks the whole thing, host to desktop, and +says at each step what to configure first. The short form: + ```sh -lfs build-system session # partition, mount point, prefix, locale +lfs build-system session # the interview: partition, prefixes, locale, login lfs run # everything up to the chroot, then enter it -lfs-helper build-all # inside: build chapters 7-9 +lfs-helper build-all # inside: chapters 7-9 +packagemanager bootstrap --run # wget, Python modules, BLFS book, certificates +packagemanager stack sway --run # then network, services +lfs-kernel # the kernel, from /etc/pkgusr/kernel.conf ``` -`lfs run` continues from wherever it stopped. Run it again after a reboot, a -failure, or a cancel. +`lfs run` and `build-all` continue from wherever they stopped. Run them again +after a reboot, a failure, or a cancel. | Command | Does | |---|---| @@ -76,19 +87,86 @@ failure, or a cancel. --- +## SysV on LFS 13.0+ + +LFS dropped the SysV edition with 13.0. `lfs-sysvbook` grafts the SysV +flavour of the last SysV book (12.4) onto a new systemd release and stores +the result as a normal book: + +```sh +lfs fetch 12.4 # donor: the last SysV book +lfs fetch 13.0-systemd # target: the new release +lfs-sysvbook make # -> 13.0-sysv (book + wget-list + md5sums) +lfs-sysvbook check --book 13.0-sysv +lfs --book 13.0-sysv run +``` + +Everything builds at 13.0 versions. The transplanted SysV pieces (udev from +systemd, sysvinit, sysklogd, bootscripts, chapter 9) stay pinned at the +donor's versions — those instructions are known to work. Section numbers on +transplanted pages keep the donor's numbering; the build order is by document +position and is correct. + +For a newer release later: `lfs-sysvbook make --target 13.1-systemd`. +`check` fails loudly if a future book layout breaks a rule. + +--- + +## Install scripts + +A generated script is the package's variables and its commands, nothing else: + +```sh +name_version="binutils-2.45" +pkg_glob="binutils-2.45.tar.*" +pkgusr_stage="cross" + +build_pkg() { +mkdir -v build +cd build +../configure --prefix=$LFS/tools ... +make +} +install_pkg() { +make install +} +test_pkg() { +make check +} + +. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@" +``` + +The last line hands over to `lfs-phases`: it finds or fetches the tarball, +unpacks it, runs each phase as its own process under `set -e`, resumes in the +build directory, and takes `all | unpack | build | test | install | configure +| update` as `$1`. Edit the functions; the rest is shared. + +Tests run only with `PKGUSR_TESTS=1` and never stop the install. The runner is +found on `PATH` or through `PKGUSR_LIB`. + +Scripts written by 1.13 and earlier carry their own runner and keep working; +`--regenerate` produces the new shape. + +--- + ## Use the built system -A fresh system has no download tool, and `lfs` and `blfs` need `requests` and -`beautifulsoup4` to read the books. Bootstrap both, offline, from what -`get-sources` already downloaded: +A freshly booted system has no download tool, no Python modules to read a book +with, no BLFS book and no certificates. `bootstrap` shows all five stages and +does the two that need no network: ```sh -packagemanager setup # what it would do -packagemanager setup --run +packagemanager bootstrap # where you are +packagemanager bootstrap --run # wget and the Python modules, offline ``` -Each module is installed as its own package user. Run it again after fixing a -failure; it skips what is already there. +Stages 1 and 2 come from what `get-sources` already downloaded, so they need no +network. Each module is installed as its own package user. The rest — the BLFS +book, `make-ca`, the certificate bundle — needs a working download, which is +what stages 1 and 2 buy you, so `--run` does those too. A stage whose +dependency failed is skipped, not attempted. Run it again after fixing +anything; it does only what is left. ```sh packagemanager install <pkg> --recursive --run @@ -206,6 +284,29 @@ let any package user delete another package's home. --- +## The kernel + +`lfs-kernel` treats the kernel as a package: its sources live in +`/usr/src/pkgusr/p_linux/src`, its files are recorded in that account's +manifest, and `packagemanager remove linux` takes it away. Firmware is its +own package (`p_linux-firmware`) and the headers update the package that owns +`/usr/include`. + +```sh +lfs-kernel list # what kernel.org offers +lfs-kernel # build the version in /etc/pkgusr/kernel.conf +lfs-kernel 6.12.40 # that version, once +lfs-kernel verify # is everything installed? +``` + +Everything it does is configured in `/etc/pkgusr/kernel.conf`, which is +installed rather than generated so it can be edited first. `ESP` and +`KERNEL_NAME` together decide whether a build replaces the working kernel or +lands beside it as something to try; the tree of the last kernel known to +boot is never deleted. + +--- + ## Checking your version The tools print a fingerprint of their own contents: @@ -229,21 +330,22 @@ bash test_lfs_crosschain.sh ./lfs Each test encodes a bug that actually happened, with a comment explaining what broke. -Run it from a directory holding all five scripts — about 100 tests skip -without them. +Run it from a directory holding all the tools — about 100 tests skip +without them. It no longer fits one sandbox slot; on a real machine run it +whole, or one block at a time (each starts `# ---- <title>`). --- ## Caveats -- **The kernel and the bootloader are yours.** Nothing here configures, builds - or installs a kernel, and nothing is written to any ESP, boot sector or - partition table. The machine keeps booting exactly as it does now. The build - ends by saying so and printing what to run. -- A rEFInd entry can be added beside your existing bootloader if you ask for it - (`lfs config bootloader refind`). It only ever adds to a mounted ESP. -- `lfs-helper` and `packagemanager_install` duplicate some logic. Unifying them - is worthwhile but not done. +- **The kernel and the bootloader are separate, deliberate steps.** + `build-all` builds neither; it ends by saying so. `lfs-kernel` builds the + kernel from a config you edit first. Nothing is ever written to an ESP, + boot sector or partition table unless you ask (`lfs config bootloader + refind` adds an entry beside your existing one, on a mounted ESP). +- Built for SysV init. BLFS pages from the systemd book are used with a + shim that turns `systemctl` calls into visible skips; `services.stack` + installs the SysV boot scripts instead. - Tested on x86_64 only. --- diff --git a/blfs b/blfs old mode 100644 new mode 100755 index 943b4ec..dbfacf1 --- a/blfs +++ b/blfs @@ -78,21 +78,19 @@ except ImportError: def _need_requests(): if requests is None: - sys.stderr.write( - "this command needs the 'requests' module, which base LFS does not\n" - "ship. Install it as a package user:\n" - " packagemanager pip install requests\n" - "Commands that only read an already-downloaded book work without it.\n") + fail("this command needs the 'requests' module, which base LFS does not\n" + "ship. Install it as a package user:\n" + " packagemanager pip install requests\n" + "Commands that only read an already-downloaded book work without it.") sys.exit(2) return requests def _need_bs4(): if BeautifulSoup is None: - sys.stderr.write( - "this command needs 'beautifulsoup4' to parse the book. Install it\n" - "as a package user:\n" - " packagemanager pip install beautifulsoup4\n") + fail("this command needs 'beautifulsoup4' to parse the book. Install it\n" + "as a package user:\n" + " packagemanager pip install beautifulsoup4") sys.exit(2) return BeautifulSoup @@ -109,16 +107,28 @@ except Exception: # --------------------------------------------------------------------------- # SITE = os.environ.get("BLFS_SITE", "https://www.linuxfromscratch.org") -DEFAULT_SELECTOR = "stable-systemd" + +def _default_selector(): + """The flavour follows the SYSTEM, not a constant. + + A hardcoded stable-systemd default handed a SysV system the systemd + book: elogind read NOT IN BOOK (it only exists in the SysV edition) and + every generated script leaned systemd. A set default (blfs set-default) + always wins; this only decides what happens when nobody chose.""" + if os.path.isdir("/run/systemd/system"): + return "stable-systemd" + return "stable-sysv" + +DEFAULT_SELECTOR = _default_selector() VALID_INITS = ("systemd", "sysv") # Bump whenever the extracted-data shape or extraction logic changes, so stale # on-disk caches are rebuilt automatically. -CACHE_VERSION = 4 +CACHE_VERSION = 56 # v56: a skipped build variant does not keep its install step # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.11.6" +BLFS_VERSION = "1.14.146" def _build_id(): @@ -136,7 +146,7 @@ def _build_id(): except OSError: return "unknown" -SCRIPT_VERSION = 4 +SCRIPT_VERSION = 17 # v17: the runner moved out into lfs-phases def _tty_out(): @@ -148,6 +158,37 @@ _C_HEAD = "\033[1m" _C_GROUP = "\033[1;36m" _C_CMD = "\033[0;32m" +# The same three-colour vocabulary as every other tool in this set: red says +# something failed, yellow says look at this, green says it worked. blfs had +# green for commands and nothing else, so its errors were the only uncoloured +# errors in the toolchain -- and an error that does not look like the other +# tools' errors reads as less serious, which is backwards. +_C_ERR = "\033[0;31m" +_C_WARN = "\033[0;33m" +_C_OK = "\033[0;32m" + + +def _paint(msg, colour, stream): + if stream.isatty(): + stream.write(colour + msg + _C_OFF + "\n") + else: + stream.write(msg + "\n") + + +def fail(msg): + """Red, stderr: something failed or cannot be done.""" + _paint(msg, _C_ERR, sys.stderr) + + +def warn(msg): + """Yellow, stderr: needs attention, nothing has gone wrong yet.""" + _paint(msg, _C_WARN, sys.stderr) + + +def ok(msg): + """Green, stdout: finished, and finished correctly.""" + _paint(msg, _C_OK, sys.stdout) + def _colour_epilog(text): """Colour the group headings in the help -- same scheme as `lfs`.""" @@ -301,10 +342,17 @@ def _find_svn_archive(base): return fn, base + fn -def discover_dir(selector): +def discover_dir(selector, listing=False): """Locate a downloadable book for `selector`. Released books are a nochunks HTML under downloads/<selector>/; the development book ('svn') is a dated - chunked-HTML .tar.xz, so fall back to that.""" + chunked-HTML .tar.xz, so fall back to that. + + `listing` marks enumeration (discover_all probing every index directory): + the cached-book fallback below is a RESOLVE-FOR-USE courtesy -- during a + listing it turned every stale index directory ('oldsvn', + 'blfs-book-6.0-html') into a note plus the cached book masquerading under + that name. A listing probe that finds nothing simply raises, and the + caller skips the directory.""" found = (_find_nochunks_at(f"{SITE}/blfs/downloads/{selector}/") or _find_nochunks_at(f"{SITE}/blfs/view/{selector}/")) if found: @@ -340,13 +388,37 @@ def discover_dir(selector): # A book imported by hand (or copied in by `lfs build-system install-tools`) # has no metadata recording which selector it is, so the lookup failed and # every package came back "not in the BLFS book" while the book sat right - # there. Fall back to what is actually on disk. + # there. Fall back to what is actually on disk -- but only when RESOLVING + # a selector someone wants to use, never while merely listing the site. cached = _cached_book_files() - if cached: + if cached and not listing: pick = _best_cached_for(selector, cached) - sys.stderr.write( - "note: no book matched '%s'; using the cached %s\n" - % (selector, os.path.basename(pick))) + # NAME WHAT ELSE WAS ON DISK. + # + # The score prefers the asked-for INIT flavour over everything, so + # 'stable-sysv' picked the 12.4 sysv book the moment it was fetched -- + # and the user, who was building 13.0, silently started building 12.4 + # packages (BlueZ-5.83 instead of 5.86). The note named the book it + # chose but not the one it passed over, and a version going backwards + # is exactly what someone needs told. + warn("note: no book matched '%s'; using the cached %s" + % (selector, os.path.basename(pick))) + try: + _pv = parse_book_filename(os.path.basename(pick))[0] + _others = [(parse_book_filename(os.path.basename(c))[0], c) + for c in cached if c != pick] + _newer = [o for o in _others + if _version_key(o[0]) > _version_key(_pv)] + if _newer: + _n = sorted(_newer, key=lambda o: _version_key(o[0]))[-1] + warn(" a NEWER book is also cached: %s (%s)" + % (os.path.basename(_n[1]), _n[0])) + warn(" building from %s. To pin the other one:" % _pv) + warn(" blfs set-default %s" % _n[0] + + ("-systemd" if "systemd" in os.path.basename(_n[1]) + else "")) + except Exception: + pass fn = os.path.basename(pick) version, init2 = parse_book_filename(fn) return {"selector": selector, "url": None, "filename": fn, @@ -367,6 +439,14 @@ def _cached_book_files(): return [] +def _version_key(v): + """Sortable version: '13.0' > '12.4' > 'unknown'.""" + try: + return tuple(int(x) for x in str(v).split(".")) + except (TypeError, ValueError): + return (-1,) + + def _best_cached_for(selector, cached): """Pick the cached book that best matches what was asked for. @@ -383,7 +463,13 @@ def _best_cached_for(selector, cached): if selector.lower() in n: s += 8 # an exact selector match wins return s - return sorted(cached, key=lambda p: (-score(p), p))[0] + # ...and among equals, the NEWEST book: a fallback should not quietly + # take a system backwards a release. + return sorted(cached, + key=lambda p: (-score(p), + [-x for x in _version_key( + parse_book_filename(os.path.basename(p))[0])], + p))[0] def book_record_from_selector(selector): @@ -412,13 +498,20 @@ def discover_all(): (stable, svn, systemd, ...) are probed live for their real filename, with a best-guess fallback so they're always listed.""" dirs = set(KNOWN_NAMED_BOOKS) + reached = False for index in (downloads_index_url(), view_index_url()): try: resp = _need_requests().get(index, timeout=60) resp.raise_for_status() dirs.update(_extract_subdirs(resp.text)) + reached = True except Exception: continue + # Offline, this would fall through to the named-book guesses and look + # like a successful (tiny) discovery -- an empty result read as a pass. + # No index reached is a failure, and the caller decides what that means. + if not reached: + raise RuntimeError("no downloads index reachable") books = {} for d in sorted(dirs): if not _looks_like_book_dir(d): @@ -426,7 +519,7 @@ def discover_all(): rec = book_record_from_selector(d) # fast path, no request if rec is None: # named book -> probe try: - rec = discover_dir(d) + rec = discover_dir(d, listing=True) except Exception: fb = _NAMED_FALLBACK.get(d) if not fb: @@ -774,6 +867,14 @@ class Book: return self.packages.get(anchor) +def _book_version_of(book): + """'13.0' from the book file's name, '' when it has none.""" + try: + return parse_book_filename(os.path.basename(book.path))[0] + except Exception: + return "" + + def build_packages(soup): """Extract data for every resolvable package/module in the book.""" packages = {} @@ -831,6 +932,25 @@ def get_book(args, rebuild=False): # anchor / section resolution (the part that was broken) # --------------------------------------------------------------------------- # +def resolve_anchor(book, anchor): + """The book's own spelling of an anchor, whatever case was typed. + + BLFS anchor ids are not all lower case -- the Python modules keep their + project capitalisation (Mako, PyYAML, Cython). `blfs search mako` found + it (search is case-insensitive) and every other command then failed: + ERROR: 'mako' not found. try: blfs debug mako + A user who has just been shown a name should be able to type it. + """ + keys = getattr(book, "packages", book) or {} + if not keys or anchor in keys: + return anchor + low = anchor.lower() + for key in keys: + if key.lower() == low: + return key + return anchor + + def anchor_from_input(value): """Reduce a bare name / page url / fragment url to the anchor we look up.""" if value.startswith(("http://", "https://")): @@ -1063,34 +1183,142 @@ def extract_md5(sec): def extract_additional_downloads(sec): + """EVERY additional download the page lists. Which ones are actually + fetched is decided later, by whether an ACTIVE command mentions the file + (_needed_additional_links) -- not by the label. + + Labels were the first attempt: keep "required" entries and every .patch. + alsa-lib then failed, because its alsa-ucm-conf tarball is labelled + "Recommended file:" and is used by a command that is NOT conditional: + make install && tar -C /usr/share/alsa ... -xf ../alsa-ucm-conf-... + Reading the label is guessing; reading the commands is knowing. + + string= matching on the h3 broke silently when the heading holds nested + tags (BeautifulSoup's string= only matches a lone text child), which is + why docbook-xsl's additional_links came back EMPTY -- match on + get_text() instead.""" pd = package_div(sec) if not pd: return [] - h3_add = pd.find("h3", string=re.compile(r"\s*Additional Downloads\s*")) - h3_dep = pd.find("h3", string=re.compile(r"\s*Dependencies\s*")) + def _h3(title): + # h3 in older book editions, h4 in 13.0 -- the tag is layout, the + # TITLE is the contract. Matching only h3 left additional_links + # empty for every package in the 13.0 book. + return pd.find(lambda t: t.name in ("h3", "h4") + and title in " ".join(t.get_text().split())) + h3_add = _h3("Additional Downloads") + h3_dep = _h3("Dependencies") if not h3_add: return [] out, cur = [], h3_add.find_next_sibling() while cur and cur != h3_dep: - for u in cur.find_all("a", class_="ulink"): - href = u.get("href") - if href: - out.append(href) + blocks = cur.find_all(["p", "li"]) or [cur] + for blk in blocks: + for u in blk.find_all("a", class_="ulink"): + href = u.get("href") + if href: + out.append(href) cur = cur.find_next_sibling() - return out + # blocks may nest (li holding p): keep order, drop duplicates + seen, uniq = set(), [] + for h in out: + if h not in seen: + seen.add(h); uniq.append(h) + return uniq + + +DEP_KINDS = ("required", "recommended", "optional") + + +def _in_parenthetical(node): + """Is this link inside a ( ... ) aside rather than in the list itself? + + A dependency line is "Package, Package, and Package", and the book + qualifies entries in parentheses -- which often NAME OTHER PACKAGES: + + Graphite2-1.3.14 (required for building texlive-20250308 or + LibreOffice-26.2.1.2 with system harfbuzz) + Qt-6.10.2 (required for okular) + + Harvesting every link made harfbuzz depend on LibreOffice and TeX Live, + and poppler on Qt6 and okular. Planning Xwayland then came to 235 + packages, of which 178 hung off that one sentence: a JDK, PostgreSQL, an + office suite, for an X server. + + The qualified package itself is the link BEFORE the parenthesis, so + dropping what is inside loses nothing real. + """ + prev = "" + for sib in node.previous_siblings: + prev = (sib.get_text() if hasattr(sib, "get_text") else str(sib)) + prev + if "(" in prev or ")" in prev: + break + opens = prev.count("(") - prev.count(")") + return opens > 0 + + +def _dep_xrefs(pd, kind): + """The <a class="xref"> nodes of one dependency kind, in a package div. + + THE CLASS IS NOT GUARANTEED. Both extractors selected `p.{kind} a.xref` + and nothing else, so on libheif -- whose Recommended list is a plain <p> + under an <h5>Recommended</h5> -- the list vanished and `blfs deps libheif` + answered "libheif". Losing a dependency silently is the worst way to be + wrong: the build gets further before failing, and fails somewhere else. + """ + nodes = [n for n in pd.select(f"p.{kind} a.xref") + if not _in_parenthetical(n)] + if nodes: + return nodes + for h in pd.find_all(["h3", "h4", "h5", "h6"]): + if " ".join(h.get_text().split()).strip().lower() != kind: + continue + out = [] + for sib in h.next_siblings: + name = getattr(sib, "name", None) + if name is None: + continue + if name in ("h3", "h4", "h5", "h6"): + break + if name == "p": + # a classed paragraph belongs to the kind its class names + classes = sib.get("class") or [] + if any(c in DEP_KINDS and c != kind for c in classes): + break + out.extend(n for n in sib.select("a.xref") + if not _in_parenthetical(n)) + if out: + return out + return [] def _xref_titles(sec, kind): + """The dependencies of one kind (required / recommended / optional). + + The book usually marks the list with a class -- <p class="required"> -- + and this selected on that alone. But the class is not guaranteed: on + libheif the Recommended list is a plain <p> under an <h5>Recommended</h5>, + so the whole list vanished and `blfs deps libheif` answered "libheif". + Silently losing dependencies is the worst way to be wrong: the build gets + further before it fails, and fails somewhere else. + + So: the class when it is there, and otherwise the heading that says what + the list is. + """ pd = package_div(sec) if not pd: return [] - out = [] - for x in pd.select(f"p.{kind} a.xref"): - title = " ".join((x.get("title") or "").split()) - if not title or re.match(r"(Chapter|Part)\b", title): - continue - out.append(title.replace(" ", "-")) - return out + + def _titles_from(nodes): + out = [] + for x in nodes: + title = " ".join((x.get("title") or "").split()) + if not title or re.match(r"(Chapter|Part)\b", title): + continue + out.append(title.replace(" ", "-")) + return out + + return _titles_from(_dep_xrefs(pd, kind)) def extract_installed_content(sec): @@ -1181,7 +1409,20 @@ def extract_commands(sec, selector): elif name == "pre": text = _pre_command_text(el) if text: - out.append(text) + # THE GUARD BELONGS HERE TOO. extract_commands_phased checks + # each <pre> for a conditional sentence; this extractor (used + # for the CONFIGURATION section) did not -- so cyrus-sasl ran + # make install-saslauthd + # which the book introduces with "If you want the saslauthd + # daemon to start at system startup ... install the + # saslauthd.service unit included in blfs-systemd-units". + # A conditional block is conditional wherever it appears. + _g = _conditional_guard(el) + if _g: + out.append("## OPTIONAL, skipped -- the book says: " + _g) + out.append(_commented(text)) + else: + out.append(text) elif name == "div": classes = list(el.get("class", [])) if "admon" in classes: @@ -1204,6 +1445,75 @@ def extract_commands(sec, selector): return "\n".join(out) +# The book's conditionals live in PROSE: "If you wish to build and install +# the manual pages ... run:" precedes a command block that hard-requires +# doxygen. The extractor took every block unconditionally, so a fresh system +# died in libxml2's documentation build for want of a tool the book calls +# optional. A block whose introducing paragraph is conditional is emitted +# COMMENTED, with the book's own sentence above it -- visible, one uncomment +# away, never a surprise requirement. +# ANY sentence that opens with a condition, not a list of verbs. +# +# The verb list caught libuv's "If you installed the optional sphinx module, +# create the man page:" and missed the very next block, "If you BUILT the man +# page, install it as the root user:" -- so the build of the page was skipped +# and its install was not, and libuv died on a file that was never made. +# Enumerating verbs is guessing at English; the pattern is the conditional +# opening. Skipping is safe either way: the block is emitted commented with +# the book's own sentence above it, one uncomment away. +_COND_GUARD_RE = re.compile(r"^(if\b|optionally\b|should you\b)", re.I) + +# A block the book offers as a TEST is not part of installing. +# +# "To test the results, issue: gs -q -dBATCH .../tiger.eps" sits in the build +# section and runs a program that is not installed yet. It was harmless while +# a failing command was ignored; the moment the dispatcher started honouring +# set -e it would fail every such package. Tests belong to `test` +# (PM_TEST=1), not to a build. +_TEST_BLOCK_RE = re.compile( + r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)", re.I) + +def _conditional_guard(pre): + """The conditional sentence introducing this <pre>, or None if the block + must run. + + A condition is only worth skipping when it depends on something we do NOT + guarantee. These tools install required AND recommended dependencies by + default -- that is the contract -- so a block conditioned on those is + satisfied by construction and must RUN. Ghostscript proved the cost of + not knowing that: + If you have installed the recommended dependencies on your system, + remove the copies of freetype, lcms2, libjpeg, libpng, and openjpeg: + rm -rf freetype lcms2mt jpeg libpng openjpeg + Commented out, ghostscript kept its bundled copies and configure died: + Mixing local libtiff with shared libjpeg not supported + "optional" wins over both: an optional tool is exactly what we do not + install.""" + prev = pre.find_previous(["p", "pre", "h3", "h4"]) + if prev is None or prev.name != "p": + return None + txt = " ".join(prev.get_text().split()) + if _TEST_BLOCK_RE.match(txt): + return txt # a test, never part of installing + if not _COND_GUARD_RE.match(txt): + return None + low = txt.lower() + # "If sudo or su is invoked for switching to the root user, ensure ..." + # is about HOW to become root, not whether to run the command -- and the + # command it introduces was `./x.py install`, the whole point of the + # package. A sentence about privilege mechanics guards nothing. + if re.match(r"^if\s+(sudo|su)\b", low): + return None + if "optional" not in low and ("recommended" in low or "required" in low): + return None # our contract satisfies it + return txt + + +def _commented(text): + return "\n".join("#" + (" " + l if l and not l.startswith("#") else l) + for l in text.splitlines()) + + def extract_commands_phased(sec): """Split the installation section into (build, install). BLFS marks the privileged install step as <pre class="root"> (make install, install -v ...); @@ -1245,7 +1555,40 @@ def extract_commands_phased(sec): text = _pre_command_text(el) if not text: continue - (install if "root" in classes else build).append(text) + # ORDER ACROSS THE STREAMS, not just within them. + # + # A page can go user -> root -> user, and GLib does: + # build glib with introspection disabled (user) + # ninja -C gi-build install (root) + # meson configure -D introspection=enabled; ninja (user) + # Sorting blocks into "build" and "install" kept each stream's + # order but lost the INTERLEAVING, so the reconfigure ran in the + # build phase, before gobject-introspection was installed: + # ERROR: Program 'g-ir-scanner' not found + # Once a root block has appeared, everything that follows belongs + # after it -- so later user blocks join the install stream, in + # the book's own order. Both streams run as the same package + # user here (a package user installs its own files), so nothing + # is lost by moving a command across. + # A BLOCK THAT IS COMMENTED OUT DOES NOT MOVE THE STREAM. + # + # "everything after a root block belongs after it" is right for a + # root step that RUNS. gegl's first root block is conditional -- + # "If you are installing over a previous version ... remove it" -- + # so it is emitted commented, and nothing has happened; but it + # flipped the latch anyway, and the whole build (mkdir build, + # meson setup, ninja) landed in install_pkg while build_pkg came + # out as `:`. The build then ran inside the install phase, and + # when meson failed the following `ninja install` still ran: + # ninja: error: loading 'build.ninja': No such file + guard = _conditional_guard(el) + is_root = "root" in classes and not guard + stream = install if (is_root or install) else build + if guard: + stream.append("## OPTIONAL, skipped -- the book says: " + guard) + stream.append(_commented(text)) + continue + stream.append(text) else: # h3 / h4 title = " ".join(el.get_text().split()) if title: @@ -1309,9 +1652,6 @@ def anchor_from_href(href): return href, None -DEP_KINDS = ("required", "recommended", "optional") - - def section_deps(sec, kinds=("required", "recommended")): """Return list of dicts: {title, anchor, external, kind}.""" pd = package_div(sec) @@ -1319,7 +1659,7 @@ def section_deps(sec, kinds=("required", "recommended")): return [] deps = [] for kind in kinds: - for x in pd.select(f"p.{kind} a.xref"): + for x in _dep_xrefs(pd, kind): title = strip_paren(" ".join(x.get("title", "").split())) # xrefs to whole chapters/parts aren't packages -- skip them if not title or re.match(r"(Chapter|Part)\b", title): @@ -1347,22 +1687,87 @@ def deps_of(data, kinds): return [d for d in data.get("deps", []) if d["kind"] in kinds] -def walk_deps(book, anchor, kinds=("required", "recommended")): +_EXTRA_DEPS_PATH = os.environ.get( + "BLFS_PACKAGE_FIXES", "/etc/pkgusr/stacks/package-fixes.conf") + + +def _extra_deps_for(anchor): + """Dependencies the BOOK does not record but the build needs. + + libass hard-requires harfbuzz in its configure and its page lists only + FreeType, FriBidi, Fontconfig and NASM -- so no ordering could place it + (libass built at 143, harfbuzz at 199) and no entry order could either, + because both sit inside one plan. glslang for mesa was the same class, + patched by hand into the stack. + + package-fixes.conf can add the edge: + [libass] + needs = harfbuzz + They are treated as REQUIRED, because that is what they are. + """ + if not hasattr(_extra_deps_for, "_map"): + m = {} + try: + import configparser + cp = configparser.ConfigParser() + cp.read(_EXTRA_DEPS_PATH) + for sect in cp.sections(): + v = cp[sect].get("needs", "") + if v.strip(): + m[sect.lower()] = [x.strip() for x in + re.split(r"[,\s]+", v) if x.strip()] + except Exception: + m = {} + _extra_deps_for._map = m + # the same shape as a dep parsed from the book, or consumers KeyError + return [{"anchor": n, "kind": "required", "name_version": n, + "external": False, "title": n, "optional": False} + for n in _extra_deps_for._map.get(anchor.lower(), [])] + + +def walk_deps(book, anchor, kinds=("required", "recommended"), ignore=()): """Depth-first walk over cached package data. Returns (order, edges, missing, via). `order` is install order (deps first); `edges` maps anchor -> child dep dicts; `via` maps each dependency anchor -> (parent_anchor, - kind) recording who first pulled it in and how.""" + kind) recording who first pulled it in and how. + + AN AVOIDED PACKAGE IS NOT WALKED INTO. Ignoring happened afterwards, on + the finished list, so `--ignore libreoffice` dropped LibreOffice and kept + everything only LibreOffice wanted: PostgreSQL, unixODBC, CLucene, + Redland, OpenJDK, apache-ant, Cups... Planning harfbuzz with libreoffice + ignored came to 463 packages, and the plan stopped on a JT_JAVA guard for + a JDK nobody asked for. sway.stack says what was meant -- "everything + they alone pull in goes with them" -- and it has an avoid list 80 names + long because that is what it took to say it by hand. + + Not walking into it is the whole fix: anything ANOTHER package needs is + still reached through that package, so shared dependencies survive. + """ order, edges, missing, via, visited = [], {}, [], {}, set() + ignored_required = [] + ignore = {x.lower() for x in (ignore or ())} - def visit(a): + def visit(a, parent="", kind=""): if a in visited: return + if a.lower() in ignore and a != anchor: + # AN IGNORED PACKAGE THAT SOMETHING REQUIRES. + # + # Not walking into an avoided branch is right (1.14.24). But + # webkitgtk REQUIRES Ruby, the stack's avoid list contains ruby, + # and the plan therefore left out a hard requirement of a package + # the user had asked for -- silently, until cmake said "Ruby 2.5 + # or higher is required" after everything else had built. + # Skipping is still correct; saying nothing is not. + if kind == "required" and parent: + ignored_required.append((parent, a)) + return visited.add(a) data = book.get(a) if data is None: missing.append(a) return - deps = deps_of(data, kinds) + deps = deps_of(data, kinds) + _extra_deps_for(a) edges[a] = deps for d in deps: if d["external"]: @@ -1370,19 +1775,1144 @@ def walk_deps(book, anchor, kinds=("required", "recommended")): via.setdefault(d["external"], (a, d["kind"])) elif d["anchor"]: via.setdefault(d["anchor"], (a, d["kind"])) - visit(d["anchor"]) + visit(d["anchor"], a, d["kind"]) order.append(a) visit(anchor) + order = _dependencies_come_first(order, edges) + walk_deps.ignored_required = ignored_required return order, edges, missing, via -# --------------------------------------------------------------------------- # -# script generation -# --------------------------------------------------------------------------- # +# (dependant, dependency) pairs whose recommended edge was dropped to break a +# cycle -- the dependant is built BEFORE something it recommends. +_CYCLE_DROPS = [] + + +def _dependencies_come_first(order, edges): + """A real topological sort: every dependency before its dependant. + + This was three times a REPAIR PASS -- walk the list, notice a dependency + that landed too late, move it earlier -- and three times something moved + it back (1.12.33, 1.12.52, 1.12.68: texlive, then GLib, then libxcb). A + pass that repairs an order cannot guarantee one; it can only fix the + cases it happens to look at, and the next pass undoes them. + + So: Kahn's algorithm over the dependency graph. A node is emitted when + everything it depends on has been emitted, and nothing else can move it. + Ties keep the walk's original order, so the result stays stable and + readable. + + Cycles are real in BLFS (GLib recommends shared-mime-info, which requires + GLib). They are broken by strength: RECOMMENDED edges inside a cycle are + dropped first, because a recommendation is a preference and a requirement + is a fact. A cycle made only of required edges is the book's own knot -- + those nodes are emitted in their original order, and no reordering + pretends to have solved it. + """ + del _CYCLE_DROPS[:] + nodes = list(order) + present = set(nodes) + deps = {a: [(d.get("anchor"), d.get("kind")) + for d in edges.get(a, []) + if d.get("anchor") in present and d.get("anchor") != a] + for a in nodes} + + result, remaining = [], set(nodes) + while remaining: + ready = [a for a in nodes + if a in remaining + and not any(dep in remaining for dep, _k in deps[a])] + if ready: + for a in ready: + result.append(a) + remaining.discard(a) + continue + # Nothing is ready: something is in a cycle. BREAK ONLY INSIDE IT. + # + # "Fewest blocking dependencies" is not the same as "in the cycle": + # it picked ffmpeg, whose recommended edge to libass was then + # dropped, and ffmpeg was ordered at 179 with libass at 208 -- + # ERROR: libass >= 0.11.0 not found using pkg-config + # The cycle was elsewhere (libass -> harfbuzz -> freetype2 -> + # libass, after 1.13.23 made libass REQUIRE harfbuzz). + # + # So find the strongly-connected components of what is left, and + # drop recommended edges only where BOTH ends are in the same one. + # An edge into or out of a cycle is not what is holding it. + _idx, _low, _on, _stack, _sccs, _counter = {}, {}, set(), [], [], [0] + + def _strong(v): + _idx[v] = _low[v] = _counter[0] + _counter[0] += 1 + _stack.append(v) + _on.add(v) + for _w, _k in deps.get(v, []): + if _w not in remaining: + continue + if _w not in _idx: + _strong(_w) + _low[v] = min(_low[v], _low[_w]) + elif _w in _on: + _low[v] = min(_low[v], _idx[_w]) + if _low[v] == _idx[v]: + comp = [] + while True: + _w = _stack.pop() + _on.discard(_w) + comp.append(_w) + if _w == v: + break + if len(comp) > 1: + _sccs.append(set(comp)) + + import sys as _sys + _lim = _sys.getrecursionlimit() + _sys.setrecursionlimit(max(_lim, len(nodes) * 4 + 100)) + try: + for v in nodes: + if v in remaining and v not in _idx: + _strong(v) + finally: + _sys.setrecursionlimit(_lim) + + broke = False + for comp in _sccs: + for a in comp: + kept = [(dep, k) for dep, k in deps[a] + if not (dep in comp and k != "required")] + if len(kept) != len(deps[a]): + # SAY WHAT WAS GIVEN UP. + # + # Breaking the cycle is right, and silent. vala + # recommends graphviz; graphviz -> gegl -> babl -> librsvg + # -> vala closes the loop, so vala's edge was dropped and + # vala built FIRST -- without valadoc, and with configure + # stopping on "Package 'libgvc' not found". The order was + # the best available; not mentioning it was not. + for _dep, _k in deps[a]: + if _dep in comp and _k != "required": + _CYCLE_DROPS.append((a, _dep)) + deps[a] = kept + broke = True + if broke: + continue + # A REQUIRED-ONLY CYCLE IS THE BOOK'S KNOT -- but only the knot. + # + # This emitted EVERY remaining node in its original order, edges and + # all, so one unbreakable cycle threw away the ordering of everything + # after it: ffmpeg came out at 192 with libass, which it needs, at + # 205. Release just the smallest strongly-connected component and + # carry on sorting the rest. + _knot = min(_sccs, key=len) if _sccs else set() + if not _knot: + _knot = {next(a for a in nodes if a in remaining)} + for a in nodes: + if a in _knot: + result.append(a) + remaining.discard(a) + for a in list(remaining): + deps[a] = [(dep, k) for dep, k in deps[a] if dep not in _knot] + continue + return result + +def _active_command_text(*chunks): + """The command text with commented-out lines removed -- what will really + run.""" + out = [] + for c in chunks: + for line in (c or "").splitlines(): + t = line.strip() + if t.startswith("#"): + continue + out.append(line) + return "\n".join(out) + + +def _page_env_definitions(sec, names): + """Variable definitions the PAGE itself makes, for variables its build + commands use. + + TeX Live's build uses $TEXLIVE_PREFIX, and the same page sets it -- in + the block that writes /etc/profile.d/texlive.sh, further down: + TEXLIVE_PREFIX=/opt/texlive/2025 + The extractor takes install commands and drops that block, so the script + used a variable the page defines a few paragraphs later. A definition on + the page is the book's answer; asking the user to supply it again is + asking them to repeat the book. + + Only simple, self-contained assignments (no command substitution beyond + what the page itself spells out), and only for variables actually used. + """ + if not names: + return {} + # FOLLOW THE CHAIN. The book defines + # export XORG_PREFIX="/usr" + # export XORG_CONFIG="--prefix=$XORG_PREFIX --sysconfdir=/etc ..." + # and rejecting a value because it names another variable ("the same + # question moved") threw away an answer the same page gives. A value + # that depends on a variable we can ALSO find here is answered; only a + # chain that runs out is not. + names = set(names) + found = {} + # The definition may be on ANOTHER page: TeX Live's build uses + # $TEXLIVE_PREFIX and the value is set in "Setting the PATH for TeX Live" + # (anchor tex-path), which is its own section. The book is one document; + # search it whole, nearest section first. + scopes = [sec] + root = sec + while getattr(root, "parent", None) is not None: + root = root.parent + if root is not sec: + scopes.append(root) + for pre in [q for sc in scopes for q in sc.find_all("pre")]: + text = pre.get_text() + for n in names: + if n in found: + continue + m = re.search(r"^\s*(?:export\s+)?%s=([^\n]+(?:\\\n[^\n]+)*)$" + % re.escape(n), text, re.M) + if m: + # a value may continue over several lines with backslashes; + # a fragment ending in "\" is not a value, it is half of one + val = " ".join(x.strip() for x in + m.group(1).replace("\\\n", " ").splitlines()) + val = val.strip().strip("&").strip() + # a definition that needs another unknown variable is not an + # answer, it is the same question moved + # "<PREFIX>" and friends are placeholders in prose, not values + if "<" in val or ">" in val: + continue + found[n] = val + # pull in what those values reference, up to a few links + for _ in range(4): + need = set() + for v in found.values(): + need |= set(re.findall(r"\$\{?([A-Z][A-Z0-9_]{2,})", v)) + need -= set(found) + if not need: + break + names |= need + for pre in [q for sc in scopes for q in sc.find_all("pre")]: + text = pre.get_text() + for n in list(need): + m = re.search(r"^\s*(?:export\s+)?%s=([^\n]+(?:\\\n[^\n]+)*)$" + % re.escape(n), text, re.M) + if m and n not in found: + v = " ".join(x.strip() for x in + m.group(1).replace("\\\n", " ").splitlines()) + found[n] = v.strip().strip("&").strip() + # drop anything still referencing something we could not find: an + # unanswered chain is not an answer + for n in list(found): + unresolved = set(re.findall(r"\$\{?([A-Z][A-Z0-9_]{2,})", found[n])) - set(found) + if unresolved or found[n].endswith("\\") or '"' in found[n].rstrip('"'): + del found[n] + return found + + +def _required_env_guard(cmds, known): + """Variables the book expects the USER to set, checked before use. + + TeX Live's page says to `export TEXLIVE_PREFIX=...` first; the commands + then use it. Unset, the shell expands it to nothing and configure gets + error: expected an absolute directory name for --datarootdir: + followed by a tar reading "--strip-components=1" as a filename. An empty + expansion produces nonsense far from its cause, so the script now refuses + at the top of the phase and names the variable. + + Only variables the script does not set itself and that are not part of + its own vocabulary.""" + # SINGLE-QUOTED TEXT IS NOT THE SHELL'S. + # + # $PACKAGE_PREFIX_DIR and $SAVE_PACKAGE_PREFIX_DIR are CMake variables, + # written by a sed expression into a .cmake file: + # sed -e '/PACKAGE_INIT/i set(SAVE_PACKAGE_PREFIX_DIR "${PACKAGE_PREFIX_DIR}"... + # The shell never expands them -- they are inside single quotes -- so + # asking the user to set them was asking about another language's + # variables. Strip single-quoted spans before looking for names. + # A COMMENTED LINE IS NOT A COMMAND. + # + # OpenJDK's page uses $JT_JAVA only in its test block, which the generated + # script comments out -- and it EXPORTS the variable there itself: + # # export JT_JAVA=$(echo $PWD/build/*/jdk) && + # # jtreg/bin/jtreg -jdk:$JT_JAVA ... + # Scanning the commented text anyway produced a guard for a variable + # nothing live would ever read, and the whole plan refused to start until + # the user invented a value for it. Read only what will run. + _live = "\n".join(l for l in (cmds or "").splitlines() + if not l.strip().startswith("#")) + _scan = re.sub(r"'[^']*'", "''", _live) + names = set(re.findall(r"\$\{([A-Z][A-Z0-9_]{2,})\}", _scan)) + names |= set(re.findall(r"\$([A-Z][A-Z0-9_]{2,})\b", _scan)) + names -= set(known or ()) + # ...and a variable the commands SET is not one to ask the user for, + # whether the assignment is live or not: the page is telling us its value. + names -= set(re.findall(r"(?:^|\s|;)(?:export\s+)?([A-Z][A-Z0-9_]{2,})=", + cmds or "", re.M)) + # The shell's own variables are not the user's to set, and asking for + # them is worse than useless: a list containing EUID teaches the user to + # ignore the list. + names -= {"PATH", "HOME", "PWD", "OLDPWD", "SHELL", "USER", "LOGNAME", + "TERM", "MAKEFLAGS", "MAKELEVEL", "CFLAGS", "CXXFLAGS", + "LDFLAGS", "CPPFLAGS", "PKG_CONFIG_PATH", "LD_LIBRARY_PATH", + "BUILD_ROOT", "PM_TEST", "DESTDIR", "SHLVL", "IFS", "EUID", + "UID", "PPID", "RANDOM", "SECONDS", "LINENO", "HOSTNAME", + "BASH", "BASH_SOURCE", "BASH_VERSION", "FUNCNAME", "REPLY", + "PIPESTATUS", "OSTYPE", "MACHTYPE", "LANG", "LC_ALL", "TMPDIR"} + if not names: + return cmds + lines = [] + for n in sorted(names): + lines.append('\t[ -n "${%s:-}" ] || { echo "this package needs %s to be set ' + '(the book says to export it before building)" >&2; exit 1; }' + % (n, n)) + return "\n".join(lines) + "\n" + (cmds or "") + + +# Subprojects the book has NO PAGE for, per package. +# +# --wrap-mode=nofallback exists to stop a package installing its own copy of +# something another package owns (cairo once vendored freetype). That is only +# the right answer when the book HAS the dependency. gegl needs libnsgif and +# poly2tri-c; neither has a BLFS page, so no amount of planning can supply +# them and the flag simply makes gegl unbuildable -- a package the book builds +# with a plain `meson setup`. +# +# `--force-fallback-for` overrides the wrap mode for exactly the named +# subprojects, so the protection stays for everything else. Add an entry only +# after checking `blfs search <name>` finds nothing: if the book gains a page, +# the entry must go, and the failure box (1.14.39) covers anything not listed. +_FALLBACK_ALLOWED = { + "gegl": ("libnsgif", "poly2tri-c"), +} + + +def _allow_missing_subprojects(cmds, anchor): + allowed = _FALLBACK_ALLOWED.get((anchor or "").lower()) + if not allowed: + return cmds + out = [] + for line in (cmds or "").splitlines(): + if (re.search(r"\bmeson\s+setup\b", line) + and "force-fallback-for" not in line + and not line.strip().startswith("#")): + line = re.sub(r"(\bmeson\s+setup\b)", + r"\1 --force-fallback-for=" + ",".join(allowed), + line, count=1) + out.append(line) + return "\n".join(out) + + +def _no_vendored_subprojects(cmds): + """A package must not quietly build a dependency it could not find. + + cairo's meson fell back to its bundled subprojects and INSTALLED its own + freetype and fontconfig: + /usr/lib/libfreetype.so.6.20.1 p_cairo (and no libfreetype.so) + There was no p_freetype2 at all -- never built, because the planner then + saw a libfreetype on disk and considered it done. Every later + "cannot find -lfreetype" came from that one silent fallback: a vendored + copy is owned by the wrong package, carries no development symlink, and + hides the missing dependency instead of reporting it. + + --wrap-mode=nofallback makes meson FAIL instead, naming the dependency, + which is what the planner and the user need to see. It is added only + where it is not already specified.""" + out = [] + for line in (cmds or "").splitlines(): + if (re.search(r"\bmeson\s+setup\b", line) + and "wrap-mode" not in line and "wrap_mode" not in line + and not line.strip().startswith("#")): + line = re.sub(r"(\bmeson\s+setup\b)", r"\1 --wrap-mode=nofallback", line, count=1) + out.append(line) + return "\n".join(out) + + +# The tools the package-user system replaces with wrappers. Kept here +# because the generator has to know them: see _use_wrapped_tools. +_WRAPPED_TOOLS = ("ldconfig", "install", "chown", "chgrp", "chmod", "mkdir", "cp") + + +def _use_wrapped_tools(cmds): + r"""Call wrapped tools by NAME, so the wrappers are actually used. + + The wrappers live in a directory on the package user's PATH, which means + an ABSOLUTE path walks straight past them. The book's Xorg loop says + as_root /sbin/ldconfig + so the package user ran the real ldconfig and hit + Renaming of /etc/ld.so.cache~ to /etc/ld.so.cache failed + with the wrapper sitting unused. This is not specific to ldconfig: any + of the wrapped tools written as /sbin/x or /usr/bin/x bypasses the whole + mechanism. + """ + out = [] + for line in (cmds or "").splitlines(): + if not line.strip().startswith("#"): + for tool in _WRAPPED_TOOLS: + line = re.sub(r"(?<![\w/])/(?:usr/)?s?bin/%s\b" % tool, tool, line) + out.append(line) + return "\n".join(out) + + +def _define_as_root(cmds): + r"""Define as_root when the book's commands use it. + + The multi-package pages call `as_root make install` inside their loop, + and define as_root() in a NOTE -- prose, which is kept as a comment. So + the loop called a function that did not exist: + install_xorg7-lib: line 198: as_root: command not found + + In this model there is nothing to elevate: the package user owns the + directories it installs into, and the wrappers handle what it may not do. + So as_root simply runs the command.""" + if not re.search(r"^[^#\n]*\bas_root\b", cmds or "", re.M): + return cmds + return ('\tas_root() { "$@"; } # the package user installs its own files\n' + + (cmds or "")) + + +def _unwrap_interactive_subshell(cmds): + r"""Drop the book's "start a subshell and paste this" wrapper. + + Xorg Libraries (and the other multi-package pages) say: + bash -e + for package in $(...); do ... done + exit + A person types `bash -e`, pastes the loop, then `exit`. A SCRIPT that + runs `bash -e` gets a shell reading its own stdin -- which is the + terminal -- and waits there forever: + bash install_xorg7-lib all + \_ bash -e <- idle, holding the build + The loop belongs in the phase itself, which already runs under set -e, + so the wrapper line and its matching `exit` are removed. + """ + lines = (cmds or "").splitlines() + if not any(re.match(r"^\s*bash\s+-e\s*$", l) for l in lines): + return cmds + out, dropped_open = [], False + for l in lines: + if not dropped_open and re.match(r"^\s*bash\s+-e\s*$", l): + out.append("## the book starts a subshell here and pastes the rest into it;") + out.append("## this phase IS that shell, and already runs under set -e") + dropped_open = True + continue + if dropped_open and re.match(r"^\s*exit\s*$", l): + out.append("## (the book's matching 'exit' -- leaving it would end the phase)") + continue + out.append(l) + return "\n".join(out) + + +def _comment_typeset_doc_builds(cmds): + r"""Comment commands that TYPESET documentation. + + libassuan's script ran `make -C doc pdf ps`, which needs a TeX + installation this desktop deliberately does not build: + You don't have a working TeX binary (tex) installed anywhere in + your PATH, and texi2dvi cannot proceed without one. + A --disable-doc flag cannot help -- the command asks for PDF by name. + + pdf/ps/dvi are TYPESET formats: no package installs itself with them, so + a make invoking those targets is documentation and nothing else. html, + info and man are left alone: they need no TeX and packages do install + them. + """ + out = [] + for line in (cmds or "").splitlines(): + stripped = line.strip() + if (not stripped.startswith("#") + and re.match(r"^\s*make\b", stripped) + and re.search(r"\b(pdf|ps|dvi)\b", stripped) + and not re.search(r"\b(install|check)\b", stripped)): + out.append("## SKIPPED -- typesets documentation (pdf/ps/dvi need TeX," + " which this system does not build)") + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +# No leading \b: there is no word boundary before "./all.sh", so an outer +# \b silently killed that alternative -- the regex compiled, matched the +# make-based drivers, and let NSS's ./all.sh through. +_FUNCDEF_RE = re.compile(r"^\s*\w[\w:-]*\s*\(\)\s*\{") + + +def _is_definition(line): + """A line that DEFINES something is never a command to skip. + + Xorg's loop defines do_build/do_test/do_install and calls them later. + The test filter commented `do_test() { make check; }`, so the loop called + a function that did not exist -- and the doc and pip filters had the same + flaw, waiting for a page that defines a function around those commands. + """ + return bool(_FUNCDEF_RE.match(line)) + + +_TEST_DRIVER_RE = re.compile( + # A TEST RUNNER IS A COMMAND, NOT A SUBSTRING. + # + # `\bctest\b` matched inside a brace list -- cargo-c installs its + # binaries with + # install -vm755 target/release/cargo-{capi,cbuild,cinstall,ctest} /usr/bin/ + # and a comma is a word boundary, so the one line that installs anything + # was commented as "runs the test suite". cargo-c then built and + # installed NOTHING, librsvg died on `Program 'cargo-cbuild' not found`, + # and verify was right to call cargo-c not installed. + # The lookbehind rejects a name that is part of a word, a path or a list. + r"(\bmake\s+(check|test)\b|(?<![\w./,{-])ctest\b|\bmeson\s+test\b|\bninja\s+test\b" + r"|(?<![\w./,{-])pytest\b|\./all\.sh\b|\./runtests\b|\bmake\s+-C\s+\S*tests?\b" + # Rust drives everything through x.py: `./x.py test` is its test suite + # (41425 tests, and the log-reading grep after it fails when they are + # skipped). `./x.py build` and `./x.py install` must NOT match. + r"|\bx\.py\s+test\b|\bcargo\s+test\b|\bgo\s+test\b" + # ffmpeg's suite is `make fate`, with `make fate-rsync` to fetch its + # samples first -- 40 GB of them, over rsync, during an install. + r"|\bmake\s+(?:\S+=\S+\s+)*fate(-\w+)?\b" + # ...and the standalone fetch of the samples, which is its own statement: + # rsync -vrltLW --delete ... rsync://fate-suite.ffmpeg.org/... + # Commenting `make fate-rsync` left this running, and rsync is not even + # installed: install_ffmpeg: line 242: rsync: command not found + r"|\brsync\b[^\n]*\bfate\b|\brsync://)") + + +_TEST_LOG_RE = re.compile(r"\btee\s+(\S*testlog\S*)|\b(\S*testlog\S*)\b") + + +# What makes a pushd group "a test suite being built" rather than part of the +# build: the names of the suites the books actually fetch. +_TEST_SETUP_RE = re.compile( + r"\b(piglit|xts|xtest|x-test-suite|test-?suite|fate-suite|jtreg|" + r"XTEST_DIR|PIGLIT_DIR)\b", re.I) + + +def _comment_test_runs(cmds): + r"""Running the test suite is not building the package. + + NSS's page puts its tests in the BUILD block: + cd tests && + HOST=localhost DOMSUF=localdomain ./all.sh + cd ../ + so `packagemanager install nss` ran 77549 tests -- hours -- and then + failed the package on a test artifact. The script already HAS a test + phase (test_pkg, run only with PM_TEST=1); tests belong there. + + The `cd tests &&` that leads into it is commented with the run, and the + `cd ..` that leads back out, or the chain would break. + """ + lines = (cmds or "").splitlines() + out, i = [], 0 + while i < len(lines): + line = lines[i] + # A FUNCTION DEFINITION IS NOT A TEST RUN. + # + # Xorg's loop defines `do_test() { make check; }` and calls it later; + # commenting the DEFINITION left the loop calling a function that did + # not exist. Only a command that RUNS the suite is commented, never a + # line that defines something. + if _is_definition(line): + out.append(line) + i += 1 + continue + if line.strip().startswith("#") or not _TEST_DRIVER_RE.search(line): + out.append(line) + i += 1 + continue + # the cd that led in (already emitted) goes with it + while out and re.match(r"^\s*cd\s+\S*tests?\S*\s*&&\s*$", out[-1]): + out[-1] = "# " + out[-1] + break + # ...AND SO DOES THE BLOCK THAT BUILT THE TEST SUITE. + # + # Xwayland's page fetches piglit and the X Test Suite, builds them + # against a temporary Xvfb, and only then runs `ninja test`. All of + # that sits in the BUILD block, so commenting the run alone left the + # setup to run on every ordinary build: two git clones, an X server, + # and finally + # install_Xwayland-24.1.9: line 54: cd: xts: No such file or + # directory ... Fatal server error + # The setup is bracketed by pushd/popd, so the whole group can be + # taken -- but only when it is recognisably a test suite being built, + # never a plain pushd group that happens to precede a test. + _j = len(out) - 1 + while _j >= 0 and not out[_j].strip(): + _j -= 1 + if _j >= 0 and re.match(r"^\s*popd\s*(&&)?\s*$", out[_j]): + _k = _j + while _k >= 0 and not re.match(r"^\s*(mkdir\s+\S+\s*&&\s*)?pushd\b", + out[_k]): + _k -= 1 + if _k >= 0: + # `mkdir tools` and `pushd tools` are often separate lines; + # the directory is made only for the suite, so it goes too + _m = _k - 1 + while _m >= 0 and not out[_m].strip(): + _m -= 1 + if _m >= 0 and re.match(r"^\s*mkdir\s+\S+\s*(&&)?\s*$", out[_m]): + _k = _m + _span = "\n".join(out[_k:_j + 1]) + if _TEST_SETUP_RE.search(_span): + for _n in range(_k, _j + 1): + if out[_n].strip() and not out[_n].strip().startswith("#"): + out[_n] = "# " + out[_n] + out.insert(_k, "## SKIPPED -- builds the test suite that the" + " commented run below uses") + # A MATCH ON A CONTINUATION LINE BELONGS TO THE COMMAND ABOVE IT. + # + # ffmpeg's sample fetch is + # rsync -vrltLW --delete ... \ + # rsync://fate-suite.ffmpeg.org/... fate-suite/ + # and only the SECOND line matches, so the first stayed active and + # ran rsync with no arguments to speak of. + while out and re.search(r"\\\s*$", out[-1]) and not out[-1].lstrip().startswith("#"): + out[-1] = "# " + out[-1] + out.append("## SKIPPED -- runs the test suite; this script has a test " + "phase for that (PM_TEST=1)") + out.append("# " + line) + # whatever reads the log it would have written goes too: rust pipes + # the run into rustc-testlog and greps it on the next lines + _logs = set(m for m in _TEST_LOG_RE.findall(line) for m in m if m) + i += 1 + while i < len(lines) and _logs and any(l in lines[i] for l in _logs): + out.append("# " + lines[i]) + i += 1 + # a pipeline continues over lines: comment the rest of it, or the + # `awk` after a commented `grep ... |` runs on its own + while (out and re.search(r"(\||&&|\\)\s*$", out[-1]) + and i < len(lines) and lines[i].strip()): + out.append("# " + lines[i]) + i += 1 + # and the cd back out + if i < len(lines) and re.match(r"^\s*cd\s+\.\.?/?\s*&*\s*$", lines[i]): + out.append("# " + lines[i]) + i += 1 + return "\n".join(out) + + +_TYPESET_RE = re.compile( + r"\b(texi2pdf|texi2dvi|dvips|dvipdf|pdflatex|dblatex|latex)\b") + + +def _comment_typeset_blocks(cmds): + r"""A loop that typesets the manuals needs TeX; skip the whole block. + + ffmpeg's page ends its build with + pushd doc && + for DOCNAME in `basename -s .html *.html` + do + texi2pdf -b $DOCNAME.texi && + texi2dvi -b $DOCNAME.texi && + dvips -o $DOCNAME.ps $DOCNAME.dvi + done && + which fails on a system that deliberately has no TeX. Commenting only + the texi2* lines would leave an empty loop body, so the block goes as a + unit: from the `pushd`/`for` that opens it to the `done`/`popd` that + closes it. + """ + lines = (cmds or "").splitlines() + out, i = [], 0 + while i < len(lines): + line = lines[i] + if line.strip().startswith("#") or not re.match(r"^\s*(pushd|for)\b", line): + out.append(line) + i += 1 + continue + # find the end of this block, and whether it typesets anything + j, depth, typesets = i, 0, False + while j < len(lines): + if re.match(r"^\s*(for|while|until)\b", lines[j]): + depth += 1 + if re.match(r"^\s*done\b", lines[j]): + depth -= 1 + if _TYPESET_RE.search(lines[j]): + typesets = True + if depth <= 0 and j > i and re.match(r"^\s*(done|popd)\b", lines[j]): + break + j += 1 + if not typesets or j >= len(lines): + out.append(line) + i += 1 + continue + # swallow a trailing popd too, and any && that chains on + while j + 1 < len(lines) and re.match(r"^\s*popd\b", lines[j + 1]): + j += 1 + out.append("## SKIPPED -- typesets the manuals (texi2pdf/dvips need TeX); " + "the package itself does not need them") + for k in range(i, j + 1): + out.append("# " + lines[k]) + i = j + 1 + return "\n".join(out) + + +def _comment_doc_format_builds(cmds): + r"""Comment `make ... pdf/ps/dvi` -- the book's OPTIONAL manuals. + + libassuan's page ends with + make -C doc pdf ps + which typesets the manual and needs a TeX installation. --disable-doc + does not help (libassuan has no such switch; autotools accepts unknown + --disable-* silently), because nothing is misconfigured: the command + explicitly asks for PDF. + + A library is not installed by typesetting its manual, so these lines are + emitted commented, with the book's own command one uncomment away. info + and html builds are left alone -- they need only texinfo, which LFS has. + """ + out = [] + for line in (cmds or "").splitlines(): + if (not line.strip().startswith("#") and not _is_definition(line) + and re.search(r"\bmake\b[^#\n]*\s(pdf|ps|dvi)\b", line)): + out.append("## SKIPPED -- typesets the manual (needs TeX); the " + "package itself does not need it") + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +_PROFILE_RUN_RE = re.compile( + r"^\s*(source|\.)\s+/etc/profile(\.d/\S+)?\s*$|^\s*/etc/profile\.d/\S+\.sh\s*$") + + +# `make install-<name>` in a CONFIGURATION section is the SysV books' idiom +# for installing a boot script, and the target lives in the blfs-bootscripts +# package -- not in the package being built. Run as-is it says: +# make: *** No rule to make target 'install-bluetooth'. Stop. +# after everything else has installed correctly, which turns a finished build +# into a failure. The systemd books have no such sections, so this only +# appears once a SysV book is in use. +_BOOTSCRIPT_RE = re.compile(r"^\s*make\s+(?:-C\s+\S+\s+)?install-([A-Za-z0-9_.+-]+)\s*$") +_AUTOMAKE_INSTALL_TARGETS = { + "strip", "exec", "data", "html", "pdf", "ps", "dvi", "info", "man", + "exec-am", "data-am", "dist", "recursive", +} + + +# "You can test your installation ..." -- prose that introduces an EXAMPLE, +# not a configuration step. enchant's page ends with a spell-check demo that +# needs an aspell dictionary; run as a configure phase it failed with +# No dictionary available for 'en_GB' +# after the package had installed perfectly. A demonstration is not part of +# installing something. +_CONFIG_TEST_INTRO_RE = re.compile( + r"you (can |may )?(now )?(test|verify|try)\b|to test (your|the) install" + r"|test your installation", re.I) + + +def _comment_config_tests(cmds): + out, skipping = [], False + for line in (cmds or "").splitlines(): + st = line.strip() + if st.startswith("#"): + if _CONFIG_TEST_INTRO_RE.search(st): + if not skipping: + out.append("## SKIPPED -- the book offers this as a way to" + " TEST the installation,") + out.append("## not as a step of it. Run it by hand if you" + " want to check the package.") + skipping = True + elif skipping and st not in ("#", "##"): + skipping = False # a new paragraph ends the example + out.append(line) + continue + if skipping and st: + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +def _comment_bootscript_installs(cmds, anchor=""): + out = [] + for line in (cmds or "").splitlines(): + m = _BOOTSCRIPT_RE.match(line) + # install-strip, install-exec, install-data and friends are REAL + # automake targets in the package itself -- only a name that is not + # one of those is a boot script. + if m and m.group(1) in _AUTOMAKE_INSTALL_TARGETS: + m = None + if m and not line.strip().startswith("#"): + out.append("## SKIPPED -- installs the '%s' boot script, whose " + "target lives in blfs-bootscripts," % m.group(1)) + out.append("## not in this package. Install it from there:") + out.append("## packagemanager install blfs-bootscripts --run") + out.append("## cd <blfs-bootscripts source> && %s" + % line.strip()) + out.append("# " + line.strip()) + continue + out.append(line) + return "\n".join(out) + + +def _comment_profile_sourcing(cmds): + r"""Writing a login profile is not running it. + + Rust's configuration writes /etc/profile.d/rustc.sh and the page then + sources it. Those snippets call pathprepend/pathappend -- helpers from + the BLFS bash startup files, absent from a build shell -- so a package + that had built and installed 2390 files failed at: + /etc/profile.d/rustc.sh: line 3: pathprepend: command not found + + The file is written; it takes effect at the next login, which is what it + is for. Sourcing it here tests the login shell, not the package. + """ + out = [] + for line in (cmds or "").splitlines(): + if not line.strip().startswith("#") and _PROFILE_RUN_RE.match(line): + out.append("## SKIPPED -- sourcing a login profile; the file is " + "written and applies at next login") + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +_LIST_LOOP_RE = re.compile( + r"^\s*for\s+\w+\s+in\s+\$\(.*?(\.\./[\w.-]+|\b[\w-]+\.md5)\b.*?\)") + + +def _guard_list_loops(cmds): + r"""A loop over a list file must fail when the file is missing. + + Xorg's build is + for package in $(grep -v '^#' ../lib-7.md5 | awk '{print $2}') + and when ../lib-7.md5 is not there, grep prints an error, the loop runs + ZERO times, and the `for` exits 0 -- so a package that built nothing was + recorded as installed. Twice, before anyone noticed. + + A guard in front of the loop turns that into a failure with a name. + """ + out = [] + for line in (cmds or "").splitlines(): + m = (None if line.strip().startswith("#") + else _LIST_LOOP_RE.match(line)) + if m: + f = m.group(1) + out.append('[ -s "%s" ] || { echo "the list this build loops over ' + 'is missing or empty: %s" >&2; exit 1; }' % (f, f)) + out.append(line) + return "\n".join(out) + + +def _comment_orphan_doc_installs(cmds, context=""): + r"""`make install-man` when nothing active ran `make man`. + + Git's page generates the man pages and html docs under "If you have + installed asciidoc-10.2.1 and xmlto-0.0.29 you can create ..." -- which + the conditional guard skips -- and then installs them unconditionally: + make install-man + make htmldir=/usr/share/doc/git-2.53.0 install-html + So the install tried to build them itself and died on + /bin/sh: line 1: asciidoc: command not found + after everything else had installed. + + Fourth in this family (libuv's man page, the orphan log reader, brotli's + wheel): whenever the guard comments a block, something may still consume + what that block would have produced. + """ + lines = (cmds or "").splitlines() + active = "\n".join(l for l in lines if not l.strip().startswith("#")) + if context: + active += "\n" + "\n".join(l for l in context.splitlines() + if not l.strip().startswith("#")) + # A BARE `make` BUILDS THE PACKAGE -- BUT NOT ITS MANUALS. + # + # Whois builds with `make` and installs with `make prefix=/usr + # install-whois`; there is no `make whois` target to look for, so all + # three of its install lines were commented and it installed nothing. + # But git ALSO has a bare `make`, and git's manuals still need `make man` + # -- which is exactly the conditional block this filter was written for. + # So a bare make covers ordinary targets and never the documentation ones. + _bare_make = bool(re.search(r"^\s*make\s*(?:\S+=\S+\s*)*$", active, re.M)) + _DOC_TARGETS = {"man", "html", "info", "pdf", "ps", "dvi", "doc", "docs", + "manual", "manuals"} + # `make htmldir=/usr/... install-html`: the book puts variable assignments + # between make and the target, so allow them. + pat = re.compile(r"\bmake\s+(?:\S+=\S+\s+)*(?:-C\s+\S+\s+)?install-(\w+)") + out, i = [], 0 + while i < len(lines): + line = lines[i] + m = (None if line.strip().startswith("#") or _is_definition(line) + else pat.search(line)) + if not m: + out.append(line) + i += 1 + continue + what = m.group(1) + if (_bare_make and what not in _DOC_TARGETS) \ + or re.search(r"\bmake\s+(?:\S+=\S+\s+)*(?:-C\s+\S+\s+)?%s\b" + % re.escape(what), active): + out.append(line) # something really does build it + i += 1 + continue + out.append("## SKIPPED -- installs '%s', which nothing here builds " + "(the book generates it in an optional step)" % what) + out.append("# " + line) + # The rest of ITS chain goes too: git follows install-html with + # `mkdir -vp .../html && mv ...`, moving files that only install-html + # would have put there. + i += 1 + while (re.search(r"&&\s*$", out[-1]) + and i < len(lines) and lines[i].strip()): + out.append("# " + lines[i]) + i += 1 + return "\n".join(out) + +def _comment_orphan_tree_shuffles(cmds): + r"""Rearranging a directory that only a SKIPPED command fills. + + After git's optional "untar the html docs" block is commented, the page + still reorganises what it would have extracted: + mkdir -vp /usr/share/doc/git-2.53.0/man-pages/{html,text} && + mv /usr/share/doc/git-2.53.0/{git*.adoc,man-pages/text} && + and the mv finds nothing: + mv: cannot stat '/usr/share/doc/git-2.53.0/git*.adoc' + + A `mv` (or find/sed) whose source directory appears ONLY in commented + commands is moving something nobody produced. `mkdir` does not count as + filling a directory -- it makes it empty. + """ + lines = (cmds or "").splitlines() + commented, active = [], [] + for l in lines: + (commented if l.strip().startswith("#") else active).append(l) + ctext, atext = "\n".join(commented), "\n".join(active) + + def _fills(d): + """Does an ACTIVE command put anything INTO d (not just create it)?""" + for l in active: + if d not in l: + continue + if re.match(r"^\s*mkdir\b", l): + continue # creating it is not filling it + # mv/find/sed REARRANGE a directory; they do not fill it. Only + # something that brings files in counts. + if re.match(r"^\s*(mv|find|sed)\b", l): + continue + if re.search(r"\b(tar|cp|install|make|ninja)\b", l): + return True + return False + + out, i = [], 0 + while i < len(lines): + line = lines[i] + if (line.strip().startswith("#") or _is_definition(line) + or not re.match(r"^\s*(mv|find|sed)\b", line)): + out.append(line) + i += 1 + continue + dirs = [d.rstrip("/") for d in + re.findall(r"(/[A-Za-z0-9._/+-]{6,})", line)] + orphan = False + for d in dirs: + base = d.rstrip("/").rsplit("/", 1)[0] if "{" not in d else d + for probe in (d, base): + if probe and probe in ctext and not _fills(probe): + orphan = True + break + if orphan: + break + if not orphan: + out.append(line) + i += 1 + continue + out.append("## SKIPPED -- rearranges files that only a skipped step " + "would have put there") + out.append("# " + line) + i += 1 + # and the rest of its chain, plus the mkdir that led into it + while out and re.search(r"&&\s*$", out[-1]) and i < len(lines) and lines[i].strip(): + out.append("# " + lines[i]) + i += 1 + for j in range(len(out) - 1, -1, -1): + if out[j].startswith("#"): + continue + if re.match(r"^\s*mkdir\b", out[j]) and re.search(r"&&\s*$", out[j]): + out[j] = "# " + out[j] + break + return "\n".join(out) + + +def _repair_dangling_operators(cmds): + r"""Commenting a command must not leave the previous one dangling. + + libpwquality's install ended: + make install && + # pip3 install ... --find-links dist pwquality <- commented (1.13.3) + and bash refused the whole file: + syntax error near unexpected token `}' + + Every filter that comments a line can do this, so the repair belongs at + the end, once: an ACTIVE line ending in && || or | with no active line + after it loses the operator. + """ + lines = (cmds or "").splitlines() + for i, line in enumerate(lines): + if line.strip().startswith("#") or not re.search(r"(&&|\|\||\|)\s*$", line): + continue + nxt = next((l for l in lines[i + 1:] + if l.strip() and not l.strip().startswith("#")), None) + if nxt is None: + lines[i] = re.sub(r"\s*(&&|\|\||\|)\s*$", "", line) + return "\n".join(lines) + + +def _comment_orphan_pip_installs(cmds, context=""): + r"""A pip install whose wheel nothing built. + + brotli's page builds optional Python bindings in two steps: + pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir $PWD + pip3 install --no-index --find-links dist Brotli + The first is conditional ("if you want the bindings") and is commented + out; the second was not, so pip looked in a directory nobody had made: + WARNING: Location 'dist' is ignored: ... non-existing path + ERROR: No matching distribution found for Brotli + -- after the C library had installed perfectly. + + An install that reads from a --find-links directory no ACTIVE command + fills is commented with its producer. + + CONTEXT MATTERS. The producer and the consumer are usually on one page + but NOT in one block: for a Python module the wheel is built in the build + phase and installed in the install phase, and judging the install block + alone found no producer and commented the only line that installs + anything. Every python module page came out as "build a wheel, install + nothing" -- pathspec's manifest then held one path, /var/mail/p_pathspec, + created by useradd, which was enough to call it installed, and Hatchling + failed on `No module named 'pathspec'` twenty-two packages later. + """ + lines = (cmds or "").splitlines() + active = "\n".join(l for l in lines if not l.strip().startswith("#")) + if context: + active += "\n" + "\n".join(l for l in context.splitlines() + if not l.strip().startswith("#")) + out = [] + for line in lines: + m = (None if line.strip().startswith("#") or _is_definition(line) + else re.search(r"--find-links[= ]+(\S+)", line)) + if m: + d = m.group(1) + builds = re.search(r"(pip3?\s+wheel[^\n]*-w[= ]+%s\b" + r"|mkdir[^\n]*\b%s\b)" % (re.escape(d), re.escape(d)), + active) + if not builds: + out.append("## SKIPPED -- installs from '%s', which nothing here " + "builds (the wheel step is optional)" % d) + out.append("# " + line) + continue + out.append(line) + return "\n".join(out) + + +def _comment_orphan_log_readers(cmds): + """Comment lines that read a log no ACTIVE command writes. + + Xorg Libraries ends with + grep -A9 summary *make_check.log + which inspects the test logs -- and the line that WRITES them + (make check ... | tee ../$package-make_check.log) is part of the optional + test step, correctly commented out. Under set -e the orphan grep then + failed the whole page: + grep: *make_check.log: No such file or directory + + Same shape as the libuv man page (1.12.19), one layer earlier: a command + whose input nothing produces is not part of installing. Only .log files + are treated this way -- a build artifact with a real name might be + produced by make itself, and guessing there would do harm.""" + lines = (cmds or "").splitlines() + active = [l for l in lines if not l.strip().startswith("#")] + writes_log = any(re.search(r"(tee|>|>>)\s*\S*\.log", l) for l in active) + if writes_log: + return cmds + out = [] + for line in lines: + if (not line.strip().startswith("#") + and re.search(r"\S*\.log\b", line) + and re.match(r"\s*(grep|cat|less|tail|head|awk|sed)\b", line)): + out.append("## SKIPPED -- reads a log that nothing here writes " + "(it belongs to the optional test step)") + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +def _comment_own_program_demos(cmds, installed): + """Comment out BUILD lines that run a program this package installs. + + The book demonstrates the result -- "You can now test the rendering ... To + do this, run the following command: gs -q -dBATCH .../tiger.eps" -- and + that command lands in the build, where the program does not exist yet: + install_gs: line 118: gs: command not found + It was harmless while failures were ignored; with the dispatcher honouring + set -e it fails the package. Chasing the wording ("To do this...") is the + mistake this file keeps making; the structural fact is that a package + cannot run its own not-yet-installed program as part of building itself. + """ + progs = set() + for chunk in (installed or {}).values(): + for tok in re.split(r"[,\s]+", str(chunk)): + tok = tok.strip() + if tok and re.fullmatch(r"[A-Za-z0-9._+-]+", tok): + progs.add(tok) + if not progs: + return cmds + out = [] + for line in (cmds or "").splitlines(): + stripped = line.strip() + first = stripped.split()[0] if stripped else "" + if (first in progs and not stripped.startswith("#") + and not re.match(r"^(make|ninja|meson|cmake)\b", stripped)): + out.append("## SKIPPED -- runs %s, which this package has not " + "installed yet (the book is demonstrating the result)" + % first) + out.append("# " + line) + else: + out.append(line) + return "\n".join(out) + + +def _needed_additional_links(data, emitted_lines): + """Of the page's additional downloads, the ones an ACTIVE command uses. + + A file nothing runs is not fetched (optional doc tarballs, test suites -- + their commands are emitted commented), and a file an active command needs + IS fetched whatever the book called it. Patches are always kept: the + command that applies one is `patch -Np1 -i ../x.patch`, which does name + it, but a patch missed for any reason is a silent mis-build.""" + links = data.get("additional_links", []) or [] + if not links: + return [] + # the PHASED text only: data["install_cmds"] is the raw page, where the + # conditional blocks are still live and would vote for their own + # downloads + active = _active_command_text(data.get("build_cmds", ""), + data.get("install_root_cmds", "")) + keep = [] + for href in links: + base = href.rstrip("/").split("/")[-1] + if href.endswith(".patch") or (base and base in active): + keep.append(href) + return keep + + +def bash_str(v): + """A book string as a bash literal that the shell will NOT interpret. + + These values are PROSE lifted out of the book -- "Installed Directories: + $TEXLIVE_PREFIX/bin, $TEXLIVE_PREFIX/lib" -- and they were emitted inside + double quotes, so the shell expanded them. pm-install sources the script + to read its metadata, and under set -u that killed the install before it + started: + install_texlive-20250308-source: line 25: TEXLIVE_PREFIX: unbound variable + A dollar sign in documentation is not a variable. Single quotes, with + the only escape single quoting needs.""" + return "'" + str(v).replace("'", "'\\''") + "'" + def bash_array(items): - return "(" + " ".join(f"'{i}'" for i in items) + ")" + return "(" + " ".join(bash_str(i) for i in items) + ")" def extract_info(sec, is_module): @@ -1455,6 +2985,11 @@ def extract_package_data(sec, is_module, group, anchor): "recommended": _xref_titles(sec, "recommended"), "optional": _xref_titles(sec, "optional"), "installed_content": extract_installed_content(sec), + # variables the PAGE defines for its own commands (see + # _page_env_definitions): collected here, where the section is in + # hand, and used by render_script + "page_env": _page_env_definitions( + sec, set(re.findall(r"\$\{?([A-Z][A-Z0-9_]{2,})", build_cmds or ""))), "install_cmds": install_cmds, "build_cmds": build_cmds, "install_root_cmds": install_root, @@ -1464,11 +2999,112 @@ def extract_package_data(sec, is_module, group, anchor): } -def render_script(data, source_url): - """Turn cached package data into a phased install_<name-version> script: - unpack -> build -> install -> configure, each a separate function with a - dispatcher, so a failed install (e.g. permissions) can be fixed and re-run - WITHOUT recompiling: bash install_<pkg> install""" +def _drop_orphaned_variant_install(cmds): + """An install step whose build variant was commented out. + + WebKitGTK's page gives two builds, each with its own `ninja install`. One + build is skipped, but its install line stayed live, so the install phase + ran `ninja install` twice. The second is a no-op rather than a second + build -- but it reads exactly like one, and someone watching a package + that takes hours has no way to tell. An identical command repeated after + a skipped block belongs to that block. + """ + # ONLY THE PATTERN "X ; <skipped variant> ; X". + # + # A first attempt commented any repeat after a skipped block, and dropped + # GLib's LAST `ninja install` -- the one that installs the reconfigured + # build with introspection enabled, several live commands later. A repeat + # is only redundant when nothing happened in between: the command + # immediately before the skipped block and the one immediately after it + # must be the same. + lines = (cmds or "").split("\n") + out, last_active, pending_skip = [], None, False + for line in lines: + st = line.strip() + if st.startswith("## OPTIONAL, skipped"): + pending_skip = True + out.append(line) + continue + if not st or st.startswith("#"): + out.append(line) + continue + if pending_skip and last_active is not None and st == last_active: + out.append("## SKIPPED -- the same command again, belonging to the " + "build variant commented out above.") + out.append("# " + st) + pending_skip = False + continue + last_active = st + pending_skip = False + out.append(line) + return "\n".join(out) + + +def _enable_one_alternative(cmds): + """A page that offers a CHOICE must still build something. + + WebKitGTK's page gives two complete builds -- "If you want to install the + GTK-3 version, run the following commands" and the same for GTK-4 -- and + each is introduced by a conditional, so `_conditional_guard` commented + BOTH. The build phase came out as `:` and the install phase then ran + `ninja install` against a directory with no build.ninja in it. + + When nothing active is left but a skipped block exists, the first block is + restored: a package that builds one of two ways is not a package that + builds no ways. The others stay commented, with a line saying so, because + which one is wanted is the reader's decision, not ours. + """ + lines = (cmds or "").split("\n") + if any(l.strip() and not l.strip().startswith("#") and l.strip() != ":" + for l in lines): + return cmds + starts = [i for i, l in enumerate(lines) + if l.strip().startswith("## OPTIONAL, skipped")] + if not starts: + return cmds + i = starts[0] + end = starts[1] if len(starts) > 1 else len(lines) + out = lines[:i] + out.append("## Every build block on this page is conditional, so none " + "would have run.") + out.append("## ENABLED HERE: %s" + % lines[i].split("the book says:", 1)[-1].strip()[:110]) + out.append("## If you want one of the page's other variants instead, " + "comment these") + out.append("## lines and uncomment that block.") + for l in lines[i + 1:end]: + st = l.strip() + if st.startswith("##"): + out.append(l) + elif st.startswith("#"): + out.append(l.replace("#", "", 1).lstrip() if st != "#" else "") + else: + out.append(l) + out.extend(lines[end:]) + return "\n".join(out) + + +def _fn_body(cmds, empty=": # (nothing to do in this phase)"): + """A command block as a function body. NOT indented: the book is full of + heredocs, and an indented terminator is no terminator. A body of comments + alone is a syntax error to bash, so it gets a command.""" + if not cmds or not cmds.strip(): + return empty + body = cmds.strip("\n") + if not any(ln.strip() and not ln.lstrip().startswith("#") + for ln in body.split("\n")): + body += "\n:" + return body + + +def render_script(data, source_url, book_version=""): + """Turn cached package data into an install_<name-version> script: the + book's variables at the top, its commands in build_pkg / install_pkg / + configure_pkg, and one line at the end that hands over to the shared + runner (lfs-phases), which does the rest -- fetch, unpack, run each phase + as its own process, resume in the build dir, the dispatcher. A failed + install can be fixed and re-run WITHOUT recompiling: + bash install_<pkg> install""" name_version = data["name_version"] pkg = data["pkg"] build_cmds = data.get("build_cmds") or data["install_cmds"] @@ -1482,6 +3118,97 @@ def render_script(data, source_url): lambda m: m.group(0) + ' ${PM_REINSTALL:+--force-reinstall}', cmds or "") build_cmds = _pip_reinstall(build_cmds) + build_cmds = _comment_own_program_demos(build_cmds, + data.get("installed_content")) + build_cmds = _unwrap_interactive_subshell(build_cmds) + install_root = _unwrap_interactive_subshell(install_root) + build_cmds = _use_wrapped_tools(build_cmds) + install_root = _use_wrapped_tools(install_root) + build_cmds = _define_as_root(build_cmds) + install_root = _define_as_root(install_root) + build_cmds = _comment_typeset_doc_builds(build_cmds) + install_root = _comment_typeset_doc_builds(install_root) + config_cmds = _comment_profile_sourcing(data.get("config_cmds") or "") + config_cmds = _comment_bootscript_installs(config_cmds, data.get("name")) + config_cmds = _comment_config_tests(config_cmds) + data["config_cmds"] = config_cmds + build_cmds = _comment_profile_sourcing(build_cmds) + install_root = _comment_profile_sourcing(install_root) + # each block is judged with the other in view: they run in one directory, + # one after the other, and a wheel built by one is installed by the next + build_cmds = _comment_orphan_pip_installs(build_cmds, install_root) + install_root = _comment_orphan_pip_installs(install_root, build_cmds) + # LAST: whatever the filters above commented, the command before it must + # not be left ending in `&&`. + build_cmds = _comment_orphan_doc_installs(build_cmds, install_root) + install_root = _comment_orphan_doc_installs(install_root, build_cmds) + build_cmds = _comment_orphan_tree_shuffles(build_cmds) + install_root = _comment_orphan_tree_shuffles(install_root) + build_cmds = _guard_list_loops(build_cmds) + install_root = _guard_list_loops(install_root) + build_cmds = _comment_test_runs(build_cmds) + install_root = _comment_test_runs(install_root) + build_cmds = _comment_typeset_blocks(build_cmds) + install_root = _comment_typeset_blocks(install_root) + build_cmds = _comment_doc_format_builds(build_cmds) + install_root = _comment_doc_format_builds(install_root) + build_cmds = _comment_orphan_log_readers(build_cmds) + install_root = _comment_orphan_log_readers(install_root) + build_cmds = _no_vendored_subprojects(build_cmds) + install_root = _no_vendored_subprojects(install_root) + build_cmds = _allow_missing_subprojects(build_cmds, data.get("name")) + install_root = _allow_missing_subprojects(install_root, data.get("name")) + # LAST, once, after EVERY filter that comments a line: a command left + # ending in `&&` with nothing active after it is a syntax error. It ran + # before five of them, and the old scaffolding line that followed each + # phase body absorbed the dangling operator -- except in install_pkg, + # where cachecontrol and 145 others failed `bash -n`. + # ...and if every way of building it was skipped, enable one + build_cmds = _enable_one_alternative(build_cmds) + install_root = _drop_orphaned_variant_install(install_root) + build_cmds = _repair_dangling_operators(build_cmds) + install_root = _repair_dangling_operators(install_root) + config_cmds = _repair_dangling_operators(config_cmds) + data["config_cmds"] = config_cmds + _known = {"pkg", "link", "name", "name_version", "url", "md5_sum", "info", + "git_ref", "installed_program", "BUILD_ROOT"} + # Everything the SCRIPT itself sets, anywhere: plain assignments, exports, + # `for NAME in ...`, `read NAME`, and assignments inside heredocs. These + # were only scanned in the build text, so a loop variable in the install + # section (DOCNAME, PACKAGE_PREFIX_DIR, JT_JAVA...) looked like something + # the user had to provide. + _alltext = (build_cmds or "") + "\n" + (install_root or "") + "\n" \ + + (data.get("install_cmds") or "") + _setvars = set(re.findall(r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)=", + _alltext, re.M)) + _setvars |= set(re.findall(r"\bfor\s+([A-Za-z_][A-Za-z0-9_]*)\s+in\b", _alltext)) + _setvars |= set(re.findall(r"\bread\s+(?:-\S+\s+)*([A-Za-z_][A-Za-z0-9_]*)", _alltext)) + _setvars |= set(re.findall(r"^\s*([A-Za-z_][A-Za-z0-9_]*)=", _alltext, re.M)) + # what the page itself defines comes first; only what remains is asked of + # the user + _used = set(re.findall(r"\$\{([A-Z][A-Z0-9_]{2,})\}", build_cmds or "")) \ + | set(re.findall(r"\$([A-Z][A-Z0-9_]{2,})\b", build_cmds or "")) + _defs = {k: v for k, v in (data.get("page_env") or {}).items() + if k in (_used - (_known | _setvars))} + if _defs: + # dependencies first: XORG_PREFIX before XORG_CONFIG, or the second + # expands to a flag with an empty --prefix + _ordered, _left = [], dict(_defs) + for _ in range(len(_defs) + 1): + for k in sorted(_left): + refs = set(re.findall(r"\$\{?([A-Z][A-Z0-9_]{2,})", _left[k])) + if not (refs & set(_left)): + _ordered.append((k, _left.pop(k))) + break + else: + break + _ordered += sorted(_left.items()) + _pre = "\n".join('\t%s="%s" # set by the book' + % (k, v.strip('"\'')) + for k, v in _ordered) + build_cmds = _pre + "\n" + (build_cmds or "") + _setvars |= set(_defs) + build_cmds = _required_env_guard(build_cmds, _known | _setvars) install_root = _pip_reinstall(install_root) installed = data["installed_content"] unpacks = not re.search(rf"^tar .* {re.escape(pkg)} .*", data["install_cmds"]) @@ -1491,178 +3218,59 @@ def render_script(data, source_url): w("#!/bin/bash") w("############################################") w(f"### {name_version}") - w(f"### generated by blfs {BLFS_VERSION} -- script format v{SCRIPT_VERSION}") - w("### phased install script -- run a single phase with its name as $1:") - w("### all (default) | unpack | build | install | configure | update") - w("### update = install + reconfigure is SKIPPED (keeps your config)") + # 'origin=' is the shared contract: both generators stamp format + origin, + # so no tool has to recognise the other's dialect. + w(f"### generated by blfs {BLFS_VERSION} -- script format v{SCRIPT_VERSION}" + f" origin=blfs") + w("### phases: all (default) | unpack | build | test | install | configure | update") + w("### update = install without reconfiguring (keeps your config)") w("### re-run just 'install' after fixing permissions, without recompiling.\n") - w(f'url="{source_url}"') - w(f'name="{data["name"]}"') - w(f'name_version="{name_version}"') - w(f'link="{data["link"]}"') - w(f'pkg="{pkg}"') - w(f'md5_sum="{data["md5"]}"') - w(f"additional_links={bash_array(data['additional_links'])}") - w(f'info="{data["info"]}"') + w(f'url={bash_str(source_url)}') + w(f'name={bash_str(data["name"])}') + w(f'name_version={bash_str(name_version)}') + w(f'link={bash_str(data["link"])}') + w(f'pkg={bash_str(pkg)}') + w(f'md5_sum={bash_str(data["md5"])}') + w(f"additional_links={bash_array(_needed_additional_links(data, L))}") + w(f'info={bash_str(data["info"])}') w(f"required={bash_array(data['required'])}") w(f"recommended={bash_array(data['recommended'])}") w(f"optional={bash_array(data['optional'])}") - w("installed_content=(" + " ".join(f'"{k.rstrip(":")}"' for k in installed) + ")") + w("installed_content=(" + " ".join(bash_str(k.rstrip(":")) for k in installed) + ")") for k, v in installed.items(): - w(f'{k}="{v}"') + w(f'{k}={bash_str(v)}') if data["is_module"] and data["group"]: w(f"install_groups={bash_array([data['group']])}") - - # where the source is unpacked/built; kept between phases so 'install' can be - # re-run against the already-built tree. - w('\nBUILD_ROOT="${BUILD_ROOT:-$PWD}"') - w('pkg_dir=""') + if not unpacks: + w('pkg_unpack="no" # the page\'s own commands unpack the tarball') + # the mirror of last resort keeps most BLFS sources by filename -- the + # BOOK's release, not a literal that goes stale with the next one + if book_version and book_version != "unknown": + w(f'pkgusr_book_mirror={bash_str(f"{SITE}/blfs/downloads/{book_version}")}') w("") - w("_enter_build() {") - w('\t# locate the already-unpacked source directory for build/install phases') - w('\tcd "$BUILD_ROOT" || exit 1') - if unpacks: - w('\tif [ -z "$pkg_dir" ]; then') - w('\t\tpkg_dir="$name_version"') - w('\t\t[ -d "$pkg_dir" ] || pkg_dir="$(ls -d */ 2>/dev/null | head -n1)"') - w('\tfi') - w('\tif [ -n "$pkg_dir" ] && [ -d "$pkg_dir" ]; then cd "$pkg_dir" || exit 1') - w('\telse echo "no unpacked source in $BUILD_ROOT -- run: $0 unpack" && exit 1; fi') - w("}\n") - - # ---- unpack ---- - w("unpack_pkg() {") - w("#### UNPACK ####") - w('\tcd "$BUILD_ROOT" || exit 1') - if data["between_cmds"] and any(c.strip() for c in data["between_cmds"]): - w("\t# before-installation steps") - for cmds in data["between_cmds"]: - if cmds.strip(): - w(cmds) - w('\t[ -f "$pkg" ] || { [ -n "$link" ] && wget -4 "$link"; }') - w('\tif [ -n "$md5_sum" ] && ! grep -wq "$md5_sum" <<< "$(md5sum "$pkg")"; then') - w('\t\techo "ERROR: md5sum check failed" && exit 1') - w("\tfi") - w('\tfor li in "${additional_links[@]}"; do wget -4 "$li"; done') - if unpacks: - w("\trm -rf tmp && mkdir tmp && cd tmp") - if pkg.endswith(".zip"): - w('\tunzip ../"$pkg"') - else: - w('\ttar -xf ../"$pkg"') - # move the extracted tree to a clean $pkg_dir in BUILD_ROOT. Move the - # single top-level dir *explicitly* to $pkg_dir -- never `mv tmp/* .`, - # which would nest it inside an existing same-named dir. - w('\tif [ "$(ls | wc -w)" = "1" ] && [ -d "$(ls)" ]; then') - w('\t\tinner="$(ls)"; pkg_dir="$inner"') - w('\t\tcd .. && rm -rf "$pkg_dir" && mv "tmp/$inner" "$pkg_dir"') - w('\telse') - w('\t\tpkg_dir="$name_version"; cd .. && rm -rf "$pkg_dir" && mv tmp "$pkg_dir"') - w('\tfi') - w('\trm -rf tmp') - w('\techo "unpacked into $BUILD_ROOT/$pkg_dir"') - w("#### UNPACK DONE ####") - w("}\n") - # ---- build ---- + # The commands, one function per phase. NOT indented: the book is full of + # heredocs, and an indented terminator is no terminator. + between = [c for c in (data["between_cmds"] or []) if c.strip()] + if between: + w("prepare_pkg() {") + w(_fn_body("\n\n".join(between))) + w("}\n") w("build_pkg() {") - w("#### BUILD ####") - w("\t_enter_build") - w('\t# remove a stale CONFIGURED build dir so re-running "build" works, but') - w('\t# never a source dir that merely happens to be named build/ -- only wipe') - w('\t# it if it holds build-system artifacts we would have generated.') - w('\tfor _d in build builddir _build; do') - w('\t\tif [ -d "$_d" ] && { [ -f "$_d/build.ninja" ] || ' - '[ -f "$_d/CMakeCache.txt" ] || [ -d "$_d/meson-private" ] || ' - '[ -f "$_d/config.status" ]; }; then rm -rf "$_d"; fi') - w('\tdone') - w(build_cmds if build_cmds.strip() else '\t: # (no separate build step)') - w('\t# remember where the build ended (e.g. a meson "build/" subdir) so the') - w('\t# install phase can resume there even when run separately.') - w('\tpwd > "$BUILD_ROOT/.pm_build_cwd" 2>/dev/null || true') - w("#### BUILD DONE ####") + w(_fn_body(build_cmds, ": # (no separate build step)")) w("}\n") - - # ---- install ---- w("install_pkg() {") - w("#### INSTALL ####") - w("\t_enter_build") - w('\t# resume where the build actually happened: prefer the dir the build') - w('\t# phase recorded; otherwise auto-detect a meson/cmake build subdir, so') - w('\t# "ninja install" / "make install" run in the right place even if the') - w('\t# tree was built by an older script.') - w('\tif [ -f "$BUILD_ROOT/.pm_build_cwd" ] && ' - '[ -d "$(cat "$BUILD_ROOT/.pm_build_cwd" 2>/dev/null)" ]; then') - w('\t\tcd "$(cat "$BUILD_ROOT/.pm_build_cwd")"') - w('\telif [ ! -e build.ninja ] && [ ! -e Makefile ]; then') - w('\t\tfor _d in build builddir _build; do') - w('\t\t\tif [ -e "$_d/build.ninja" ] || [ -e "$_d/Makefile" ]; then ' - 'cd "$_d"; break; fi') - w('\t\tdone') - w('\tfi') - if install_root.strip(): - w(install_root) - else: - w('\t: # (this package installs during the build phase; nothing separate)') - w("#### INSTALL DONE ####") + w(_fn_body(install_root, + ": # (this package installs during the build phase; nothing separate)")) w("}\n") - - # ---- configure ---- if config_cmds.strip(): w("configure_pkg() {") - w("#### CONFIGURE ####") - w('\techo "## Configuration"') - w(config_cmds) - w("#### CONFIGURE DONE ####") + w('echo "## Configuration"') + w(_fn_body(config_cmds)) w("}\n") - else: - w("configure_pkg() { : ; }\n") - - # ---- test (best-effort; edit for package-specific test commands) ---- - w("test_pkg() {") - w("#### TEST ####") - w("\t_enter_build") - w('\tif [ -f "$BUILD_ROOT/.pm_build_cwd" ] && ' - '[ -d "$(cat "$BUILD_ROOT/.pm_build_cwd" 2>/dev/null)" ]; then') - w('\t\tcd "$(cat "$BUILD_ROOT/.pm_build_cwd")"') - w('\telif [ ! -e build.ninja ] && [ ! -e Makefile ]; then') - w('\t\tfor _d in build builddir _build; do [ -e "$_d/build.ninja" ] || ' - '[ -e "$_d/Makefile" ] && cd "$_d" && break; done') - w('\tfi') - w('\techo "## Running test suite"') - w('\tif [ -f build.ninja ]; then meson test || ninja test') - w('\telif [ -f Makefile ]; then make check || make test') - w('\telse echo "no recognized test target -- edit test_pkg() for this package"; fi') - w("#### TEST DONE ####") - w("}\n") - - # ---- dispatcher ---- - w('# run only when executed directly, not when sourced') - w('if [ "${BASH_SOURCE[0]}" = "${0}" ]; then') - w('case "${1:-all}" in') - w('\tall)') - w('\t\tunpack_pkg && build_pkg || exit 1') - w('\t\t[ "${PM_TEST:-}" = "1" ] && test_pkg') - w('\t\tinstall_pkg && configure_pkg ;;') - w('\tunpack) unpack_pkg ;;') - w('\tbuild) build_pkg ;;') - w('\ttest) test_pkg ;;') - w('\tinstall) install_pkg ;;') - w('\tconfigure) configure_pkg ;;') - w('\tupdate)') - w('\t\tunpack_pkg && build_pkg || exit 1') - w('\t\t[ "${PM_TEST:-}" = "1" ] && test_pkg') - w('\t\tinstall_pkg ;;') - w('\t*) echo -e "usage: $0 {all|unpack|build|test|install|configure|update}\\n' - ' all full run (default; set PM_TEST=1 to also run tests)\\n' - ' unpack fetch + extract only\\n' - ' build configure + compile (needs unpack)\\n' - ' test run the test suite (needs build)\\n' - ' install install step only -- re-run this after fixing permissions\\n' - ' configure post-install configuration\\n' - ' update unpack + build + install, no reconfigure (keeps your config)" ;;') - w("esac") - w("fi") + # no test_pkg: the runner autodetects `meson test` / `make check`; define + # one here to override. + w('. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"') safe = re.sub(r"[^A-Za-z0-9._-]+", "-", name_version).strip("-") body = "\n".join(L) + "\n" @@ -1671,7 +3279,8 @@ def render_script(data, source_url): # computed over the body WITHOUT this line, so verification just strips the # 'pm_md5' line and recomputes. digest = hashlib.md5(body.encode()).hexdigest() - stamp = f"### generated by blfs {BLFS_VERSION} -- script format v{SCRIPT_VERSION}" + stamp = (f"### generated by blfs {BLFS_VERSION} -- script format " + f"v{SCRIPT_VERSION} origin=blfs") body = body.replace(stamp, stamp + f"\n### pm_md5={digest}", 1) return f"install_{safe}", body @@ -1717,8 +3326,33 @@ def cmd_install(args): def cmd_fetch(args): """Download (or refresh) a book into the cache.""" sel = args.selector or DEFAULT_SELECTOR - ensure_book(sel) - print("cached.") + # SAY IT WHEN THERE IS NO SUCH BOOK. + # + # `blfs fetch stable-sysv` printed the routine "no book matched + # 'stable-sysv'; using the cached ...-systemd" note and then "cached." -- + # so a request that fetched NOTHING reported success, and the user + # reasonably believed they now had the sysv book. There is no 13.0 sysv + # book to have: BLFS stopped publishing the sysv edition after 12.4. + avail = load_available() + if sel not in avail and not sel.startswith(("http://", "https://")) \ + and not os.path.isfile(sel): + try: + avail = discover_all() or avail + except Exception: + pass + if sel not in avail and not sel.startswith(("http://", "https://")) \ + and not os.path.isfile(sel): + sys.stderr.write("no book is published as '%s'.\n" % sel) + _sysv = sorted((k for k, v in avail.items() + if (v.get("init") or "") == "sysv"), + key=str) + if _sysv: + sys.stderr.write(" The editions that exist: blfs books\n") + sys.stderr.write(" Newest SysV edition: %s\n" % _sysv[-1]) + sys.stderr.write(" Nothing was fetched.\n") + sys.exit(1) + path = ensure_book(sel) + print("cached: %s" % path) def cmd_import(args): @@ -1740,6 +3374,7 @@ def cmd_sources(args): """The download URLs for a package, exactly as the book gives them.""" book = get_book(args) anchor = anchor_from_input(args.package) + anchor = resolve_anchor(book, anchor) data = book.get(anchor) if data is None: sys.stderr.write("no package '%s' in the book (try: blfs search %s)\n" @@ -1756,6 +3391,127 @@ def cmd_sources(args): print(u) +# What makes a difference an INIT difference rather than version churn. +_INIT_WORDS_RE = re.compile( + r"\b(systemd|systemctl|sysv|bootscript|init\.d|rc\.d|tmpfiles|sysusers|" + r"unit[-_ ]?dir|--(en|dis)able-systemd|service)\b", re.I) + + +def cmd_diff_init(args): + """Show where a SysV book and a systemd book disagree. + + The generated scripts come from ONE book. Using the systemd book on a + SysV system is workable -- a missing systemctl becomes a visible skip -- + right up to a page whose CONFIGURE line differs, and then it is not: + + BlueZ: configure: error: systemd system unit directory is required + + because the sysv page passes --disable-systemd and the systemd page has + no reason to. Per-package machine.conf entries paper over that one page + at a time, and only after each has failed. This says which pages differ + BEFORE any of them are built -- and for one package, exactly how. + """ + import difflib + other = args.other + if not os.path.isfile(other): + url, local, _is_svn, _v = resolve_book(other) + if not os.path.isfile(local): + ensure_book(other) + other = local + here = get_book(args) + # the same loader the rest of the tool uses: cached if the cache is + # current, indexed once otherwise + there = get_book(argparse.Namespace(book_file=other, book=None)) + # NAME BOTH SIDES, AND REFUSE TO COMPARE ONE WITH ITSELF. + # + # `blfs diff-init 12.4` reported "0 of 947 shared pages differ", which + # was true and useless: the default selector (stable-sysv) has no book, + # so it fell back to the newest cached one -- 12.4, the very book being + # compared against. A comparison that silently compares a thing with + # itself is worse than an error. + print("comparing:\n A %s\n B %s\n" % (here.path, there.path)) + if os.path.realpath(here.path) == os.path.realpath(there.path): + sys.stderr.write("those are the same file -- nothing to compare.\n") + sys.stderr.write(" Name the book to compare AGAINST, and make sure a\n") + sys.stderr.write(" different one is in use here:\n") + sys.stderr.write(" blfs --book-file <this book> diff-init <other>\n") + sys.exit(1) + names = args.packages or sorted(set(here.packages) & set(there.packages)) + # ONLY THE PAGES THIS MACHINE WILL EVER BUILD. + # + # 47 of 929 shared pages differ in their init handling, and most of them + # are for software this system does not have and never will (GNOME, samba, + # qemu, postgresql...). The useful question is not "what differs" but + # "what differs AND is on my machine", and the answer is on disk: a + # package with an account has been built or is about to be. + if getattr(args, "installed", False): + root = os.environ.get("LFS_PKGUSR_ROOT", "/usr/src/pkgusr") + pfx = os.environ.get("LFS_PKGUSR_PREFIX", "p_") + try: + have = {d[len(pfx):] for d in os.listdir(root) if d.startswith(pfx)} + except OSError: + have = set() + if not have: + sys.stderr.write("no package users under %s -- --installed has " + "nothing to filter by.\n" % root) + sys.exit(1) + names = [n for n in names if n.lower() in {h.lower() for h in have}] + shown = 0 + for anchor in names: + a = here.get(resolve_anchor(here, anchor_from_input(anchor))) + try: + b = there.get(resolve_anchor(there, anchor_from_input(anchor))) + except SystemExit: + b = None + if not a or not b: + continue + ka = "\n".join([a.get("build_cmds") or "", a.get("install_root_cmds") or ""]) + kb = "\n".join([b.get("build_cmds") or "", b.get("install_root_cmds") or ""]) + if ka.strip() == kb.strip(): + continue + # ACROSS VERSIONS, ALMOST EVERYTHING DIFFERS. + # + # There is no 13.0 sysv book (BLFS stopped after 12.4), so the only + # comparison available is 12.4-sysv against 13.0-systemd -- where most + # pages differ for version reasons that have nothing to do with init. + # Default to the differences that MENTION init, which is what the + # question is; --all for the rest. + if not getattr(args, "all", False): + # A COMMENTED LINE IS NOT A COMMAND -- HERE TOO (1.14.23 again). + # + # Of eight pages this reported for a real system, four were + # nothing: gcc, libxml2 and rust differed only in NOTE prose that + # happens to say "systemd journal", "systemctl stop httpd.service" + # or a book URL containing "13.0-systemd". Judge the difference + # on lines that RUN. + _live = lambda t: [l for l in t.splitlines() + if l.strip() and not l.strip().startswith("#")] + _delta = "\n".join(l for l in difflib.ndiff(_live(kb), _live(ka)) + if l[:1] in "+-") + if not _INIT_WORDS_RE.search(_delta): + continue + shown += 1 + if args.packages: + print("=== %s ===" % anchor) + for line in difflib.unified_diff( + kb.splitlines(), ka.splitlines(), + fromfile="%s (%s)" % (anchor, os.path.basename(other)), + tofile="%s (%s)" % (anchor, os.path.basename(here.path)), + lineterm=""): + print(line) + print() + else: + print(" %-28s commands differ" % anchor) + if not args.packages: + _what = "differ" if getattr(args, "all", False) \ + else "differ in their init handling" + print("\n%d of %d shared pages %s." % (shown, len(names), _what)) + if not getattr(args, "all", False): + print("(--all compares everything, including version churn)") + print("For one of them: blfs diff-init %s <package>" % args.other) + return 0 + + def cmd_books(args): if getattr(args, "rebuild_cache", False): book = get_book(args, rebuild=True) @@ -1781,29 +3537,44 @@ def cmd_books(args): if not found: print(" (none yet)") + # --brief is the wrapper's contract: the Default line and the cached + # list, NOTHING live. packagemanager reads the book name through this, + # and the auto-discovery below is for a person at a terminal -- routed + # through the wrapper it made every `packagemanager update` probe the + # site, twice, and print the probes' notes. + if getattr(args, "brief", False): + return + + def _print_available(books, note): + print(note) + for sel, rec in sorted(books.items()): + mark = " (default)" if sel == default else "" + print(f" {sel:<22} version {rec['version']:<8} [{rec['init']}]{mark}") + print(f"\nPick one with: blfs set-default <selector>") + + # Discovery is automatic: with a connection, the list is live; without + # one, the cached list from the last discovery is shown and SAYS it is + # cached. --discover is kept as the explicit refresh (it reports the + # failure loudly instead of falling back). if args.discover: - print("\nAvailable books (from the downloads index):") try: - books = discover_all() + _print_available(discover_all(), + "\nAvailable books (from the downloads index):") except Exception as e: print(f" could not read {downloads_index_url()} ({e})") return - if not books: - print(" (none found)") - for sel, rec in sorted(books.items()): - mark = " (default)" if sel == default else "" - print(f" {sel:<22} version {rec['version']:<8} [{rec['init']}]{mark}") - print(f"\nPick one with: blfs set-default <selector>") else: - avail = load_available() - if avail: - print("\nAvailable books (cached from last --discover):") - for sel, rec in sorted(avail.items()): - mark = " (default)" if sel == default else "" - print(f" {sel:<22} version {rec['version']:<8} [{rec['init']}]{mark}") - print("\nRe-run with --discover to refresh this list.") - else: - print("\nRun 'blfs books --discover' to list what the site offers.") + try: + _print_available(discover_all(), + "\nAvailable books (from the downloads index):") + except Exception: + avail = load_available() + if avail: + _print_available(avail, "\nAvailable books (offline -- cached " + "from the last discovery):") + else: + print("\n(offline, and no cached discovery yet -- re-run with " + "a connection to list what the site offers)") def cmd_set_default(args): @@ -1811,6 +3582,19 @@ def cmd_set_default(args): cfg["default"] = args.selector save_config(cfg) print(f"Default book set to: {args.selector}") + # A default that is not cached is a decision waiting to be silently + # overridden: every lookup would fall back to the closest cached book. + # That happened -- set-default 13.0-systemd, and update read 12.4 with + # the note swallowed. So the book is cached NOW, while the person who + # made the choice is looking. + try: + ensure_book(args.selector) + except Exception as e: + warn(f"note: could not cache '{args.selector}' ({e}).") + warn(" Until it is cached, every lookup falls back to the " + "closest cached book, and says so.") + warn(f" Fetch it when there is a connection: blfs fetch " + f"{args.selector}") def cmd_search(args): @@ -1830,31 +3614,52 @@ def cmd_search(args): def cmd_script(args): + # `blfs script <name>` has always BEEN fetch; `blfs script fetch <name>` + # is accepted so the verb reads the same across lfs, blfs and + # packagemanager. + _t = list(args.target) + if _t and _t[0] == "fetch" and len(_t) > 1: + _t = _t[1:] + # SEVERAL PACKAGES, ONE BOOK. The parser has always taken a list and this + # used _t[0] and dropped the rest, so packagemanager regenerating a plan + # ran one `blfs script` per package -- 151 subprocesses, each loading and + # indexing the book again. Loading it once and writing them all is the + # same work minus 150 book loads. book = get_book(args) - anchor = anchor_from_input(args.target) - data = book.get(anchor) - if data is None: - print(f"ERROR: could not find anchor '{anchor}' in the book.") - print(" run: blfs debug " + args.target) - sys.exit(1) - src = args.target if args.target.startswith("http") else f"#{anchor}" - filename, script = render_script(data, src) + ver = _book_version_of(book) os.makedirs(args.outdir, exist_ok=True) - path = os.path.join(args.outdir, filename) - with open(path, "w") as f: - f.write(script) - print(f"Created {path}") - try: - subprocess.run(["bash", "-n", path], check=True) - print("Bash syntax OK.") - except subprocess.CalledProcessError: - print("!!! bash syntax error -- review the file") - return path + written = [] + for target in _t: + anchor = anchor_from_input(target) + anchor = resolve_anchor(book, anchor) + data = book.get(anchor) + if data is None: + print(f"ERROR: could not find anchor '{anchor}' in the book.") + print(" run: blfs debug " + target) + if len(_t) == 1: + sys.exit(1) + continue + src = target if target.startswith("http") else f"#{anchor}" + filename, script = render_script(data, src, ver) + path = os.path.join(args.outdir, filename) + with open(path, "w") as f: + f.write(script) + print(f"Created {path}") + try: + subprocess.run(["bash", "-n", path], check=True) + print("Bash syntax OK.") + except subprocess.CalledProcessError: + print("!!! bash syntax error -- review the file") + written.append(path) + if not written: + sys.exit(1) + return written[0] if len(written) == 1 else written def cmd_deps(args): book = get_book(args) anchor = anchor_from_input(args.name) + anchor = resolve_anchor(book, anchor) if book.get(anchor) is None: print(f"ERROR: '{anchor}' not found. try: blfs debug {args.name}") sys.exit(1) @@ -1933,6 +3738,7 @@ def cmd_rdeps(args): seen.add(dependent) order.append(dependent) queue.append(dependent) + seen_anchors = {a for a in order} if getattr(args, "anchors", False): for a in order: data = book.get(a) @@ -1947,16 +3753,20 @@ def cmd_rdeps(args): def cmd_order(args): book = get_book(args) anchor = anchor_from_input(args.name) + anchor = resolve_anchor(book, anchor) if book.get(anchor) is None: print(f"ERROR: '{anchor}' not found. try: blfs debug {args.name}") sys.exit(1) kinds = selected_kinds(args) + _ign = [x for x in (getattr(args, "ignore", "") or "").split(",") if x] + _ign = [resolve_anchor(book, anchor_from_input(x)) for x in _ign] - order, edges, missing, via = walk_deps(book, anchor, kinds) + order, edges, missing, via = walk_deps(book, anchor, kinds, ignore=_ign) # machine-readable mode: one TSV line per package in install order, # "anchor<TAB>name_version<TAB>kind<TAB>parent_anchor" (root has empty # kind/parent), then "#missing<TAB>name<TAB>kind<TAB>parent". + seen_anchors = {a for a in order} if getattr(args, "anchors", False): for a in order: parent, kind = ("", "") if a == anchor else via.get(a, ("", "")) @@ -1964,6 +3774,14 @@ def cmd_order(args): for m in sorted(set(missing)): parent, kind = via.get(m, ("", "")) print(f"#missing\t{m}\t{kind}\t{parent}") + # a dependant built before something it recommends, because a cycle + # left no order that satisfies both -- the caller decides what to say + for _a, _d in _CYCLE_DROPS: + if _a in seen_anchors and _d in seen_anchors: + print(f"#cycle\t{_a}\t{_d}") + for _parent, _ign in getattr(walk_deps, "ignored_required", ()): + if _parent in seen_anchors: + print(f"#ignored-required\t{_parent}\t{_ign}") return os.makedirs(args.outdir, exist_ok=True) @@ -1974,7 +3792,7 @@ def cmd_order(args): data = book.get(a) lines.append(data["name_version"]) if args.scripts: - fn, script = render_script(data, f"#{a}") + fn, script = render_script(data, f"#{a}", _book_version_of(book)) with open(os.path.join(args.outdir, fn), "w") as f: f.write(script) @@ -1999,6 +3817,7 @@ def cmd_debug(args): print(f"input : {args.target}") print(f"anchor : {anchor}") + anchor = resolve_anchor(book, anchor) data = book.get(anchor) if data is None: # not a package in the cache -- parse the soup to explain why @@ -2067,7 +3886,7 @@ def main(): book_args, remaining = pre.parse_known_args() ap = argparse.ArgumentParser(prog="blfs", description="BLFS book tool") - ap.add_argument("--version", action="version", + ap.add_argument("-V", "--version", action="version", version=f"blfs {BLFS_VERSION} (build {_build_id()}, " f"cache v{CACHE_VERSION}, " f"script v{SCRIPT_VERSION})") @@ -2114,8 +3933,25 @@ def main(): p.add_argument("package") p.set_defaults(func=cmd_sources) + p = sub.add_parser("diff-init", + help="which pages differ between this book and another " + "(e.g. the sysv book vs the systemd one)") + p.add_argument("other", help="selector (stable-sysv) or a book file") + p.add_argument("packages", nargs="*", + help="show the command diff for these; omit to list every " + "page that differs") + p.add_argument("--all", action="store_true", + help="every difference, not only the init-related ones " + "(across two book VERSIONS most pages differ anyway)") + p.add_argument("--installed", action="store_true", + help="only pages this machine has a package user for -- " + "what differs AND concerns you") + p.set_defaults(func=cmd_diff_init) + p = sub.add_parser("books", help="list/inspect books") p.add_argument("--discover", action="store_true", help="probe the live site") + p.add_argument("--brief", action="store_true", + help="default + cached list only, no network (for scripts)") p.add_argument("--rebuild-cache", action="store_true", help="re-index the current book (after upgrading blfs)") p.set_defaults(func=cmd_books) @@ -2129,7 +3965,10 @@ def main(): p.set_defaults(func=cmd_search) p = sub.add_parser("script", help="write an install_<name-version> script") - p.add_argument("target", help="package name, or a BLFS url (optionally #fragment)") + p.add_argument("target", nargs="+", + help="package name, or a BLFS url (optionally #fragment); " + "'fetch <name>' is accepted too, matching the other " + "tools' script commands") p.add_argument("-o", "--outdir", default=".") p.set_defaults(func=cmd_script) @@ -2152,6 +3991,9 @@ def main(): help="also follow optional dependencies") p.add_argument("--anchors", action="store_true", help="machine-readable: print 'anchor<TAB>name_version' in order") + p.add_argument("--ignore", default="", + help="comma-separated packages not to build -- their own " + "dependencies are not walked into either") p.set_defaults(func=cmd_order) p = sub.add_parser("debug", help="explain how an anchor resolves") diff --git a/kernel.conf b/kernel.conf new file mode 100644 index 0000000..f9074e2 --- /dev/null +++ b/kernel.conf @@ -0,0 +1,114 @@ +# kernel.conf -- read by `lfs-kernel`. Edit before building. + +# VERSION: an exact version (7.2.5), or "latest" for the newest stable on +# kernel.org. `lfs-kernel <version>` overrides it for one run, +# `lfs-kernel list` shows what is on offer. +VERSION=latest + +# ---- where the kernel is installed --------------------------------------- +# +# ESP + KERNEL_NAME are the pair that decides WHICH kernel this is. Point +# them at your boot manager's directory and the file it loads: +# +# KERNEL_NAME=vmlinuz_new # a kernel to try, beside the working one +# KERNEL_NAME=vmlinuz # the one rEFInd boots by default +# +# The previous file is always kept as ${KERNEL_NAME}_old, and a copy is left +# as ${KERNEL_NAME}-${VERSION} so several versions can sit side by side. +# `lfs-kernel detect-esp` prints the vfat mount that already holds a kernel; +# `findmnt -t vfat` lists them all. +ESP=/boot/efi/EFI/NimGnu +KERNEL_NAME=vmlinuz_new + +# ---- the package --------------------------------------------------------- +# +# The kernel is a package like any other: PKG names the package user that +# owns it, and its sources live in that account's home +# (/usr/src/pkgusr/p_<PKG>/src), like every other package's. There is no +# separate source directory to configure. +# +# What it owns: /lib/modules/<version>, /usr/include (the headers), +# /usr/share/doc/linux-<version>. The copies onto the EFI partition are +# root's -- vfat has no ownership and the mount is root's. +# +# The tree of the last kernel known to boot is NEVER removed; its version is +# recorded in /etc/pkgusr/kernel.last-good. +PKG=linux + +# Leaving a core or two free keeps the machine usable during a build, and a +# kernel link is memory-hungry. +JOBS=-j14 + +# CONFIG_FROM: running (/proc/config.gz) | file:<path> | keep (the tree's own) +CONFIG_FROM=running + +# CONFIG_REFRESH: a config from one kernel is never complete for another -- +# every version adds symbols the old file has no answer for, and something +# has to decide them: +# olddefconfig take upstream's DEFAULT for each new symbol, silently. +# Right for an unattended run; the defaults are not yours. +# oldconfig ask about each new symbol. Needs someone at the +# terminal, decides nothing behind your back. +# menuconfig open the editor and decide everything at once. +# none use the .config as it is and let `make` deal with it. +# Either way the tool reports how many lines changed. +CONFIG_REFRESH=olddefconfig + +# ---- firmware ------------------------------------------------------------ +# +# Most drivers need firmware blobs: amdgpu will not bring up a display +# without them, and a config that lists them in CONFIG_EXTRA_FIRMWARE cannot +# even BUILD without them. They are installed as their own package +# (p_linux-firmware), so they are tracked, verifiable and updatable on their +# own. +# +# FIRMWARE_FROM: +# git clone/pull kernel.org's linux-firmware (needs network) +# copy:<path> copy from a directory you already have, e.g. +# copy:/lib/firmware on a host that is already running +# none do not touch firmware at all +FIRMWARE_FROM=git +FIRMWARE_GIT=https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git + +# FIRMWARE_ONLY: what to install. linux-firmware is over 10 GB; listing +# what your hardware needs keeps it to a few hundred megabytes. Empty means +# everything. +# +# Entries are directories OR file patterns, because not all firmware lives in +# a subdirectory -- Intel wireless blobs sit at the top level: +# amdgpu Radeon / Renoir / Raven / Picasso graphics +# amd-ucode AMD CPU microcode (a distribution config embeds it) +# amd AMD SEV and other platform blobs +# intel EVERYTHING Intel, in one directory: +# intel/iwlwifi/ wireless (AX200 and the rest) +# intel/ibt-* Bluetooth +# Current linux-firmware nests iwlwifi under intel/; +# older releases kept iwlwifi-*.ucode at the top level, +# which is what a FILE pattern like iwlwifi-cc-* is for. +# rtl_bt Realtek Bluetooth +# Find what your machine asks for, once it is running, with: +# dmesg | grep -i firmware +# (quoted: the value has spaces and a wildcard, and the config is sourced by +# the shell -- unquoted, `intel` was run as a command and the whole +# assignment came out empty, which meant "install everything") +FIRMWARE_ONLY="amdgpu amd-ucode amd intel" + +# ---- headers ------------------------------------------------------------- +# +# /usr/include is owned by the headers package (p_linux-headers), installed +# by the LFS step of the same name. Installing a newer kernel's headers is +# an UPDATE of that package, so the copy runs as that account -- a second +# package writing over its files could not work, and should not. +# +# HEADERS=no leaves /usr/include alone, which is what you want when the +# running system's headers are deliberately older than the kernel you build. +HEADERS=yes +HEADERS_PKG=linux-headers + +# ---- extras -------------------------------------------------------------- +# A 224-colour ppm boot logo, or empty for the stock one. +BOOTLOGO=https://christianimmanuel.de/bootlogo.ppm + +# The out-of-tree v4l2loopback module. +V4L2LOOPBACK=yes +V4L2LOOPBACK_GIT=https://github.com/v4l2loopback/v4l2loopback.git diff --git a/lfs b/lfs old mode 100644 new mode 100755 index ac5c67b..cd534f8 --- a/lfs +++ b/lfs @@ -60,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.11.6" +LFS_VERSION = "1.14.146" def _build_id(): @@ -243,6 +243,39 @@ LAYOUT_DEFAULTS = { } +def _accounts_root(kind): + """Where the homes of one kind of account actually live. + + The config has TWO keys that both claim to answer this: `pkgusr_home`, + whose default is /usr/src/pkgusr, and `pkgusr_subdir`, "folder under the + base for package homes (pkgusr)". Read one way the base already includes + the folder; read the other it does not -- and pkgusr_home's own help text + says "(/usr/src)", which invites the second reading. A config written + that way exported LFS_PKGUSR_ROOT=/usr/src, so every account's home became + /usr/src/p_<name>, `add_package_user` made it there, and every tool that + computes /usr/src/pkgusr/p_<name> disagreed with the passwd file -- for + the whole life of that tree. + + Accept both readings: append the subdir when the base does not already + end in it. + """ + # (packagemanager already composes it this way -- see PKGUSR_SUBDIR + # there -- which is exactly why the two disagreed: one appended the + # subdir and the other exported the bare base.) + base = layout("pkgusr_home" if kind == "pkgusr" else "cfguser_home", + target=True).rstrip("/") + _c = {} + try: + _c = read_config() or {} + except Exception: + _c = {} + sub = str(_c.get("pkgusr_subdir" if kind == "pkgusr" else "cfguser_subdir") + or ("pkgusr" if kind == "pkgusr" else "cfg")).strip("/") + if not sub or os.path.basename(base) == sub: + return base + return base + "/" + sub + + def layout(key, target=False): """One layout setting, target or host. @@ -490,15 +523,32 @@ def resolve_book_arg(val): # --------------------------------------------------------------------------- # # book loading + section walking # --------------------------------------------------------------------------- # +_SOUP_CACHE = {} + + def load_soup(path): + """Parse a book, once per path per process. + + Without the cache `_find_lfs_package` re-parsed the whole book for EVERY + name it was asked about -- `lfs update` with 77 packages parsed a 10 MB + HTML file 77 times, which is minutes of work and long enough for a caller + with a timeout to give up on it.""" + key = os.path.abspath(path) + if key in _SOUP_CACHE: + return _SOUP_CACHE[key] with open(path, "rb") as f: html = f.read() + soup = None for parser in ("lxml", "html.parser"): try: - return BeautifulSoup(html, parser) + soup = BeautifulSoup(html, parser) + break except Exception: continue - return BeautifulSoup(html, "html.parser") + if soup is None: + soup = BeautifulSoup(html, "html.parser") + _SOUP_CACHE[key] = soup + return soup def _is_admonition(tag): @@ -678,18 +728,34 @@ def _tar_compressor(): return "--gzip", ".tar.gz" +def _build_is_finished(lfs): + """Has the build run to the end of its own step order? + + The same question lfs-sanity.sh asks, answered the same way: every step in + progress/steporder is in progress/steps-built. Nothing else in the tree + distinguishes "chapter 6, and something went wrong" from "finished", and + several checks mean opposite things on either side of that line. + """ + prog = pkgusr_state(lfs, "progress") + try: + order = [l for l in open(os.path.join(prog, "steporder")) if l.strip()] + built = [l for l in open(os.path.join(prog, "steps-built")) if l.strip()] + except OSError: + return False + return bool(order) and len(built) >= len(order) + + def _toolchain_state(lfs): """What state is the toolchain in, judged from the tree alone? Checkable from the host, without entering the chroot: the C++ headers must - exist under the target the tree was built for, and a NATIVE gcc directory - must not be present -- chapter 6 only ever installs under $LFS_TGT, so a - native one means a chapter-8 gcc was installed on top. + exist under the target the tree was built for. Returns a short phrase for the snapshot metadata. """ import glob as _glob tgt = lfs_tgt() + finished = _build_is_finished(lfs) inc = _glob.glob(os.path.join(lfs, "usr", "include", "c++", "*")) if not inc: return "chapter 6 unfinished (no C++ headers)" @@ -698,12 +764,31 @@ def _toolchain_state(lfs): if os.path.isfile(os.path.join(d, "bits", "c++config.h"))] native = _glob.glob(os.path.join(lfs, "usr", "lib", "gcc", "*-pc-linux-gnu")) + # A NATIVE GCC MEANS OPPOSITE THINGS ON EITHER SIDE OF THE BUILD. + # + # Chapter 6 installs only under $LFS_TGT, so during the build a native gcc + # on top of the cross one is the toolchain being clobbered. Chapter 8 + # REPLACES the cross toolchain with exactly that native gcc -- it is the + # finished state, the thing the build is for. Reporting one rule for both + # told someone snapshotting a complete, working system: + # toolchain: BROKEN: a native gcc (x86_64-pc-linux-gnu) is installed + # ! this snapshot is of a tree whose toolchain is already broken. + # about a tree with nothing wrong with it. Same species as the sticky-bit + # check in lfs-sanity.sh, which had to learn the same distinction. if native and tgt not in [os.path.basename(n) for n in native]: - return ("BROKEN: a native gcc (%s) is installed over the cross one" + if finished: + return ("ok (native %s -- chapter 8 replaced the cross toolchain)" + % os.path.basename(native[0])) + return ("BROKEN: a native gcc (%s) is installed over the cross one, " + "and the build has not finished" % os.path.basename(native[0])) if not have: return "chapter 6 unfinished (no c++config.h)" if tgt not in have: + # Finished, and the headers moved to the native triplet: chapter 8 + # again, not damage. + if finished and native: + return "ok (C++ headers under %s -- finished system)" % ", ".join(have) return "BROKEN: headers are under %s, but LFS_TGT is %s" % ( ", ".join(have), tgt) return "ok (C++ headers under %s)" % tgt @@ -833,8 +918,8 @@ def cmd_snapshot(args): mounted = [d for d in ("dev", "proc", "sys", "run") if _is_mounted(os.path.join(lfs, d))] if mounted: - sys.stderr.write( - "\n! these are still mounted in the tree: %s\n" + fail( + "\n! these are still mounted in the tree: %s" " Restoring now could write through them onto this system.\n" " Unmount first:\n" " %slfs build-system chroot unmount --run\n" @@ -1034,8 +1119,6 @@ def cmd_set_default(args): cfg["default"] = ver save_config(cfg) print(f"book for installing packages set to {ver}") - print(" (the book the SYSTEM is built from is 'build_book':") - print(" lfs build-system set-default <version>)") def cmd_fetch(args): @@ -1069,8 +1152,6 @@ def cmd_import(args): cfg["default"] = ver save_config(cfg) print(f"book for installing packages set to {ver}") - print(" (the book the SYSTEM is built from is 'build_book':") - print(" lfs build-system set-default <version>)") def cmd_sources(args): @@ -1200,9 +1281,32 @@ def _lfs_env_prefix(lfs): f'CONFIG_SITE="$LFS/usr/share/config.site"\n' f'export CONFIG_SITE\n' f'export MAKEFLAGS="{mkf}"\n' + + (f'export PKGUSR_LIB="{lib}"\n' if (lib := _host_phases_lib()) else "") ) +def _host_phases_lib(): + """The phase runner, where the lfs user can read it. The checkout may sit + in /root, so the copy goes into the shared store, refreshed from the one + beside this tool whenever they differ.""" + src = _find_helper_source("lfs-phases") + if not src: + return None + try: + import shutil + dest = os.path.join(_ensure_store(), "lfs-phases") + import hashlib + def _sum(f): + with open(f, "rb") as fh: + return hashlib.md5(fh.read()).hexdigest() + if not os.path.isfile(dest) or _sum(src) != _sum(dest): + shutil.copy(src, dest) + _make_world_readable(dest) + return dest + except OSError: + return src + + def run_as_lfs(script, lfs=None): """Run a script as the lfs user via `su - lfs`. IMPORTANT: `su -` fully resets the environment (that's what login simulation does), so anything the @@ -1290,8 +1394,8 @@ def _refuse_host_tree(lfs): if not _is_host_path(lfs): return lfs real = os.path.realpath(lfs) - sys.stderr.write( - "\n!! REFUSING: %s is part of the running system.\n" + fail( + "\n!! REFUSING: %s is part of the running system." " Every path this tool writes is $LFS/something, so this would\n" " build into, and `restart` would delete, the machine you are\n" " sitting on.\n" @@ -1305,10 +1409,7 @@ def require_lfs_for_root(): back to the saved session config and exports it so child shells inherit it.""" d = _lfs_dir() if not d: - sys.stderr.write( - "! no LFS mount set (neither $LFS nor a saved session).\n" - " Configure it once: lfs build-system session\n" - " or for this shell: export LFS=/mnt/lfs\n") + fail('! no LFS mount set (neither $LFS nor a saved session).\n Configure it once: lfs build-system session\n or for this shell: export LFS=/mnt/lfs') sys.exit(2) _refuse_host_tree(d) os.environ["LFS"] = d @@ -1395,12 +1496,11 @@ _SESSION_KEYS = [ "no"), # Off by default and deliberately so: stripping REWRITES each binary, and # in this system a file's owner is the record of which package installed - # it, so stripping has to run as that owner or the record drifts. + # it -- which is why `lfs-helper strip` runs each file as its owner. ("strip", "strip debug symbols at the end? Saves several GB, " "but rewrites every binary -- slower, and harder to " - "debug a crash afterwards. NOT YET IMPLEMENTED: the " - "answer is stored and the build says so at the end " - "rather than acting on it (yes/no)", + "debug a crash afterwards. Runs as `lfs-helper " + "strip --run` when the build finishes (yes/no)", "no"), # The account you will actually log in as. The `init-accounts` step reads # this; without it, a freshly booted system has only root, with no password. @@ -1439,8 +1539,8 @@ def _prompt_session_config(cfg, force=False): # the fix is to work out which config key to edit. if k == "lfs_mount" and val: while _is_host_path(val): - sys.stderr.write( - " !! %s is part of the running system -- refusing.\n" + fail( + " !! %s is part of the running system -- refusing." " The LFS mount point is an empty directory the LFS\n" " partition mounts on, e.g. /mnt/lfs.\n" % val) val = input(f" {desc} [{cur}]: ").strip() or cur @@ -1496,15 +1596,13 @@ def _emit_or_run(args, name, script, needs_lfs=False): print(script) if not args.run: if needs_lfs and not _lfs_dir(): - sys.stderr.write("\n! no LFS mount set. Configure it: " - "lfs build-system session\n") + fail('\n! no LFS mount set. Configure it: lfs build-system session') dry_run_note("execute") return if needs_lfs: require_mounted_lfs() if os.geteuid() != 0: - sys.stderr.write("\n! not root -- these steps write under $LFS / add " - "users and will likely fail.\n") + fail('\n! not root -- these steps write under $LFS / add users and will likely fail.') rc = _run_bash(script) if rc != 0: sys.stderr.write(f"\n{name}: exited {rc}\n") @@ -1752,9 +1850,40 @@ def cmd_bs_session(args): if active and not args.reconfigure: print(f"Active session: {mount} is already mounted.") else: - print("Session config:") + # SAY WHICH FILE, AND WHOSE PREFIXES. + # + # The user set pkgusr_prefix empty on their HOST and then saw + # pkgusr_prefix = (unset) + # in the session that builds the new system -- the host's setting + # about to name the accounts of a different machine. 1.13.51 stopped + # the host's prefixes reaching the chroot's environment; this is the + # same leak one step earlier, in the session config itself. + # + # A tree that already has a config decides its own prefixes. And the + # file being used is named, because "which config is this?" was the + # question that could not be answered. + print("Session config: (%s)" % config_path()) + _tree = cfg.get("lfs_mount") or "/mnt/lfs" for k, desc, _ in _SESSION_KEYS: - print(f" {k:<16} = {cfg.get(k, '') or '(unset)'}") + _v = cfg.get(k, "") + _note = "" + if k in ("pkgusr_prefix", "cfguser_prefix", "collector_prefix"): + _tv = _tree_prefix(_tree, k, None) + if _tv is not None and _tv != (_v or "").rstrip("_"): + _v, _note = _tv, " <- from the tree's own config" + # AN EMPTY PREFIX IS A DECISION, NOT A GAP. + # + # "pkgusr_prefix = (unset)" read as "nothing chosen yet" when it + # was in fact an empty value -- and an empty prefix means every + # account in the BUILT system is named plainly (mesa, not + # p_mesa). The display could not tell the two apart, which is + # the same ambiguity that cost 1.13.26 and 1.13.69. + if not _v and k.endswith("_prefix"): + _shown = ("(none -- accounts will be named without a prefix)" + if k in cfg else "(unset -- will ask)") + else: + _shown = _v or "(unset)" + print(f" {k:<16} = {_shown}{_note}") have_all = all(cfg.get(k) for k, _, d in _SESSION_KEYS if d != "" or k == "lfs_device") if sys.stdin.isatty(): @@ -1791,8 +1920,7 @@ def cmd_bs_session(args): require_lfs_for_root() if os.geteuid() != 0: - sys.stderr.write("! session --run must be run as root (it mounts " - "partitions and chowns the build tree).\n") + fail('! session --run must be run as root (it mounts partitions and chowns the build tree).') sys.exit(2) import subprocess subprocess.run(["mkdir", "-pv", mount], cwd="/") @@ -1948,11 +2076,7 @@ def cmd_bs_get_sources(args): # Verified both ways -- rc was 0 either way. Check the status itself. rc = _run_bash(f'cd "{srcdir}" && md5sum -c md5sums --quiet') if rc != 0: - sys.stderr.write( - "\n! some downloads do not match their md5sum.\n" - " The lines above name them. A mirror returning an error\n" - " page with status 200 looks exactly like this.\n" - " Delete those files and run get-sources again.\n") + fail('\n! some downloads do not match their md5sum.\n The lines above name them. A mirror returning an error\n page with status 200 looks exactly like this.\n Delete those files and run get-sources again.') sys.exit(1) print("all md5sums OK") if fail: @@ -2041,6 +2165,14 @@ def _pkg_glob_for(pkgver, tarpfx): pats = [f"{st}.{e}" for st in stems for e in exts] # last resort: allow a suffix (tcl8.6.16-src.tar.gz, foo-1.2-source.tar.gz) pats += [f"{st}*.{e}" for st in stems for e in ("tar.*", "tgz")] + # very last: an INFIX between name and version -- SQLite-3510200 ships as + # sqlite-autoconf-3510200.tar.gz, a new package in 13.0 that broke unpack. + # The unpack loop's doc filter keeps this from grabbing sqlite-doc-* or + # systemd-man-pages-*. (from the SysV-book session) + m = re.match(r"(.+?)-([0-9][\w.]*)$", pkgver or "") + if m: + pats += [f"{m.group(1).lower()}-*{m.group(2)}.{e}" + for e in ("tar.*", "tgz")] return " ".join(pats) @@ -2194,7 +2326,7 @@ def crosschain_script_path(ver, name): return os.path.join(crosschain_dir(ver), f"{name}.sh") -CROSSCHAIN_SCRIPT_VERSION = 17 # v17: test scaffolding stays with the tests +CROSSCHAIN_SCRIPT_VERSION = 18 # v18: the runner moved out into lfs-phases _INSTALL_BLOCK_RE = re.compile(r'\b(make|ninja)\b[^\n]*\binstall\b') @@ -2366,286 +2498,72 @@ def _indent(text, tab="\t"): for ln in text.splitlines()) +def _fn_body(cmds, empty=": # (nothing to do in this phase)"): + """A command block as a function body. NOT indented: the books are full + of heredocs (cat > file << "EOF" ... EOF), and an indented terminator is + no terminator. The commands read exactly as the book prints them.""" + if not cmds or not cmds.strip(): + return empty + body = cmds.strip("\n") + # a body of comments alone is a syntax error to bash; give it a command + if not any(ln.strip() and not ln.lstrip().startswith("#") + for ln in body.split("\n")): + body += "\n:" + return body + + def _crosschain_script_body(name, sid, pkgver, glob, cmds): + """A generated install script: the package's variables at the top, its + commands in one function per phase, and one line at the end that hands + over to the shared runner (lfs-phases). Everything that is the same for + every package -- finding the tarball, unpacking, running each phase as its + own process under set -e, resuming in the build subdir, the dispatcher -- + lives in the runner, not in every script.""" ts = time.strftime("%Y-%m-%d %H:%M:%S") build_cmds, install_cmds, config_cmds, test_cmds = \ _split_crosschain_phases(cmds) + book = "LFS %s" % (load_config().get("default") or "?") L = [] w = L.append w("#!/bin/bash") w("############################################") w(f"### {name} ({pkgver})") + # The SHARED stamp: both generators write this line, so a tool can ask + # what a script is instead of guessing from its dialect. `origin` says + # whose format counter the number belongs to. w(f"{CROSSCHAIN_MARK} v{CROSSCHAIN_SCRIPT_VERSION}: {name}") - w("### phased build script -- run a single phase with its name as $1:") - w("### all (default) | unpack | build | install | configure | test") - w("### re-run just 'install' after fixing a failure, without recompiling.") - w("###") + w(f"### book : {book}") + w(f"### generated : {ts}") + w(f"### script format v{CROSSCHAIN_SCRIPT_VERSION} origin=lfs") + w("### phases: all (default) | unpack | build | test | install | configure") w("### This is YOUR editable script -- lfs reuses it as-is on every --run.") - w(f"### To pull a fresh copy from the book (discarding edits):") - w(f"### lfs build-system crosschain {name} --regenerate\n") - w(f'section="{sid}"') + w(f"### Fresh copy from the book (discarding edits): " + f"lfs build-system crosschain {name} --regenerate") + w("") w(f'name_version="{pkgver}"') w(f'pkg_glob="{glob}"') + w(f'section="{sid}"') + w('pkgusr_stage="cross"') w("") - w("# safety net: these are normally embedded by 'lfs build-system crosschain") - w("# --run' itself, but if you run this script directly (e.g. from inside your") - w("# own 'su - lfs' session) it still notices and fixes an unset LFS_TGT/PATH,") - w("# and refuses rather than silently building against the wrong toolchain.") - w('if [ -z "$LFS" ]; then') - w('\techo "!! \\$LFS is not set -- refusing to guess. ' - 'export LFS=/mnt/lfs (see book 2.6) and re-run." >&2') - w('\texit 2') - w("fi") - w('if [ -z "$LFS_TGT" ]; then') - w('\tLFS_TGT=$(uname -m)-lfs-linux-gnu') - w('\texport LFS_TGT') - w('\techo "note: \\$LFS_TGT was unset -- defaulting to $LFS_TGT (book 4.4)" >&2') - w("fi") - w('case ":$PATH:" in') - w('\t*":$LFS/tools/bin:"*) ;;') - w('\t*) PATH="$LFS/tools/bin:$PATH"; export PATH') - w('\t echo "note: added $LFS/tools/bin to \\$PATH (book 4.4)" >&2 ;;') - w("esac") - w('if [ -z "$MAKEFLAGS" ]; then') - w('\texport MAKEFLAGS="-j$(nproc)"') - w('\techo "note: \\$MAKEFLAGS was unset -- defaulting to $MAKEFLAGS "' - '"(book 4.4, use all cores)" >&2') - w("fi") - - w('\n# exported: the phase bodies below run as separate `bash -s` processes,') - w('# so anything they reference must be in the ENVIRONMENT, not just a shell') - w('# variable of this script.') - w('export LFS') - w('# Two directories, two jobs.') - w('# SOURCES_DIR the downloaded tarballs and patches. Written once by') - w('# get-sources, read-only to every build, root:root 1777.') - w('# BUILD_ROOT scratch: unpacked trees, build markers, anything a') - w('# book command drops beside the source. Deleted whole.') - w('# They used to be one directory, which is why a rebuild had to pick') - w('# stale markers and foreign ownership out of the tarball store.') - w('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"') - w('BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"') - w('export SOURCES_DIR BUILD_ROOT') - w('pkg_dir=""') - w("") - w("# Make BUILD_ROOT look like the sources directory to anything that") - w("# walks up out of an unpacked tree. The book writes commands like") - w("# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1") - w("# from INSIDE the unpacked directory, so '..' has to hold the") - w("# downloaded files or the command fails.") - w("#") - w("# Link EVERY downloaded file rather than the ones some package is known") - w("# to reach for. A list of those would be a second thing to keep in") - w("# sync with the book, and it would be wrong the first time a new") - w("# release adds a package that does the same trick. Linking the lot") - w("# means any '../<anything-we-downloaded>' resolves, for any package,") - w("# in any future book -- and it costs a few hundred symlinks.") - w("_link_sources() {") - w('\t[ -d "$SOURCES_DIR" ] || return 0') - w('\tmkdir -p "$BUILD_ROOT" || return 1') - w('\tlocal f b') - w('\t# a link whose target went away (re-downloaded, renamed) would make') - w('\t# the glob in unpack_pkg match a file that cannot be read') - w('\tfor b in "$BUILD_ROOT"/*; do') - w('\t\t[ -L "$b" ] || continue') - w('\t\t[ -e "$b" ] || rm -f "$b"') - w('\tdone') - w('\tfor f in "$SOURCES_DIR"/*; do') - w('\t\t[ -f "$f" ] || continue') - w('\t\tb="$BUILD_ROOT/${f##*/}"') - w('\t\t# never shadow real content: an unpacked tree wins over a link') - w('\t\tif [ -e "$b" ] && [ ! -L "$b" ]; then continue; fi') - w('\t\tln -sfn "$f" "$b" 2>/dev/null || true') - w('\tdone') - w("}\n") - w("# Run a phase body as its OWN process, from a FILE -- never piped in on") - w("# stdin. A book command that reads stdin (Expect's PTY check does:") - w("# python3 -c 'from pty import spawn; spawn([\"echo\", \"ok\"])' )") - w("# would otherwise swallow the rest of the script and silently skip the") - w("# real build. stdin is /dev/null so nothing can consume it or hang.") - w("_phase_body() {") - w('\tlocal _f; _f="$(mktemp)" || return 1') - w('\tcat > "$_f"') - w('\tbash "$_f" </dev/null') - w('\tlocal _r=$?') - w('\trm -f "$_f"') - w('\treturn $_r') - w("}\n") - w("_enter_build() {") - w('\tcd "$BUILD_ROOT" || return 1') - w(f'\t[ -z "$pkg_dir" ] && pkg_dir="$(cat .cc-dir-{name} 2>/dev/null)"') - w('\tif [ -n "$pkg_dir" ] && [ -d "$pkg_dir" ]; then cd "$pkg_dir" || return 1') - w('\telse echo "no unpacked source in $BUILD_ROOT -- run: $0 unpack" >&2; return 1; fi') - w("}\n") - - w("# resume in the SAME subdir build_pkg finished in (e.g. pkg_dir/build),") - w("# not just pkg_dir -- the book's own post-build commands (install,") - w("# post-install fixups) assume they're still there, same as reading the") - w("# book top-to-bottom would put you.") - w("_resume_build_dir() {") - w(f'\tif [ -f "$BUILD_ROOT/.cc-build-{name}" ] && ' - f'[ -d "$(cat "$BUILD_ROOT/.cc-build-{name}")" ]; then') - w(f'\t\tcd "$(cat "$BUILD_ROOT/.cc-build-{name}")"') - w('\t\treturn 0') - w('\tfi') - w('\t# No marker: build_pkg records the build directory as its LAST step, so') - w('\t# a build that died part-way (a failed test suite, an interrupted') - w('\t# compile) never wrote one. Fall back to the usual out-of-tree layout') - w('\t# -- without this, `make install` runs in the source root and glibc') - w('\t# stops with "objdir must be defined by the build-directory Makefile".') - w('\tlocal _d') - w('\tfor _d in build builddir _build obj; do') - w('\t\tif [ -d "$_d" ] && { [ -f "$_d/Makefile" ] || [ -f "$_d/build.ninja" ] ' - '|| [ -f "$_d/config.status" ]; }; then') - w('\t\t\techo "# resuming in ./$_d (no build marker was recorded)" >&2') - w('\t\t\tcd "$_d"; return 0') - w('\t\tfi') - w('\tdone') - w('\treturn 0') - w("}\n") - - w("unpack_pkg() {") - w("#### UNPACK ####") - w('\t_link_sources') - w('\tcd "$BUILD_ROOT" || return 1') - w('\t# Pick the source tarball. Two traps here:') - w('\t# * do NOT use `ls $pkg_glob` -- patterns that match nothing make ls') - w('\t# fail and swallow the one that did match;') - w('\t# * several packages ship docs beside the source (tcl8.6.16-html.tar.gz') - w('\t# sorts BEFORE tcl8.6.16-src.tar.gz), so prefer an explicit -src') - w('\t# archive and never pick a documentation one.') - w('\tpkg=""') - w('\tfor _c in $pkg_glob; do') - w('\t\t[ -e "$_c" ] || continue') - w('\t\tcase "$_c" in *-src.*|*-source.*) pkg="$_c"; break ;; esac') - w('\tdone') - w('\tif [ -z "$pkg" ]; then') - w('\t\tfor _c in $pkg_glob; do') - w('\t\t\t[ -e "$_c" ] || continue') - w('\t\t\tcase "$_c" in') - w('\t\t\t\t*-html.*|*-doc.*|*-docs.*|*-man.*|*-manual.*|*-tests.*) continue ;;') - w('\t\t\tesac') - w('\t\t\tpkg="$_c"; break') - w('\t\tdone') - w('\tfi') - w('\t[ -n "$pkg" ] || { echo "no $pkg_glob in $SOURCES_DIR -- run: ' - 'lfs build-system get-sources --run" >&2; return 1; }') - w('\tdir=$(tar tf "$pkg" | head -n1 | cut -d/ -f1)') - w('\t# --no-same-owner: as root, tar would restore the UIDs recorded IN the') - w('\t# archive, littering /sources with owners like 8282 or 15399 that do') - w('\t# not exist here. The unpacking user should own what it unpacks.') - w('\trm -rf "$dir"; tar --no-same-owner -xf "$pkg"') - w('\tpkg_dir="$dir"') - w(f'\techo "$pkg_dir" > "$BUILD_ROOT/.cc-dir-{name}"') - w('\techo "unpacked into $BUILD_ROOT/$pkg_dir"') - w("#### UNPACK DONE ####") - w("}\n") - - w("# Each phase body runs as a SEPARATE 'bash -s' process. This matters:") - w("# bash disables 'set -e' inside any function reached from a tested") - w("# context (an && chain, an || handler), and that suppression is even") - w("# inherited by nested subshells -- so a failing 'make' would be silently") - w("# ignored and the step would wrongly report success. A separate process") - w("# has its own -e state that nothing upstream can suppress, so failures") - w("# always propagate.") w("build_pkg() {") - w("#### BUILD ####") - w("\t_enter_build") - w("\t_phase_body <<'__LFS_PHASE__'") - w("set -e") - # record the build directory as soon as the commands have cd'd into it, so - # a later failure still leaves install/configure able to find it - w("_mark_build_dir() { pwd > \"$BUILD_ROOT/.cc-build-%s\"; }" % name) - w("trap _mark_build_dir EXIT") - w(build_cmds if build_cmds.strip() else ': # (nothing to build)') - w(f'pwd > "$BUILD_ROOT/.cc-build-{name}"') - w("__LFS_PHASE__") - w("\t_rc=$?; [ $_rc -eq 0 ] || return $_rc") - w("#### BUILD DONE ####") - w("}\n") - + w(_fn_body(build_cmds, ": # (nothing to build)")) + w("}") + w("") w("install_pkg() {") - w("#### INSTALL ####") - w("\t_enter_build") - w("\t_resume_build_dir") - w("\t_phase_body <<'__LFS_PHASE__'") - w("set -e") - w(install_cmds if install_cmds.strip() - else ': # (this step installs during the build phase)') - w("__LFS_PHASE__") - w("\t_rc=$?; [ $_rc -eq 0 ] || return $_rc") - w("#### INSTALL DONE ####") - w("}\n") - + w(_fn_body(install_cmds, ": # (this step installs during the build phase)")) + w("}") if config_cmds.strip(): + w("") w("configure_pkg() {") - w("#### CONFIGURE ####") - w("\t_enter_build") - w("\t_resume_build_dir") - w("\t_phase_body <<'__LFS_PHASE__'") - w("set -e") - w(config_cmds) - w("__LFS_PHASE__") - w("\t_rc=$?; [ $_rc -eq 0 ] || return $_rc") - w("#### CONFIGURE DONE ####") - w("}\n") - else: - w("configure_pkg() { : ; }\n") - + w(_fn_body(config_cmds)) + w("}") + w("") w("test_pkg() {") - if test_cmds.strip(): - w("#### TEST ####") - w("\t_enter_build") - w("\t_resume_build_dir") - w("\t# The book expects some tests to fail, so this reports the result") - w("\t# instead of aborting -- read the summary and judge for yourself.") - w("\t_phase_body <<'__LFS_PHASE__'") - w(test_cmds) - w("__LFS_PHASE__") - w('\t_rc=$?') - w('\tif [ $_rc -ne 0 ]; then') - w('\t\techo "!! test suite exited $_rc -- check the summary above." >&2') - w('\t\techo " Some failures are normal (the book lists the usual"' - ' >&2') - w('\t\techo " suspects); this does NOT stop the install." >&2') - w('\tfi') - w('\treturn 0') - w("#### TEST DONE ####") - else: - w('\t: # the book defines no test suite for this step') - w('\techo "no test suite for this step"') - w("}\n") - - w('_phase_fail() { echo "!! phase \'$1\' FAILED (exit $2)" >&2; exit "$2"; }') - w('case "${LFS_CC_PHASE:-${1:-all}}" in') - w("\tall) unpack_pkg || _phase_fail unpack $?") - w("\t build_pkg || _phase_fail build $?") - w('\t [ "${LFS_RUN_TESTS:-0}" = "1" ] && test_pkg') - w("\t install_pkg || _phase_fail install $?") - w("\t configure_pkg || _phase_fail configure $? ;;") - w("\tunpack) unpack_pkg || _phase_fail unpack $? ;;") - w("\tbuild) build_pkg || _phase_fail build $? ;;") - w("\tinstall) install_pkg || _phase_fail install $? ;;") - w("\tconfigure) configure_pkg || _phase_fail configure $? ;;") - w("\ttest) test_pkg || _phase_fail test $? ;;") - w('\t*) echo -e "usage: $0 {all|unpack|build|install|configure|test}\\n' - ' all full run (default)\\n' - ' unpack fetch tarball from $LFS/sources + extract\\n' - ' build configure + compile (needs unpack)\\n' - ' install install step only -- re-run after fixing a failure\\n' - ' configure post-install fixups (symlinks, sanity checks)\\n' - ' test run this step\'s tests (edit test_pkg first)" ;;') - w("esac") - - header_tag = f"{CROSSCHAIN_MARK} v{CROSSCHAIN_SCRIPT_VERSION}: {name}" - body = "\n".join(L) + "\n" - # Stamp which book this came from: the scripts outlive the book version, - # and "where did this come from?" is otherwise unanswerable. - book = "LFS %s" % (load_config().get("default") or "?") - body = body.replace(header_tag, - f"{header_tag}\n### book : {book}" - f"\n### generated : {ts}", 1) - return body + w(_fn_body(test_cmds, ": # the book defines no test suite for this step")) + w("}") + w("") + w('. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"') + return "\n".join(L) + "\n" def ensure_crosschain_script(args, ver, name, sid, tarpfx, force_regen=False, @@ -2766,8 +2684,7 @@ def _require_lfs_writable_by_lfs(lfs): try: want = _pwd.getpwnam("lfs").pw_uid except KeyError: - sys.stderr.write("! the 'lfs' user doesn't exist yet.\n" - " Run: lfs build-system add-user --run\n") + fail("! the 'lfs' user doesn't exist yet.\n Run: lfs build-system add-user --run") sys.exit(2) # After book 7.2 the whole tree is handed to root on purpose, and chapters # 5-6 are finished -- don't demand it be given back to lfs. @@ -2846,19 +2763,14 @@ def cmd_bs_fix_ownership(args): dry_run_note("apply") return if os.geteuid() != 0: - sys.stderr.write("! must be run as root.\n") + fail('! must be run as root.') sys.exit(2) # The third copy of this chown, and the third guard. `fix-ownership` is a # chapter 5-6 repair: it hands the tree to the build user so packages can # install as `lfs`. After chapter 7 the packages ARE the users, and doing # it would strip all of them in one pass. if not _handover_is_safe(lfs): - sys.stderr.write( - "! not doing that: package users own files here, so this tree is\n" - " past chapter 7. chown -R lfs would strip every package's\n" - " ownership, and only the manifests could rebuild it.\n" - " To repair ownership on a built tree, inside the chroot:\n" - " lfs-helper verify --fix\n") + fail("! not doing that: package users own files here, so this tree is\n past chapter 7. chown -R lfs would strip every package's\n ownership, and only the manifests could rebuild it.\n To repair ownership on a built tree, inside the chroot:\n lfs-helper verify --fix") sys.exit(2) for p in targets: _run_bash(f'chown -R lfs "{p}"') @@ -2997,7 +2909,7 @@ def require_complete_config(): missing = [(k, why, how) for k, why, how in REQUIRED_CONFIG if not cfg.get(k)] if not missing: return - sys.stderr.write("! the build session is not fully configured:\n") + fail('! the build session is not fully configured:') for k, why, how in missing: sys.stderr.write(f" {k:<18} -- {why}\n" f" {'':<18} set it with: {sudo_prefix()}{how}\n") @@ -3102,7 +3014,101 @@ def strip_binaries(): return v in ("1", "on", "true", "yes") +# The keys worth asking about, with what they mean. Anything else can still +# be set with `lfs config --target <key> <value>`. +_TARGET_KEYS = ( + ("pkgusr_prefix", + "prefix for package users in the new system (p -> p_mesa; empty -> mesa)"), + ("cfguser_prefix", "prefix for its config users (cfg -> cfg_bootscripts)"), + ("collector_prefix", "prefix for its shared install-dir groups"), + ("pkgusr_subdir", "folder under the base for package homes (pkgusr)"), + ("cfguser_subdir", "folder under the base for config-user homes (cfg)"), + ("pkgusr_home", "where package homes live (/usr/src/pkgusr); the " + "subdir is appended if you give just the base"), +) + + +def _target_config_path(): + return os.path.join(store_dir(), "target-config.json") + + +def cmd_config_target(args): + """Show or change the config the BUILT system will get. + + Laid out like `packagemanager config --show`: value, then where it comes + from. The user asked for the config commands to look and behave the same + across the tools, and this is the first of them to follow that shape. + """ + path = _target_config_path() + try: + with open(path) as f: + cfg = json.load(f) + except (OSError, ValueError): + cfg = {} + # INTERACTIVE, LIKE THE OTHERS. + # + # `packagemanager config` asks for what is missing and `packagemanager env + # --ask` walks the build variables; this took only a key and a value -- so + # the config a person most needs to set deliberately, the NEW SYSTEM'S, + # was the one with no guided way to set it. + if getattr(args, "ask", False): + print() + print("# === Config the BUILT system will get ===") + print(" file: %s" % path) + print(" Enter keeps the current value; '-' clears it (means 'none').") + print() + for key, what in _TARGET_KEYS: + cur = cfg.get(key) + shown = ("(none)" if cur == "" else + cur if cur is not None else "this host's value") + try: + ans = input(" %s\n %-16s [%s]: " % (what, key, shown)).strip() + except EOFError: + print() + break + if ans == "": + continue + cfg[key] = "" if ans == "-" else ans + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + with open(path, "w") as f: + json.dump(cfg, f, indent=2, sort_keys=True) + f.write("\n") + print() + print(" saved %s" % path) + print(" it reaches the build system on the next `lfs run`.") + return + if args.key and args.value is not None: + cfg[args.key] = args.value + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w") as f: + json.dump(cfg, f, indent=2, sort_keys=True) + f.write("\n") + print(" %s = %r -> %s" % (args.key, args.value, path)) + print(" it reaches the build system on the next `lfs run`.") + return + print() + print("# === Config the BUILT system will get ===") + print(" file: %s%s" % (path, "" if os.path.isfile(path) + else " (not created yet)")) + print() + if not cfg: + print(" nothing set -- the build system will use this host's values") + print(" for the shared keys (prefixes, layout, books).") + print() + print(" set one with, for example:") + print(" lfs config --target pkgusr_prefix p") + print(" lfs config --target pkgusr_subdir packages") + return + for k in sorted(cfg): + print(" %-22s = %r" % (k, cfg[k])) + print() + print(" these override this host's values when the tree is built;") + print(" a config the tree already has wins over both.") + + def cmd_config(args): + if getattr(args, "target", False): + return cmd_config_target(args) """Show or change every setting, and show where things live on disk.""" cfg = load_config() @@ -3558,16 +3564,16 @@ def write_chroot_env(lfs, quiet=False): f'# this prefix when it reads them.\n' f'export LFS_HOST_MOUNT="{lfs}"\n' f'# prefix for collector groups -- see `lfs config`\n' - f'export LFS_COLLECTOR_PREFIX="{collector_prefix()}"\n' + f'export LFS_COLLECTOR_PREFIX="{_tree_prefix(lfs, "collector_prefix", collector_prefix())}"\n' f'# prefix for package users. lfs-helper has always READ this;\n' f'# nothing wrote it, so the default applied by accident.\n' - f'export LFS_PKGUSR_PREFIX="{pkgusr_prefix()}"\n' + f'export LFS_PKGUSR_PREFIX="{_tree_prefix(lfs, "pkgusr_prefix", pkgusr_prefix())}"\n' f'# prefix for config-step users. Its own, NOT the package\n' f'# prefix stacked on top -- one account, one prefix.\n' - f'export LFS_CFGUSR_PREFIX="{cfguser_prefix()}"\n' + f'export LFS_CFGUSR_PREFIX="{_tree_prefix(lfs, "cfguser_prefix", cfguser_prefix())}"\n' f'# where accounts live, by kind -- see LAYOUT_DEFAULTS\n' - f'export LFS_PKGUSR_ROOT="{layout("pkgusr_home", target=True)}"\n' - f'export LFS_CFGUSR_ROOT="{layout("cfguser_home", target=True)}"\n' + f'export LFS_PKGUSR_ROOT="{_accounts_root("pkgusr")}"\n' + f'export LFS_CFGUSR_ROOT="{_accounts_root("cfg")}"\n' f'# languages to keep -- see `lfs-helper prune-locales`\n' f'export LFS_LOCALES="{cfg.get("locales") or "en de"}"\n' f'# answers collected by `lfs build-system session`, so the\n' @@ -3606,8 +3612,31 @@ def _find_helper_source(name="lfs-helper"): # They cannot be USED until the last build step provides Python and the two # book-parsing modules. That is fine: they are there, and `packagemanager # setup` completes them. -TOOLCHAIN_SCRIPTS = ("lfs-helper", "packagemanager", "packagemanager_install", - "blfs", "lfs") +# packagemanager_install is retired (1.12.0): folded into `lfs-helper +# pm-install`. Four tools is the endpoint item 6 named. +# Every tool `make install` places, so a new one is refreshed into the chroot +# without anyone remembering this list. lfs-kernel was added to the Makefile +# and not here, so `lfs run` quietly refreshed five of six tools -- and the +# sixth was the one just written. The Makefile's TOOLS line is the source of +# truth; the literal tuple below is the fallback for a tree with no Makefile +# beside the tools. +def _toolchain_scripts(): + here = os.path.dirname(os.path.abspath(__file__)) + mk = os.path.join(here, "Makefile") + try: + with open(mk) as fh: + for line in fh: + if re.match(r"^TOOLS\s*:?=", line): + names = line.split("=", 1)[1].split() + if names: + return tuple(names) + except OSError: + pass + return ("lfs-helper", "packagemanager", "blfs", "lfs", "lfs-phases", + "lfs-kernel", "lfs-sysvbook") + + +TOOLCHAIN_SCRIPTS = _toolchain_scripts() # The package user that owns them. # @@ -3646,6 +3675,20 @@ def install_helper(lfs, quiet=False): if name == "lfs-helper": helper_dest = dest + # Sweep retired tools OUT of the tree. packagemanager_install was folded + # into `lfs-helper pm-install` (1.12.0); a stale copy left on the chroot's + # PATH is exactly the "old code running after the fix" this sync exists to + # prevent. + for retired in ("packagemanager_install",): + r = os.path.join(dest_dir, retired) + if os.path.lexists(r): + try: + os.unlink(r) + if not quiet: + print("# removed retired %s from the tree" % retired) + except OSError as e: + fail("! could not remove retired %s: %s" % (retired, e)) + # Give them an owner. The account may not exist yet -- chapter 7 creates # /etc/passwd -- so this is best-effort now and repeated on the next run, # by which time it will exist. Failing to chown is not worth stopping a @@ -3657,10 +3700,10 @@ def install_helper(lfs, quiet=False): print("# installed %s -> %s (on PATH in the chroot)" % (", ".join(copied), dest_dir)) for name in missing: - sys.stderr.write("! %s not found next to this tool -- the chroot " - "won't have it.\n" % name) + fail("! %s not found next to this tool -- the chroot " + "won't have it." % name) if helper_dest is None and not quiet: - sys.stderr.write("! lfs-helper is missing: the chroot cannot build.\n") + fail('! lfs-helper is missing: the chroot cannot build.') return helper_dest @@ -3682,7 +3725,7 @@ def _chown_tools(lfs, dest_dir, names, quiet=False): os.chown(p, uid, uid) n += 1 except OSError as e: - sys.stderr.write("! could not give %s to %s: %s\n" % (p, user, e)) + fail("! could not give %s to %s: %s" % (p, user, e)) if n and not quiet: print("# gave %d tool(s) to '%s'" % (n, user)) @@ -3707,23 +3750,11 @@ def _uid_in_tree(lfs, user): def _chrootify(text): - """Turn a cross-compile script into an IN-CHROOT one. Inside the chroot the - final system is simply '/', so $LFS must not be required (the cross-build - guard would refuse to run), sources live at /sources, and there's no - cross-compiler prefix to put on PATH.""" - text = text.replace('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"', - 'SOURCES_DIR="${SOURCES_DIR:-/sources}"') - text = text.replace('BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"', - 'BUILD_ROOT="${BUILD_ROOT:-/build}"') - text = text.replace('export LFS\n', 'export LFS="${LFS:-/}"\n') - # drop the "refusing to guess without $LFS" guard: correct outside, wrong here - text = re.sub(r'if \[ -z "\$LFS" \]; then\n.*?\nfi\n', '', text, count=1, - flags=re.S) - # $LFS/tools/bin doesn't exist inside the chroot -- the real toolchain is on - # the normal PATH - text = re.sub(r'case ":\$PATH:" in\n.*?\nesac\n', '', text, count=1, - flags=re.S) - return text + """Turn a cross-compile script into an IN-CHROOT one. Inside the chroot + the final system is simply '/', so $LFS is not required, sources live at + /sources and there is no cross-compiler prefix to put on PATH. The runner + (lfs-phases) decides all of that from one variable.""" + return text.replace('pkgusr_stage="cross"', 'pkgusr_stage="chroot"', 1) def _plain_root_script(name, sid, title, cmds): @@ -3764,6 +3795,7 @@ def _plain_root_script(name, sid, title, cmds): ############################################ ### {name} -- {title} {CROSSCHAIN_MARK} v{CROSSCHAIN_SCRIPT_VERSION}: {name} +### script format v{CROSSCHAIN_SCRIPT_VERSION} origin=lfs ### section : {sid} ### book : {book} ### generated : {ts} @@ -4072,8 +4104,8 @@ def _install_collector_groups(lfs): return src = os.path.expanduser(src) if not os.path.isfile(src): - sys.stderr.write("! collector_import_file is set but %s does not " - "exist\n" % src) + fail("! collector_import_file is set but %s does not " + "exist" % src) return import shutil # groups/, where lfs-helper looks for it. These two paths are the whole @@ -4121,8 +4153,8 @@ def _gen_blfs_extra(outdir, order, unresolved): return 0 blfs = _find_blfs() if not blfs: - sys.stderr.write("! extra_packages is set (%s) but the 'blfs' tool is " - "not next to this one -- skipping\n" % " ".join(extras)) + fail("! extra_packages is set (%s) but the 'blfs' tool is " + "not next to this one -- skipping" % " ".join(extras)) return 0 import subprocess written = 0 @@ -4144,7 +4176,7 @@ def _gen_blfs_extra(outdir, order, unresolved): capture_output=True, text=True, timeout=600, cwd=tmp) except (OSError, subprocess.SubprocessError) as e: - sys.stderr.write("! blfs script %s: %s\n" % (pkg, e)) + fail("! blfs script %s: %s" % (pkg, e)) continue made = [f for f in os.listdir(tmp) if f.startswith("install_")] if r.returncode != 0 or not made: @@ -4153,7 +4185,7 @@ def _gen_blfs_extra(outdir, order, unresolved): if line.strip(): detail = ": " + line.strip() break - sys.stderr.write("! blfs has no script for '%s'%s\n" % (pkg, detail)) + fail("! blfs has no script for '%s'%s" % (pkg, detail)) continue with open(os.path.join(tmp, made[0])) as f: raw = f.read() @@ -4191,17 +4223,33 @@ def bootstrap_sources(): """ import subprocess try: - out = subprocess.run(["packagemanager", "setup", "--sources"], + # `bootstrap` since 1.11.7; `setup` is still accepted as an alias, so + # this keeps working against an older packagemanager on the tree. + out = subprocess.run(["packagemanager", "bootstrap", "--sources"], capture_output=True, text=True, timeout=180) + if out.returncode != 0 and not (out.stdout or "").strip(): + out = subprocess.run(["packagemanager", "setup", "--sources"], + capture_output=True, text=True, timeout=180) except (OSError, subprocess.SubprocessError): - warn("! could not run `packagemanager setup --sources`.") + warn("! could not run `packagemanager bootstrap --sources`.") warn(" The built system will have no way to fetch anything until you") warn(" put a download tool on it by hand.") return [] urls = [l.strip() for l in (out.stdout or "").splitlines() if l.strip().startswith(("http://", "https://", "ftp://"))] if not urls: - warn("! `packagemanager setup --sources` listed nothing.") + if not _blfs_book_on_host(): + # No package named here, deliberately: which packages the built + # system needs is packagemanager's list, and a copy of it in this + # file is the thing a test forbids. What lfs knows is that the + # answer comes from a book, and that there is no book. + warn("! No BLFS book on this machine, so nothing can say where") + warn(" those sources are -- and the built system cannot fetch the") + warn(" book until it has a download tool, which is itself looked") + warn(" up in the book. Break the circle here:") + warn(" blfs fetch") + warn(" then run get-sources again.") + warn("! `packagemanager bootstrap --sources` listed nothing.") for line in (out.stderr or "").splitlines(): warn(" %s" % line) return urls @@ -4278,6 +4326,84 @@ def _run_choose_book(): return True +def _blfs_book_on_host(): + """Is a BLFS book cached where install-tools will find it? + + Its own store, not lfs's -- /usr/share/blfs/books. This is the same list + install-tools copies into the tree, so asking here answers exactly the + question the built system will have. + + When $BLFS_STORE is set it is the ONLY place looked: an override that + half-applies is this project's oldest bug shape -- an empty overridden + store fell through to /usr/share/blfs and reported the system's books as + the override's. + """ + if os.environ.get("BLFS_STORE"): + cands = (os.environ["BLFS_STORE"],) + else: + cands = ("/usr/share/blfs", os.path.expanduser("~/.cache/blfs")) + for cand in cands: + d = os.path.join(cand, "books") + try: + if any(f.endswith((".html", ".html.gz")) for f in os.listdir(d)): + return True + except OSError: + continue + return False + + +def _run_choose_blfs_book(): + """Offer to fetch the BLFS book, HERE, where there is still a network. + + The chicken and egg this closes: on the built system `packagemanager + bootstrap` installs wget by looking it up in the BLFS book -- and without + a book there is nothing to look wget up in, while fetching the book needs + the wget that cannot be installed: + + wget: there is no BLFS book on this system to look it up in. + Could not install: wget + + The host has a network and the copy machinery already exists; all that was + missing was asking, at a moment when the answer can still be acted on. + + Declining is recorded, so the checklist does not ask again on every resume. + """ + print("\nThe built system needs the BLFS book to install anything beyond") + print("LFS itself -- including wget, which is how it would otherwise") + print("fetch the book. This machine has a network; that one will not") + print("until wget exists.") + print("\nFetching it now copies it into the tree with the tools.") + try: + ans = input("\nFetch the BLFS book now? [Y/n] ").strip().lower() + except (EOFError, KeyboardInterrupt): + print() + return False + if ans in ("n", "no", "skip"): + cfg = load_config() + cfg["blfs_book"] = "skip" + save_config(cfg) + print("\nSkipped. Get it later with `blfs fetch` on this machine, or") + print("inside the chroot once it has a download tool:") + print(" blfs import /path/to/BLFS-BOOK-...-nochunks.html") + return True + + blfs = _find_blfs() + if not blfs: + fail('! blfs is not on this machine, so it cannot fetch.\n Install the tools first: make install') + return False + rc = subprocess.call(blfs + ["fetch"]) + if rc != 0: + fail('\n! `blfs fetch` failed. It needs a working network here.\n Fetch it by hand, or skip and import it later:\n blfs books --discover\n blfs fetch <version>') + return False + if not _blfs_book_on_host(): + fail('! blfs fetch reported success but cached nothing.') + return False + cfg = load_config() + cfg["blfs_book"] = "fetched" + save_config(cfg) + return True + + def _run_banner(step, steps, cmd=None): """One consistent header before every step, so it is always clear what is happening now, how far along it is, and what comes next. Long steps print @@ -4306,12 +4432,67 @@ def _run_banner(step, steps, cmd=None): print("+" + "-" * (W - 2) + "+") +_STACK_VERSION_RE = re.compile(r"^#\s*stack-version:\s*(\S+)", re.M) + + +def _stack_version(path): + """The version a stack file declares, if it declares one.""" + try: + with open(path, encoding="utf-8", errors="replace") as f: + m = _STACK_VERSION_RE.search(f.read(2000)) + except OSError: + return None + return m.group(1) if m else None + + +def _version_tuple(v): + out = [] + for part in str(v).split("."): + digits = "".join(c for c in part if c.isdigit()) + out.append(int(digits) if digits else 0) + return tuple(out) + + +def _tree_prefix(lfs, key, fallback): + """The BUILT SYSTEM'S own prefix, not this host's. + + The chroot scripts export LFS_*_PREFIX, and the values were read from the + HOST -- so a prefix the user changed on their own machine was baked into + the build system's environment, where it overrode the tree's config. + That is how an empty LFS_PKGUSR_PREFIX reached a chroot whose config says + "p". + + The tree's config decides for the tree. The host's value is only a + fallback for a tree that has not been configured yet. + """ + for cand in ("usr/share/lfs/config.json", "root/.local/share/lfs/config.json"): + path = os.path.join(lfs, cand) + try: + with open(path) as f: + v = json.load(f).get(key) + if v is not None: + return str(v).rstrip("_") + except (OSError, ValueError): + continue + return fallback + + def _refresh_chroot_tools(lfs, quiet=False): - """Copy the tools into the tree if what is in there is out of date. + """Copy the TOOLS into the tree if what is in there is out of date. Compares content, so an unchanged tool is not touched and there is no - output in the common case.""" - import hashlib + output in the common case. + + All five, not just lfs-helper. Refreshing only the bash driver left the + python tools inside the chroot older than the host's, and every + packagemanager run opened with "These tools differ from the copies on the + host" plus advice to sync -- from a tool that had just been given the + chance to sync and had refreshed one file. + + Nothing here touches the tree's config or its books: `lfs set-default` + inside the chroot is the system's own decision, and entering must not + quietly undo it.""" + import hashlib, shutil def _md5(p): try: @@ -4320,16 +4501,195 @@ def _refresh_chroot_tools(lfs, quiet=False): except OSError: return None - src = _find_helper_source() - if not src: - return - dst = os.path.join(lfs, "usr", "bin", "lfs-helper") - if _md5(src) == _md5(dst): - return - install_helper(lfs, quiet=True) - if not quiet: - print("# lfs-helper in the chroot was out of date -- updated to %s " - "(build %s)" % (LFS_VERSION, _build_id())) + here = os.path.dirname(os.path.abspath(__file__)) + updated, failed, pairs = [], [], [] + # Sweep retired tools out of the tree first -- a stale + # packagemanager_install on the chroot's PATH is old code waiting to run. + for retired in ("packagemanager_install",): + r = os.path.join(lfs, "usr", "bin", retired) + if os.path.lexists(r): + try: + os.unlink(r) + updated.append(retired + " (removed, retired)") + except OSError as e: + failed.append("%s (%s)" % (retired, e)) + # THE STACKS TRAVEL WITH THE TOOLS. + # + # This refreshed the four binaries and nothing else, so a chroot kept the + # sway.stack it was first given -- the user updated the tools four times + # and kept reading a 39-entry file while the source had 47, wondering why + # the package count never changed. machine.conf is the exception: it is + # the user's own choice of build options, so it is only placed when it + # does not exist yet. + refreshed_stacks = [] + # WHERE THE STACKS ACTUALLY ARE. + # + # This looked only beside the running tool -- and the running tool is + # /usr/bin/lfs, so it looked in /usr/bin/stacks and found nothing, in + # silence. The host had 31 book entries and the chroot kept 23. A + # checkout keeps them beside the tool; an installed system keeps them in + # /etc/pkgusr/stacks. Look in both, nearest first. + _stack_src = "" + for _cand in (os.path.join(here, "stacks"), "/etc/pkgusr/stacks"): + if os.path.isdir(_cand) and os.path.abspath(_cand) != os.path.abspath( + os.path.join(lfs, "etc", "pkgusr", "stacks")): + _stack_src = _cand + break + if not _stack_src and not quiet: + print("# no stack files found to refresh (looked beside the tool and" + " in /etc/pkgusr/stacks)") + if _stack_src and os.path.isdir(_stack_src): + # kernel.conf lives beside the stacks, not in them, and lfs-kernel + # refuses to run without it -- so refreshing the tool and not the + # config would put a working command in the chroot that cannot start. + # An edited copy in the chroot is kept, as everywhere else. + # ...found either beside the tool (a checkout) or in /etc/pkgusr + # (an installed system), the same two places as the stacks. + for _cf in ("kernel.conf",): + _cs = "" + for _c2 in (os.path.join(here, _cf), + os.path.join("/etc/pkgusr", _cf)): + if os.path.isfile(_c2) and os.path.abspath(_c2) != os.path.abspath( + os.path.join(lfs, "etc", "pkgusr", _cf)): + _cs = _c2 + break + if not _cs: + continue + _cd = os.path.join(lfs, "etc", "pkgusr", _cf) + try: + os.makedirs(os.path.dirname(_cd), exist_ok=True) + if not os.path.exists(_cd): + shutil.copy2(_cs, _cd) + refreshed_stacks.append(_cf) + elif _md5(_cs) != _md5(_cd): + shutil.copy2(_cs, _cd + ".new") + refreshed_stacks.append(_cf + ".new (yours kept)") + except OSError as e: + fail("! could not refresh %s: %s" % (_cf, e)) + + _stack_dst = os.path.join(lfs, "etc", "pkgusr", "stacks") + try: + os.makedirs(_stack_dst, exist_ok=True) + for fn in sorted(os.listdir(_stack_src)): + # NOT EVERYTHING IN THAT DIRECTORY IS A STACK. + # + # .shipped/ is make install's record of what it last shipped + # (1.14.98), and .new/.bak are the merge leftovers. The + # refresh copied all of them into the chroot -- ".shipped/ + # desktop-user.sh" and a services.stack.new that had nothing + # to do with the chroot's own file. Bookkeeping does not + # travel. + if fn.startswith(".") or fn.endswith((".new", ".bak")): + continue + src_f = os.path.join(_stack_src, fn) + dst_f = os.path.join(_stack_dst, fn) + # A STACK ENTRY MAY HAVE A DIRECTORY. + # + # stacks/mytools/install_mytools is the package-user script + # the mytools step runs, and this loop took files only -- so + # the directory never crossed into the chroot and the step + # died with "missing .../mytools/install_mytools". The + # Makefile had the same fault (1.14.102) and was fixed there; + # this is the second copy of the same walk. + if os.path.isdir(src_f): + try: + os.makedirs(dst_f, exist_ok=True) + for sub in sorted(os.listdir(src_f)): + s_sub, d_sub = os.path.join(src_f, sub), os.path.join(dst_f, sub) + if not os.path.isfile(s_sub): + continue + if os.path.exists(d_sub) and _md5(s_sub) == _md5(d_sub): + continue + shutil.copy2(s_sub, d_sub) + os.chmod(d_sub, 0o755) + refreshed_stacks.append("%s/%s" % (fn, sub)) + except OSError as e: + fail("! could not refresh %s: %s" % (fn, e)) + continue + if not os.path.isfile(src_f): + continue + if fn == "machine.conf" and os.path.exists(dst_f): + continue # the machine's own options: never overwritten + if os.path.exists(dst_f) and _md5(src_f) == _md5(dst_f): + continue # already current + # NEVER GO BACKWARDS. A stack file edited in the chroot, or + # one from a newer release, must not be replaced by an older + # copy from the host -- the tools are versioned, so ask. + _sv, _dv = _stack_version(src_f), _stack_version(dst_f) + if _sv and _dv and _version_tuple(_dv) > _version_tuple(_sv): + if not quiet: + print("# kept the newer %s already in the chroot (%s > %s)" + % (fn, _dv, _sv)) + continue + # AND A STACK FILE THAT DIFFERS IS THE CHROOT'S OWN. + # + # Stack files carry no version, so the guard above never + # fires for them: the host's copy simply won. The user had + # just taken the 6-entry services.stack inside the chroot + # with --take-new, and this loop put the host's 3-entry file + # back over it. Same rule as `make install`: an existing + # file that differs is theirs, and the incoming one waits + # beside it. + if fn.endswith(".stack") and os.path.exists(dst_f): + shutil.copy2(src_f, dst_f + ".new") + if not quiet: + print("# %s differs in the chroot -- kept it, new one" + " is %s.new" % (fn, fn)) + continue + shutil.copy2(src_f, dst_f) + if fn.endswith(".sh"): + os.chmod(dst_f, 0o755) + refreshed_stacks.append(fn) + if refreshed_stacks and not quiet: + print("# refreshed in the chroot: %s" % ", ".join(refreshed_stacks)) + except OSError as e: + fail("! could not refresh the stacks: %s" % e) + + for tool in TOOLCHAIN_SCRIPTS: + src = _find_helper_source() if tool == "lfs-helper" \ + else os.path.join(here, tool) + if not src or not os.path.isfile(src): + continue + pairs.append((tool, src)) + dst = os.path.join(lfs, "usr", "bin", tool) + if _md5(src) == _md5(dst): + continue + try: + shutil.copy(src, dst) + os.chmod(dst, 0o755) + except OSError as e: + failed.append("%s (%s)" % (tool, e)) + continue + # Verify, rather than assume. A copy that does not stick -- a + # read-only mount, a busy binary, the wrong destination -- would + # otherwise show up only as this same message on every entry, which + # is exactly what it did. + if _md5(src) != _md5(dst): + failed.append("%s (copied, but the file did not change)" % tool) + continue + updated.append(tool) + # the completion travels with them; stale completion is worse than none + if updated: + try: + _install_completion(lfs, verbose=False) + except Exception: + pass + # The stamps are what `packagemanager` compares against to decide whether + # the chroot is behind. Refreshing the binaries and leaving the stamps + # alone made it announce "these tools differ from the copies on the host" + # about tools it had just been given, and tell the user to sync -- on the + # path that had synced them. + if pairs and (updated or failed): + try: + _write_tool_stamps(lfs, pairs) + except OSError: + pass + if failed and not quiet: + fail("! could not refresh in the chroot: %s" % ", ".join(failed)) + if updated and not quiet: + print("# refreshed in the chroot: %s (now %s, build %s)" + % (", ".join(updated), LFS_VERSION, _build_id())) + return updated def cmd_bs_run(args): @@ -4346,6 +4706,31 @@ def cmd_bs_run(args): if inside_chroot(): _refuse_inside_chroot("build-system", "run") + # A FRESH run (mounted tree, nothing laid out yet) states the book and the + # whole saved config ONCE before touching anything. Reusing last month's + # settings -- or last month's default book -- should be a decision made + # here, not a surprise discovered nine steps in. (SysV-book session) + cfg = load_config() + ver = build_version() + lfs_dir = _lfs_dir() + mounted = bool(lfs_dir) and (not cfg.get("lfs_device") + or _is_mounted(lfs_dir)) + if (ver and mounted and sys.stdin.isatty() + and not os.path.isdir(os.path.join(lfs_dir, "tools"))): + print(f"\nFresh build -- nothing is laid out under {lfs_dir} yet.") + print(f"\n book = {ver}") + for k, _desc, _d in _SESSION_KEYS: + print(f" {k:<16} = {cfg.get(k, '') or '(unset)'}") + ans = input("\nBuild with this? [Y/n/e(dit)]: ").strip().lower() + if ans in ("n", "no"): + print("stopped, nothing touched.") + print(" another book: lfs set-default <version> " + "(lfs books shows them)") + print(" other settings: lfs build-system session --reconfigure") + sys.exit(0) + if ans in ("e", "edit"): + _prompt_session_config(cfg, force=True) + def enter_chroot(): lfs = require_mounted_lfs() # Refresh the in-chroot tools FIRST. They are copies: fixing a bug in @@ -4382,9 +4767,9 @@ def cmd_bs_run(args): enter_chroot() if title == last_title: - sys.stderr.write("\n! '%s' ran but is still pending -- stopping " + fail("\n! '%s' ran but is still pending -- stopping " "rather than looping.\n Run it by hand to see " - "why:\n %s\n" + "why:\n %s" % (title, nxt["cmds"][0] if nxt["cmds"] else "?")) sys.exit(1) last_title = title @@ -4403,6 +4788,12 @@ def cmd_bs_run(args): sys.exit(1) last_title = None continue + if title == "fetch the BLFS book (for the built system)": + _run_banner(nxt, steps) + if not _run_choose_blfs_book(): + sys.exit(1) + last_title = None + continue if title == "complete the configuration": _run_banner(nxt, steps) argv = [sys.executable, os.path.abspath(__file__), @@ -4428,12 +4819,11 @@ def cmd_bs_run(args): argv = [sys.executable, os.path.abspath(__file__)] + shlex.split(cmd)[1:] rc = subprocess.call(argv) if rc != 0: - sys.stderr.write("\n! step failed (exit %d): %s\n" + fail("\n! step failed (exit %d): %s" " Fix it, then just run `lfs build-system run` " "again -- it continues from here.\n" % (rc, title)) sys.exit(rc) - sys.stderr.write("! 30 steps and still not at the chroot -- something is " - "cycling. Run `lfs build-system next --all`.\n") + fail('! 30 steps and still not at the chroot -- something is cycling. Run `lfs build-system next --all`.') sys.exit(1) @@ -4572,7 +4962,7 @@ def _restart_source_debris(srcdir): return markers, foreign for n in names: p = os.path.join(srcdir, n) - if n.startswith(".cc-"): + if n.startswith((".cc-", ".pkgusr-")): markers.append(p) continue try: @@ -4600,8 +4990,8 @@ def cmd_bs_restart(args): mounted = [d for d in ("dev/pts", "dev/shm", "dev", "proc", "sys", "run") if _is_mounted(os.path.join(lfs, d))] if mounted: - sys.stderr.write( - "\n! the chroot is still mounted: %s\n" + fail( + "\n! the chroot is still mounted: %s" " Deleting through those would reach the running system.\n" " Unmount first:\n %slfs build-system chroot unmount --run\n" % (", ".join(mounted), sudo_prefix())) @@ -4797,7 +5187,10 @@ def _tree_state(lfs): pass native = _glob.glob(os.path.join(lfs, "usr", "lib", "gcc", "*-pc-linux-gnu")) - if native: + if native and not _build_is_finished(lfs): + # Only worth saying mid-build. On a finished tree the native gcc IS + # the answer -- printing it as a WARNING there says a completed system + # is wrong. out.append(("WARNING", "a native gcc is present (%s) -- chapter 8 has run" % os.path.basename(native[0]))) @@ -4868,7 +5261,11 @@ def cmd_bs_sync_tools(args): updated in place, so a chroot built earlier keeps running an old copy until this is re-run. Copies: * /etc/pkgusr/skel-package -- the package-user home skeleton + * /etc/pkgusr/skel-u_xdg -- the shared-user XDG profile (host's + verbatim, else the placeholder copy + beside the tools; never clobbered) * /etc/pkgusr/{bash_profile,bashrc,build} + * the bash completion (lfs, with blfs/packagemanager/lfs-helper links) * the hint's helper scripts (add_package_user and friends) * lfs-helper itself * a collector-group export, if one is configured @@ -4879,6 +5276,10 @@ def cmd_bs_sync_tools(args): _sync_report("package-user skeleton", ["/etc/pkgusr/skel-package", "/etc/pkgusr/{bash_profile,bashrc,build}"]) + _sync_report("shared-user XDG profile", + ["/etc/pkgusr/skel-u_xdg/.bash_profile"]) + _sync_report("bash completion", + ["lfs-completion.bash -> completions/{lfs,blfs,...}"]) _sync_report("helper scripts", ["add_package_user, install_package, list_package, ...", "the chown/chgrp/chmod/mkdir/install wrappers"]) @@ -4896,6 +5297,10 @@ def cmd_bs_sync_tools(args): _sync_report("package-user skeleton", ["/etc/pkgusr/skel-package" if skel_ok else "none on this machine -- a default will be written"]) + _sync_report("shared-user XDG profile", + [_install_xdg_skel(lfs, verbose=False)]) + _sync_report("bash completion", + [_install_completion(lfs, verbose=False)]) copied = _install_pkgusr_helpers(lfs, verbose=False) if copied: @@ -4905,6 +5310,13 @@ def cmd_bs_sync_tools(args): ["none on this machine"], note="the tools use Shadow's useradd/groupadd instead") + # ONE SYNC PATH. `run` and `chroot enter` refresh the chroot through + # _refresh_chroot_tools; sync-tools installed the helper and the skels and + # left the other three tools and the stack files to whatever was there -- + # so which command you used decided what got updated. It calls the same + # function now, and every entry point keeps the chroot current. + _refresh_chroot_tools(lfs, quiet=True) + install_helper(lfs, quiet=True) _sync_report("the build driver", ["lfs-helper -> %s/usr/bin/lfs-helper" % lfs], @@ -5273,9 +5685,7 @@ def cmd_bs_gen_chroot_scripts(args): print(f"# wrote {written}, kept {skipped} existing " f"(use --overwrite to replace)") if unresolved: - sys.stderr.write( - "\n! these scripts still contain a placeholder the book expects " - "you to fill in;\n edit the script before building them:\n") + fail('\n! these scripts still contain a placeholder the book expects you to fill in;\n edit the script before building them:') for nm, left in unresolved: sys.stderr.write(" %-18s %s\n" % (nm, ", ".join("<%s>" % x for x in left))) @@ -5349,8 +5759,8 @@ def _copy_resolv_conf(lfs): print("# copied the host's /etc/resolv.conf into the chroot (for DNS " "during the build)") except OSError as e: - sys.stderr.write("! could not copy /etc/resolv.conf into the chroot: " - "%s\n" % e) + fail("! could not copy /etc/resolv.conf into the chroot: " + "%s" % e) def cmd_bs_chroot(args): @@ -5403,8 +5813,8 @@ def cmd_bs_chroot(args): mism = _crosschain_tgt_mismatch(lfs) if mism and not getattr(args, "force", False): was, now = mism - sys.stderr.write( - "\n! this tree was built for a different target:\n" + fail( + "\n! this tree was built for a different target:" " built with : %s\n" " configured : %s\n" " The C++ headers are installed under the first name and the\n" @@ -5419,8 +5829,8 @@ def cmd_bs_chroot(args): cc_ok, cc_n = _crosschain_looks_done(lfs) if not cc_ok and not getattr(args, "force", False): - sys.stderr.write( - "\n! chapters 5-6 are not finished: %d of %d steps built.\n" + fail( + "\n! chapters 5-6 are not finished: %d of %d steps built." " Preparing the chroot gives the tree to root, after which\n" " they cannot be resumed. Finish them first:\n" " %slfs build-system crosschain --run\n" @@ -5623,9 +6033,15 @@ def cmd_bs_chroot(args): print("# lfs-helper status # where the session stands") print("# lfs-helper list # all steps + progress") print("#") + # SAY that this is the chroot, rather than guessing later. + # + # inside_chroot() used to answer "does /usr/src/lfs-pkgusr exist" -- + # which is true on ANY system these tools built, so a finished host + # was told it was inside the chroot and refused every build command. + # The shell that IS the chroot knows it; it says so. inner = (f'chroot "{lfs}" /usr/bin/env -i ' f'HOME=/root TERM="$TERM" PS1="(lfs chroot) \\u:\\w\\$ " ' - f'PATH=/usr/bin:/usr/sbin ' + f'PATH=/usr/bin:/usr/sbin LFS_IN_CHROOT=1 ' f'MAKEFLAGS="{mkf}" TESTSUITEFLAGS="{mkf}" ' f'/bin/bash --login') import subprocess @@ -5798,7 +6214,9 @@ def cmd_bs_crosschain(args): sys.exit(preclean_rc) print(f"# [lfs] running {path} (phase: {phase}) ...") - full = f'export LFS_CC_PHASE="{phase}"\n' + script + # By PATH, not on stdin: the runner re-executes the script once + # per phase, so it has to be a file the lfs user can read. + full = f'exec bash "{path}" "{phase}"\n' rc = _run_tracked(name, full, as_lfs=True, lfs=lfs) if rc != 0: bar = "!" * 58 @@ -6111,7 +6529,7 @@ def _install_pkgusr_skel(lfs, verbose=True): shutil.rmtree(dst) shutil.copytree(PKGUSR_SKEL_SRC, dst, symlinks=True) except OSError as e: - sys.stderr.write("! could not copy %s: %s\n" % (PKGUSR_SKEL_SRC, e)) + fail("! could not copy %s: %s" % (PKGUSR_SKEL_SRC, e)) return False # the shared files the skeleton's symlinks point at for name in ("bash_profile", "bashrc", "build"): @@ -6126,6 +6544,92 @@ def _install_pkgusr_skel(lfs, verbose=True): return True +# The shared-user XDG profile. skel-package above covers PACKAGE users; +# shared application users get their session environment from this ONE file, +# which `packagemanager user create --shared` links from +# /etc/pkgusr/skel-u_xdg and fills in. It used to arrive only via `make +# install`, so a system that got its tools through this sync never had it and +# every --shared creation warned about the missing file. +# +# The host's copy is preferred VERBATIM, like skel-package: a host that has +# one has usually filled the runtime directory in already, and that answer is +# right for the machine being synced to. The placeholder copy beside the +# tools is the fallback; packagemanager only substitutes where the +# placeholder is still present, so a filled-in file passes through untouched. +# An existing copy in the tree is never clobbered -- same as the Makefile. +XDG_SKEL_REL = "etc/pkgusr/skel-u_xdg/.bash_profile" + + +def _install_xdg_skel(lfs, verbose=True): + """Place the shared-user XDG profile in the tree. Returns a short + report string for the sync summary.""" + import shutil + dst = os.path.join(lfs, XDG_SKEL_REL) + if os.path.isfile(dst): + return "kept the tree's existing copy" + src = "/" + XDG_SKEL_REL + if not os.path.isfile(src): + src = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "skel-u_xdg", ".bash_profile") + if not os.path.isfile(src): + if verbose: + print("# no skel-u_xdg/.bash_profile on this machine or beside " + "the tools -- shared users will warn until one exists") + return "none found -- shared users will warn" + try: + os.makedirs(os.path.dirname(dst), exist_ok=True) + shutil.copy(src, dst) + os.chmod(dst, 0o644) + except OSError as e: + fail("! could not copy %s: %s" % (src, e)) + return "copy failed: %s" % e + if verbose: + print("# shared-user XDG profile copied from %s" % src) + return "copied from %s" % src + + +# The bash completion. Same forgotten-file story as the XDG profile above: +# the Makefile installed it, but the chroot gets its tools from THIS sync, +# so a synced system never had tab completion. Tooling, so it is +# overwritten, not kept -- stale completion is worse than none. +COMPLETION_REL = "usr/share/bash-completion/completions" + + +def _install_completion(lfs, verbose=True): + """Place the bash completion + its tool symlinks in the tree. Returns a + short report string for the sync summary.""" + import shutil + src = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "lfs-completion.bash") + if not os.path.isfile(src): + src = "/" + COMPLETION_REL + "/lfs" + if not os.path.isfile(src): + if verbose: + print("# no lfs-completion.bash beside the tools or installed " + "here -- skipping completion") + return "none found -- skipped" + dst_dir = os.path.join(lfs, COMPLETION_REL) + try: + os.makedirs(dst_dir, exist_ok=True) + shutil.copy(src, os.path.join(dst_dir, "lfs")) + os.chmod(os.path.join(dst_dir, "lfs"), 0o644) + for t in ("blfs", "packagemanager", "lfs-helper"): + link = os.path.join(dst_dir, t) + try: + if os.path.islink(link) or os.path.isfile(link): + os.remove(link) + os.symlink("lfs", link) + except OSError: + pass + except OSError as e: + fail("! could not install completion: %s" % e) + return "copy failed: %s" % e + if verbose: + print("# bash completion installed (lfs, blfs, packagemanager, " + "lfs-helper)") + return "lfs + 3 symlinks" + + def _install_pkgusr_helpers(lfs, verbose=True): """Copy the hint's helper scripts into the chroot, where present.""" import shutil @@ -6144,7 +6648,7 @@ def _install_pkgusr_helpers(lfs, verbose=True): copied.append("%s -> /%s" % (name, subdir)) claimed.append((name, subdir)) except OSError as e: - sys.stderr.write("! %s: %s\n" % (name, e)) + fail("! %s: %s" % (name, e)) # Give them an owner, the same as install_helper does for lfs/lfs-helper/ # packagemanager/blfs. # @@ -6208,16 +6712,12 @@ def cmd_bs_install_tools(args): including for BLFS packages later on.""" lfs = require_mounted_lfs() if not _chroot_has_python(lfs): - sys.stderr.write( - "! there is no Python inside the chroot yet, so the Python tooling\n" - " cannot run there. Build it first (inside the chroot):\n" - " lfs-helper build Python\n") + fail('! there is no Python inside the chroot yet, so the Python tooling\n cannot run there. Build it first (inside the chroot):\n lfs-helper build Python') sys.exit(2) missing = _chroot_missing_tool_deps(lfs) if missing and not args.force: - sys.stderr.write( - "! the Python tooling needs modules that base LFS does not ship:\n") + fail('! the Python tooling needs modules that base LFS does not ship:') for mod, pkg in missing: sys.stderr.write(" %-16s (pip install %s)\n" % (mod, pkg)) sys.stderr.write( @@ -6235,13 +6735,12 @@ def cmd_bs_install_tools(args): here = os.path.dirname(os.path.abspath(__file__)) tools = [] - for name in ("lfs", "packagemanager", "blfs", "lfs-helper", - "packagemanager_install"): + for name in ("lfs", "packagemanager", "blfs", "lfs-helper"): src = os.path.join(here, name) if os.path.isfile(src): tools.append((name, src)) if not tools: - sys.stderr.write("! no tools found next to %s\n" % here) + fail("! no tools found next to %s" % here) sys.exit(2) dest_dir = os.path.join(lfs, "usr", "bin") @@ -6292,9 +6791,19 @@ def cmd_bs_install_tools(args): print(" %-22s -> /usr/share/blfs/books" % ("BLFS book" if _n == 1 else "BLFS books")) else: - print(" %-22s (none cached here -- `blfs fetch` inside, or" - % "BLFS book:") - print(" %-22s `blfs import <file>`)" % "") + # NOT an aside. This is the whole of `packagemanager bootstrap` + # failing later, announced early and quietly enough to scroll past: + # with no book here, `blfs sources wget` gave nothing, so get-sources + # staged no wget tarball either, and the tree ends up unable to fetch + # the book because it has no download tool and unable to install the + # download tool because it has no book. + warn("! BLFS book: none cached on this host, so none was copied in.") + warn(" The tree will have no book AND no wget tarball, and") + warn(" `packagemanager bootstrap` inside cannot break that circle.") + warn(" Fix it here, while there is still a network:") + warn(" blfs fetch # or: blfs import <book.html>") + warn(" lfs build-system get-sources --run") + warn(" lfs build-system install-tools --run") for sub in ("books", "crosschain"): s_dir = os.path.join(store_dir(), sub) @@ -6305,7 +6814,57 @@ def cmd_bs_install_tools(args): shutil.copytree(s_dir, d_dir) cfgsrc = config_path() if os.path.isfile(cfgsrc): - shutil.copy(cfgsrc, os.path.join(store, "config.json")) + # The tree's OWN choices win. A system inside the chroot has its own + # `lfs set-default` / `blfs set-default`, and copying the host's + # config over it silently moved a built system back to the host's + # book -- the setting the person changed last, undone by the next + # `chroot enter`. So: merge, host values only where the tree has + # nothing to say. + cfgdst = os.path.join(store, "config.json") + merged = {} + try: + with open(cfgsrc) as f: + merged.update(json.load(f)) + except Exception: + pass + # THE BUILD SYSTEM'S OWN CONFIG, WRITTEN BEFORE IT EXISTS. + # + # The user asked for this directly: a file on the host that IS the new + # system's config, so it can be configured before there is a tree to + # configure -- and the run copies THAT in, not the host's own + # settings. Otherwise the only way to give the new system a different + # prefix or layout was to wait for the tree, then edit it there. + # + # Precedence: the host's session config is the floor, the target + # config overrides it, and a config the TREE already has wins over + # both (it is the system's own choice, made after this file was + # written). + _target = os.path.join(store_dir(), "target-config.json") + if os.path.isfile(_target): + try: + with open(_target) as f: + _tgt = json.load(f) + merged.update(_tgt) + print(" %-22s from %s" % ("target config", _target)) + for _k in sorted(_tgt): + print(" %-20s = %r" % (_k, _tgt[_k])) + except (OSError, ValueError) as e: + print(" could not read %s: %s" % (_target, e)) + try: + with open(cfgdst) as f: + tree_cfg = json.load(f) + merged.update(tree_cfg) # the tree overrides both + except Exception: + tree_cfg = {} + try: + with open(cfgdst, "w") as f: + json.dump(merged, f, indent=2) + if tree_cfg.get("default") and \ + tree_cfg.get("default") != load_config().get("default"): + print(" %-22s kept: %s (the tree's own choice)" + % ("book default", tree_cfg["default"])) + except OSError: + shutil.copy(cfgsrc, cfgdst) # packagemanager's own config, matching the settings used out here so the # collector groups it creates line up with the ones already on the system @@ -6321,10 +6880,25 @@ def cmd_bs_install_tools(args): f.write("# written by lfs build-system install-tools\n") f.write("collector_prefix=%s\n" % collector_prefix()) f.write("user_prefix=u\n") + # The rest of what the interview already asked for. Without these + # packagemanager's first run opened a wizard asking for the + # package-user prefix and the login account -- on a system built + # from the very answers it was asking for. + # WITHOUT the trailing underscore. lfs's accessors return `p_` + # ready to concatenate; packagemanager stores the bare word and + # adds the underscore itself. Writing `p_` here survives only + # because it rstrips on read -- an accident, not an agreement. + f.write("pkgusr_prefix=%s\n" % pkgusr_prefix().rstrip("_")) + f.write("cfguser_prefix=%s\n" % cfguser_prefix().rstrip("_")) + _mu = load_config().get("main_user") + if _mu: + f.write("main_user=%s\n" % _mu) # the layout of the system being BUILT, not this machine's f.write("pkgusr_home=%s\n" % layout("pkgusr_home", target=True)) f.write("appuser_home=%s\n" % layout("appuser_home", target=True)) _install_pkgusr_skel(lfs) + _install_xdg_skel(lfs) + _install_completion(lfs) _install_pkgusr_helpers(lfs) _write_tool_stamps(lfs, tools) _run_bash('chmod -R a+rX "%s" "%s" 2>/dev/null; true' % (store, cfg_dir)) @@ -6357,17 +6931,42 @@ def _blfs_knows(name): BLFS is the better source when both books carry a package: its scripts carry the dependency information and the configure options that matter for a package used as a library, where LFS only builds what the base system - needs.""" + needs. + + Asks `blfs versions` -- a fact, listing every anchor the book has. It + used to ask `blfs debug <name>`, which is a DIAGNOSTIC: it prints + "resolve: NOT FOUND" and exits 0, so this returned True for every string + ever passed to it. `lfs update` then told the user to run + `blfs update gcc-pass1 cfg_fstab init-dirs ...` -- build steps of the LFS + book, offered as BLFS packages, because the check could not say no.""" blfs = _find_blfs() if not blfs: return False - import subprocess - try: - r = subprocess.run(blfs + ["debug", name], capture_output=True, - text=True, timeout=60, cwd="/") - return r.returncode == 0 and bool(r.stdout.strip()) - except (OSError, subprocess.SubprocessError): - return False + return name.lower() in _blfs_anchor_set() + + +_BLFS_ANCHORS = None + + +def _blfs_anchor_set(): + """Every anchor in the BLFS book, once per process.""" + global _BLFS_ANCHORS + if _BLFS_ANCHORS is not None: + return _BLFS_ANCHORS + _BLFS_ANCHORS = set() + blfs = _find_blfs() + if blfs: + import subprocess + try: + r = subprocess.run(blfs + ["versions"], capture_output=True, + text=True, timeout=120, cwd="/") + if r.returncode == 0: + for line in r.stdout.splitlines(): + if "\t" in line: + _BLFS_ANCHORS.add(line.split("\t", 1)[0].strip().lower()) + except (OSError, subprocess.SubprocessError): + pass + return _BLFS_ANCHORS def _lfs_script_path(lfs, name): @@ -6392,6 +6991,131 @@ def _find_lfs_package(args, name): return None, None, None +def _have_source_for(lfs, pkgver): + """Is a tarball for this package version already in the tree's /sources? + + Deliberately loose, and deliberately the same shape the generated script + globs for: the book calls it 'M4-1.4.21', the tarball is 'm4-1.4.21.tar.xz'. + A stricter test here than in the script would fetch a file the build then + refuses to see.""" + srcdir = os.path.join(lfs, "sources") + if not os.path.isdir(srcdir): + return False + want = pkgver.lower().replace("_", "-") + try: + for f in os.listdir(srcdir): + fl = f.lower().replace("_", "-") + if fl.startswith(want + ".") or fl.startswith(want + "-") \ + or fl.startswith(want): + if re.search(r"\.(tar\.[a-z0-9]+|tgz|tbz2|zip)$", fl): + return True + except OSError: + return False + return False + + +def _wget_list_url_for(ver, pkgver): + """The wget-list URL whose filename matches this package version. + + The book ships a wget-list with every source URL in it, and the tools + already cache it per book -- but only `get-sources` ever read it, in bulk. + So moving to a newer book left `lfs install --reinstall m4` failing with + "no M4-1.4.21.tar.* in /sources", when the URL for exactly that tarball + was sitting in a file on disk. Matched on the version token, so + 'M4-1.4.21' finds 'm4-1.4.21.tar.xz' whatever the site calls its + directories.""" + urls = _wget_list_urls(ver) or [] + if not pkgver: + return None + want = pkgver.lower() + # 'M4-1.4.21' -> 'm4' + '1.4.21'; a tarball is <name>-<version>.<ext> + m = re.match(r"^(.*?)-([0-9][0-9a-z.+_-]*)$", want) + name, version = (m.group(1), m.group(2)) if m else (want, "") + name = name.replace("_", "-") + for u in urls: + base = os.path.basename(u).lower() + if version and version not in base: + continue + stem = re.split(r"-[0-9]", base, maxsplit=1)[0].replace("_", "-") + if stem == name or base.startswith(name + "-") or base.startswith(name): + return u + return None + + +def _fetch_one_source(args, lfs, pkgver, quiet=False): + """Download one package's tarball into the tree's /sources. True if it is + there afterwards (including when it already was). + + `lfs` is passed in rather than guessed: inside the chroot the tree is "/" + and /sources is /sources, outside it is $LFS/sources, and a fetch that + landed in the wrong one would look like it worked and change nothing.""" + ver, _path = resolve_book(args) + srcdir = os.path.join(lfs, "sources") + url = _wget_list_url_for(ver, pkgver) + if not url: + return False + dest = os.path.join(srcdir, os.path.basename(url)) + if os.path.isfile(dest): + return True + if not quiet: + print(" fetching %s" % os.path.basename(url)) + print(" from %s" % url) + try: + os.makedirs(srcdir, exist_ok=True) + _download_to(url, dest) + except Exception as e: + fail(" ! download failed: %s" % e) + return False + return os.path.isfile(dest) + + +def _ensure_lfs_script(args, lfs, name, sid, title, sec, regenerate): + """Generate (or keep) the LFS-book install script for one package. + Returns the script path, or None when the book has nothing to write. + The ONE generator: cmd_install builds through it and `lfs script fetch` + stops right after it, so the two can never drift apart.""" + script = _lfs_script_path(lfs, name) + print("%s (%s)" % (name, title.strip())) + print(" script: %s" % script) + + if regenerate or not os.path.isfile(script): + pkgver = _title_pkgver(title) or name + glob = _pkg_glob_for(pkgver, name) + text = _chrootify(_crosschain_script_body(name, sid, pkgver, glob, + sec["commands"])) + text, left = _fill_placeholders(text, name) + os.makedirs(os.path.dirname(script), exist_ok=True) + with open(script, "w") as f: + f.write(text) + os.chmod(script, 0o755) + print(" regenerated from the book") + if left: + fail(" ! still contains: %s -- edit the script " + "before building" + % ", ".join("<%s>" % x for x in left)) + else: + # The existing script is KEPT -- it is yours, and regenerating would + # discard your edits. But the line above prints the BOOK's title, so + # a script written for an older book sat under a heading naming the + # new version, with nothing to say they differ. Say it. + book_ver = _title_pkgver(title) + have = None + try: + m = re.search(r'^name_version="([^"]+)"', + open(script, errors="replace").read(), re.M) + have = m.group(1) if m else None + except OSError: + pass + if have and book_ver and have != book_ver: + print(" kept your existing script, which builds %s" % have) + print(" (the book now has %s -- rebuild the script from it " + "with --regenerate," % book_ver) + print(" which discards any edits you made)") + else: + print(" kept your existing script") + return script + + def cmd_install(args): """Install or reinstall an LFS package, as its package user. @@ -6423,34 +7147,32 @@ def cmd_install(args): sid, title, sec = _find_lfs_package(args, name) if not sid: - sys.stderr.write("! '%s' is not an LFS package.\n" + fail("! '%s' is not an LFS package." " List them with: lfs packages\n" " BLFS packages are blfs' job: blfs script %s\n" % (name, name)) continue if not sec["commands"]: - sys.stderr.write("! %s has no build commands in the book.\n" % name) + fail("! %s has no build commands in the book." % name) continue - script = _lfs_script_path(lfs, name) - print("%s (%s)" % (name, title.strip())) - print(" script: %s" % script) - - if args.regenerate or not os.path.isfile(script): - pkgver = _title_pkgver(title) or name - glob = _pkg_glob_for(pkgver, name) - text = _chrootify(_crosschain_script_body(name, sid, pkgver, glob, - sec["commands"])) - text, left = _fill_placeholders(text, name) - os.makedirs(os.path.dirname(script), exist_ok=True) - with open(script, "w") as f: - f.write(text) - os.chmod(script, 0o755) - print(" regenerated from the book") - if left: - sys.stderr.write(" ! still contains: %s -- edit the script " - "before building\n" - % ", ".join("<%s>" % x for x in left)) + script = _ensure_lfs_script(args, lfs, name, sid, title, sec, + args.regenerate) + + # The tarball has to be there before the build starts. Moving to a + # newer book means the new tarballs are NOT in /sources, and the build + # failed with "no M4-1.4.21.tar.* in /sources -- run: get-sources" + # while the URL for exactly that file sat in the book's cached + # wget-list. Fetch the one that is needed, and say so. + _pv = _title_pkgver(title) + if _pv and not _have_source_for(lfs, _pv): + if not _fetch_one_source(args, lfs, _pv): + fail(" ! no tarball for %s in %s/sources, and none found in" + " the book's wget-list." % (_pv, lfs)) + fail(" Fetch everything the book needs with:") + fail(" %slfs build-system get-sources --run" + % sudo_prefix()) + sys.exit(1) # Reinstalling re-runs the CONFIGURE phase too, and that is not always # harmless: those are the post-install steps that move files about, @@ -6459,7 +7181,9 @@ def cmd_install(args): if reinstall: has_cfg = False if os.path.isfile(script): - has_cfg = "#### CONFIGURE ####" in open(script).read() + # a phase is a function (format v18+); older scripts carried a marker + has_cfg = bool(re.search(r"^configure_pkg\(\)|^#### CONFIGURE ####", + open(script).read(), re.M)) if has_cfg: print() print(" %s has a configure phase -- the post-install steps" % name) @@ -6493,6 +7217,57 @@ def cmd_install(args): print(" %s" % " ".join(cmd)) +def cmd_search(args): + """Find a package or section in the LFS book by name fragment.""" + label, path = resolve_book(args) + soup = load_soup(path) + needle = args.name.lower().replace(" ", "-") + hits = [] + for sid, title, sec in iter_sections(soup): + hay = (sid + " " + title).lower().replace(" ", "-") + if needle not in hay: + continue + nv = package_name_version(sid, title) + kind = "package" if (nv and sec["commands"]) else "section" + hits.append((kind, sid, title.strip())) + if not hits: + print(f"No match for '{args.name}' in the {label} book.") + print(f" BLFS is searched separately: blfs search {args.name}") + sys.exit(1) + for kind, sid, title in hits: + print(f" [{kind:>7}] {sid:<34} {title}") + print(f"\n{len(hits)} result(s) [{label}].") + + +def cmd_script_fetch(args): + """Get a package's install script from the book -- run nothing. + + Stops exactly where cmd_install would start building: the script lands at + the same path, generated by the same code, so "fetch, edit, install" edits + the file the install will run.""" + inside = os.path.isdir(os.path.join("/", PKGUSR_DIR)) + lfs = "/" if inside else require_mounted_lfs() + name = args.package + if not args.lfs_book and _blfs_knows(name): + print("%s is in the BLFS book -- fetch that script (more complete):" + % name) + print(" packagemanager script fetch %s" % name) + print(" (force the LFS book's with: lfs script fetch --lfs-book %s)" + % name) + return + sid, title, sec = _find_lfs_package(args, name) + if not sid: + fail("! '%s' is not an LFS package. List them with: lfs packages" + % name) + sys.exit(1) + if not sec["commands"]: + fail("! %s has no build commands in the book." % name) + sys.exit(1) + script = _ensure_lfs_script(args, lfs, name, sid, title, sec, + args.regenerate) + print(" edit it, then: lfs install --reinstall %s" % name) + + def cmd_update(args): """Rebuild packages whose installed version differs from the book's.""" inside = os.path.isdir(os.path.join("/", PKGUSR_DIR)) @@ -6506,13 +7281,46 @@ def cmd_update(args): installed = {f[:-6] for f in os.listdir(manifests) if f.endswith(".files")} names = args.packages or sorted(installed) + + # BOOK ORDER, not alphabetical. The book builds glibc before Python for + # a reason -- each package is compiled against the ones before it -- and + # an alphabetical list puts Python, binutils and coreutils ahead of + # glibc. Rebuilt in that order, the earlier ones link against the old + # glibc and are then stale again the moment it is replaced. Chapter 8's + # own sequence is the safe one, so the report is sorted by where each + # package appears in the book, and anything the book does not name goes + # last rather than jumping the queue. + _order = {} + for _i, (_sid, _title, _sec) in enumerate(iter_sections(soup)): + _nv = package_name_version(_sid, _title) + if _nv: + _order.setdefault(_nv[0].lower(), _i) + names = sorted(names, key=lambda n: (_order.get(n.lower(), 10 ** 6), + n.lower())) if not names: print("nothing is installed yet -- build the system first:") print(" %slfs build-system next" % sudo_prefix()) return stale, current, unknown = [], [], [] + # The chapter 5-7 STAGE VARIANTS are not upgradable packages. + # gettext-tmp, python-tmp and util-linux-tmp are the TEMPORARY builds that + # existed to get chapter 8 built; binutils-pass1 and gcc-pass2 are cross + # stages. Comparing them with the current book reported "util-linux-tmp + # 2.41.1 -> 2.41.3" and put them in the reinstall line, which would + # rebuild the temporary toolchain onto a finished system. + # + # Only the VARIANT names are excluded, never the plain one. Chapter 6 + # builds glibc and coreutils under their own names, so taking every name + # in the step tables would have silently hidden two of the packages most + # worth knowing about -- the first draft of this did exactly that. + _stage_steps = {n.lower() for n, _sid in CHROOT_STEPS_7 + if n.lower().endswith("-tmp")} + _stage_steps |= {n.lower() for n, _sid, _pkg in CROSSCHAIN_STEPS + if re.search(r"-(pass\d+|tmp)$", n.lower())} for name in names: + if name.lower() in _stage_steps: + continue sid, title, sec = _find_lfs_package(args, name) if not sid: unknown.append(name) @@ -6541,11 +7349,41 @@ def cmd_update(args): else: current.append((name, have or book_ver)) + if getattr(args, "porcelain", False): + # One fact per line, tab-separated, and NOTHING else -- no headings, + # no advice, no colour. packagemanager builds its plan from this; + # parsing the human report would break the first time a word changed. + # + # --include-current adds the packages whose version did NOT move. + # After glibc changes, "up to date" and "built against the current + # system" stop being the same statement: the version is right and the + # binary is still the old one. Rebuilding those, in book order, is + # the only way to know the whole system was built against the new + # toolchain. Emitted in the same shape, with installed == book. + rows = list(stale) + if getattr(args, "include_current", False): + _stale_names = {n for n, _h, _b in stale} + for name, v in current: + if name not in _stale_names: + rows.append((name, v or "?", v or "?")) + rows.sort(key=lambda r: (_order.get(r[0].lower(), 10 ** 6), + r[0].lower())) + for name, have, book_ver in rows: + print("%s\t%s\t%s" % (name, have, book_ver)) + return + if stale: print("Out of date -- comparing what is installed with the %s book:\n" % ver) + _c = _C_OK if _tty_out() else "" + _o = _C_OFF if _tty_out() else "" for name, have, book_ver in stale: - print(" %-20s %s -> %s" % (name, have, book_ver)) + # the version you would move TO, in colour -- the same emphasis + # packagemanager's plan gives it, so the two reports read alike. + # Guarded by _tty_out() like every other colour here: escape codes + # in a piped report are noise in a file. + print(" %-20s %s -> %s%s%s" + % (name, have, _c, book_ver, _o)) print() print("Rebuild them with:") print(" lfs install --reinstall %s" @@ -6678,10 +7516,36 @@ def _next_steps(): "%slfs build-system add-user --run" % sudo_prefix())) srcdir = os.path.join(lfs, "sources") - have_sources = os.path.isdir(srcdir) and any( - f.endswith((".tar.xz", ".tar.gz", ".tar.bz2", ".tgz")) - for f in os.listdir(srcdir)) if os.path.isdir(srcdir) else False - steps.append(_step(have_sources, "download the sources", + # "done" is EVERY basename of the current book's wget-list being present, + # not "some tarball exists". Leftovers from an older book made the step + # look finished, and the download was skipped for a book whose tarballs + # were not there. (from the SysV-book session) + src_missing = 0 + if os.path.isdir(srcdir): + wl_urls = _wget_list_urls(build_version()) + if wl_urls is not None: + have = set(os.listdir(srcdir)) + src_missing = sum(1 for u in wl_urls + if u.split("/")[-1] not in have) + have_sources = src_missing == 0 + else: + # no wget-list cached yet: the old heuristic is all there is + have_sources = any( + f.endswith((".tar.xz", ".tar.gz", ".tar.bz2", ".tgz")) + for f in os.listdir(srcdir)) + else: + have_sources = False + # BEFORE the sources, deliberately. `get-sources` asks + # `packagemanager bootstrap --sources`, which asks the BLFS book where + # wget's tarball is -- with no book that URL is silently absent from the + # download list, and the gap only shows up on the built system. + steps.append(_step(_blfs_book_on_host() or cfg.get("blfs_book") == "skip", + "fetch the BLFS book (for the built system)", + "blfs fetch", + note="the built system cannot fetch it itself until it " + "has wget, and wget is looked up in this book")) + steps.append(_step(have_sources, "download the sources" + + (f" ({src_missing} missing)" if src_missing else ""), "%slfs build-system get-sources --run" % sudo_prefix())) cc_done, cc_n = _crosschain_looks_done(lfs) @@ -6828,12 +7692,53 @@ _INSIDE_EQUIVALENT = { } +def _root_differs_from_pid1(): + """Is this process's / a different directory from PID 1's? + + THE KERNEL KNOWS. That is what a chroot IS -- a root that is not the + real one -- and /proc/1/root resolves to the real one, so comparing the + two answers the question outright instead of inferring it from files + that happen to be there. Returns None when it cannot be asked: no + /proc mounted, or not privileged enough to read PID 1's root. + """ + if os.geteuid() != 0: + return None # /proc/1/root is root's to read + try: + a, b = os.stat("/"), os.stat("/proc/1/root") + except OSError: + return None # no /proc, or a kernel that will not say + return (a.st_dev, a.st_ino) != (b.st_dev, b.st_ino) + + def inside_chroot(): """Are we running inside the LFS chroot? - The state directory lives at the root of the built system, so it is only at - /usr/src/lfs-pkgusr when we are in there.""" - return os.path.isdir(os.path.join("/", PKGUSR_DIR)) + The chroot SAYS SO: `lfs build-system chroot enter` sets LFS_IN_CHROOT in + the environment it starts. Failing that, the kernel is asked + (_root_differs_from_pid1). + + It used to be inferred from "/usr/src/lfs-pkgusr exists", and that is true + of any system these tools have built -- so a finished host running its own + lfs refused every command with "you are inside the LFS chroot". An + inference that is true of the destination cannot identify the journey. + + The guard on that inference was "...and /mnt/lfs/usr/src does not exist", + which fails at the one moment it matters: a FRESH build, where the + partition is mounted and still empty, on a host these tools built. That + is the report this fix comes from. The old test survives only as a last + resort -- no LFS_IN_CHROOT, no readable /proc/1 -- and then against the + CONFIGURED mount point, not a hardcoded one. + """ + if os.environ.get("LFS_IN_CHROOT"): + return True + differs = _root_differs_from_pid1() + if differs is not None: + return differs + if os.environ.get("LFS"): + return False # there is a build target: we are outside + mount = load_config().get("lfs_mount") or "/mnt/lfs" + return (os.path.isdir(os.path.join("/", PKGUSR_DIR)) + and not os.path.isdir(mount)) def _refuse_inside_chroot(cmd, sub=None): @@ -6855,6 +7760,19 @@ def _refuse_inside_chroot(cmd, sub=None): " lfs-helper status # where things stand\n" " lfs-helper list # every step\n") sys.stderr.write("\n To run this, leave the chroot first (type 'exit').\n") + # WHICH lfs SAID SO, AND WHY. + # + # A host was refused by this message for days. 1.14.1 fixed the + # detection, but the binary being run was 1.11.26 -- `make install` had + # been going somewhere else, and nothing on screen said which copy was + # talking or what convinced it. A refusal has to identify itself, or a + # stale install looks like a bug in the current one. + how = "LFS_IN_CHROOT is set" if os.environ.get("LFS_IN_CHROOT") \ + else ("this process's / is not PID 1's" + if _root_differs_from_pid1() else "no $LFS, and no mounted tree") + sys.stderr.write("\n (said by %s %s, build %s -- because %s)\n" + % (os.path.abspath(sys.argv[0]), LFS_VERSION, + _build_id(), how)) sys.exit(2) @@ -6903,6 +7821,13 @@ def main(): "LFS_TGT, mirror, ...)") # values can start with '-' (e.g. makeflags -j4), so don't let argparse # try to parse them as flags + # --target: the config the BUILT system will get, editable here, before + # the tree exists. `lfs run` copies it in (see target-config.json). + p.add_argument("--target", action="store_true", + help="show or change the config the BUILT system will get " + "(not this host's)") + p.add_argument("--ask", action="store_true", + help="walk the settings one at a time (with --target)") p.add_argument("key", nargs="?", help="setting to show, or set with a value") p.add_argument("value", nargs="?", help="new value") p.usage = ("lfs config [key] [value] | lfs config --edit | " @@ -6966,12 +7891,34 @@ def main(): help="packages to check (default: everything installed)") p.add_argument("--verbose", "-v", action="store_true", help="list up-to-date packages too") + p.add_argument("--porcelain", action="store_true", + help="machine-readable: name<TAB>installed<TAB>book, one " + "per line, nothing else (for packagemanager)") + p.add_argument("--include-current", action="store_true", + help="with --porcelain: list up-to-date packages too, so a " + "caller can rebuild the whole system in book order") p.set_defaults(func=cmd_update) p = sub.add_parser("packages", help="list chapter-8 build packages " "(name<TAB>version<TAB>id) of the main book") p.set_defaults(func=cmd_packages) + p = sub.add_parser("search", help="find a package or section in the " + "LFS book by name") + p.add_argument("name", help="name fragment to look for") + p.set_defaults(func=cmd_search) + + p = sub.add_parser("script", help="fetch a package's install script from " + "the book for editing (runs nothing)") + p.add_argument("action", choices=["fetch"], + help="'fetch': generate/keep the script and print its path") + p.add_argument("package") + p.add_argument("--regenerate", action="store_true", + help="rebuild from the book even if the script exists") + p.add_argument("--lfs-book", action="store_true", + help="use the LFS book even when BLFS also has the package") + p.set_defaults(func=cmd_script_fetch) + p = sub.add_parser("commands", help="print a section's shell commands") p.add_argument("section", help="section id or exact title") p.add_argument("book_pos", nargs="?", metavar="BOOK", diff --git a/lfs-helper b/lfs-helper old mode 100644 new mode 100755 index 6503e33..2e69f94 --- a/lfs-helper +++ b/lfs-helper @@ -1,4 +1,26 @@ #!/bin/bash + +# --------------------------------------------------------------------------- # +# Read ourselves ONCE, before doing anything. +# --------------------------------------------------------------------------- # +# +# bash reads a script INCREMENTALLY, by byte offset, while it runs it. Replace +# the file mid-run -- `make install` of a new build in another terminal, a tool +# refresh, anything -- and execution resumes at the same offset in the NEW +# content, mid-word: +# /usr/bin/lfs-helper: line 8312: mmand: command not found +# /usr/bin/lfs-helper: line 8314: syntax error near unexpected token `)' +# That happened during a package install: the build had finished, the package +# was installed, and the tool died on its own source. Nothing about the +# package was wrong. +# +# Everything below is one group command, so bash must parse the WHOLE file +# before it executes any of it. After that the file on disk can change freely +# -- this run is already in memory. Functions and variables defined inside a +# group are global, `exit` still exits, and the group's status is the last +# command's, so nothing else changes. +{ + # # lfs-helper -- manage the LFS build from INSIDE the chroot. # @@ -19,10 +41,15 @@ set -u +# lfs-helper runs before init-dirs has created /tmp, and `sort` needs a +# temporary directory: "sort: cannot create temporary file in '/tmp'". +# (from the SysV-book session) +[ -d /tmp ] || mkdir -m 1777 -p /tmp 2>/dev/null + # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.11.6" +LFS_HELPER_VERSION="1.14.146" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -177,7 +204,11 @@ create_collector_group() { # command's own output. C_OK=$'\033[0;32m'; C_WARN=$'\033[0;33m'; C_ERR=$'\033[0;31m' C_DIM=$'\033[2m'; C_OFF=$'\033[0m' -[ -t 1 ] || { C_OK=; C_WARN=; C_ERR=; C_DIM=; C_OFF=; } +# Bold, for the words in a question that the ANSWER depends on: which package +# is asking, which directory, and the name being proposed. A prompt where +# every word looks the same makes the reader hunt for the three that matter. +C_B=$'\033[1m' +[ -t 1 ] || { C_OK=; C_WARN=; C_ERR=; C_DIM=; C_OFF=; C_B=; } # prose -- the ordinary running commentary of a build say() { echo "$*"; } @@ -260,6 +291,13 @@ unmark() { [ -f "$PROGRESS" ] && sed -i "\|^$1\$|d" "$PROGRESS"; } # The ordered list of steps. Each is "name" and the script is # $SCRIPTS/<name>.sh -- generated by `lfs build-system gen-chroot-scripts`. +# Is this name one of the LFS book's steps? A name that is NOT tells the +# person something different from one that is: a missing step script is a +# generation problem, an unknown name is probably a BLFS package or a typo. +in_step_order() { + steps | grep -Fxq "$1" +} + steps() { if [ -f "$STATE_PROGRESS/steporder" ]; then grep -v '^[[:space:]]*$' "$STATE_PROGRESS/steporder" @@ -619,6 +657,16 @@ is_install_dir() { return 1 } +# "Answer for me" has ONE meaning, whoever asked. +# +# The group-choice prompt honoured LFS_ASSUME_YES; `packagemanager --yes` +# sets PM_YES; pm-install reads PM_YES as well. Three spellings of one +# decision, so `stack sway --run --yes` still stopped to ask which collector +# group to use -- the exact thing --yes exists to prevent. +assume_yes() { + [ "${LFS_ASSUME_YES:-0}" = 1 ] || [ "${PM_YES:-0}" = 1 ] +} + pkgusr_ready() { group_exists install; } cmd_init_pkgusr() { @@ -655,6 +703,12 @@ cmd_init_pkgusr() { say " (Shadow isn't built yet -- no groupadd, so this was done by hand)" fi + # Shared index FILES, same rule as the directories. + share_shared_files + + # the shell startup helpers every BLFS profile.d snippet calls + install_profile_helpers + # Directories packages install into. Anything missing is skipped quietly. local d n=0 failed="" didnt_take="" while IFS= read -r d; do @@ -1009,6 +1063,7 @@ snapshot_dirs() { -not -path "$r/sys/*" -not -path "$r/run/*" \ -not -path "$r/tmp/*" -not -path "$r/sources/*" \ -not -path "$r/build/*" \ + -not -path "$r/root/*" -not -path "$r/home/*" \ 2>/dev/null | LC_ALL=C sort } @@ -1029,6 +1084,7 @@ snapshot() { -not -path "$r/sys/*" -not -path "$r/run/*" \ -not -path "$r/tmp/*" -not -path "$r/sources/*" \ -not -path "$r/build/*" \ + -not -path "$r/root/*" -not -path "$r/home/*" \ 2>/dev/null | LC_ALL=C sort } @@ -1095,15 +1151,31 @@ PHASES="$STATE_PROGRESS/phases" # job's message can be thousands of lines above the end, so showing the tail # (what you see on screen) is usually useless. show_build_errors() { - local log="$1" n=0 + local log="$1" hits="" [ -f "$log" ] || return 0 say "" - fail "--- first errors in the log ---" - grep -nE "(^|[^-])\\berror\\b:|\\*\\*\\* \\[|Error [0-9]+|No such file or directory|command not found|undefined reference" \ - "$log" 2>/dev/null | grep -vE "Error 1$|Error 2$" | head -12 | while IFS= read -r l; do - say " $l" - done - n=$(grep -cE "\\berror\\b:|\\*\\*\\* \\[" "$log" 2>/dev/null || echo 0) + hits="$(grep -nE "(^|[^-])\\berror\\b:|\\*\\*\\* \\[|Error [0-9]+|No such file or directory|command not found|undefined reference" \ + "$log" 2>/dev/null | grep -vE "Error 1$|Error 2$" | head -12)" + if [ -n "$hits" ]; then + fail "--- first errors in the log ---" + printf '%s\n' "$hits" | while IFS= read -r l; do + say " $l" + done + else + # An EMPTY section under a heading promising errors is worse than no + # section: the reader concludes there is nothing to see. m4 failed + # with "no M4-1.4.21.tar.* ... in /sources" -- a perfectly clear + # message that matched none of the patterns above, so the report said + # "--- first errors ---" and then nothing at all. + # + # No pattern matched is a statement about the PATTERNS, not about the + # log. Show the end of it, which is where a failure that does not + # look like a compiler error usually is. + fail "--- no line matched the usual error patterns; the end of the log ---" + tail -n 12 "$log" 2>/dev/null | while IFS= read -r l; do + say " $l" + done + fi fail "-------------------------------" say " full log: $log" } @@ -1114,17 +1186,25 @@ record_phase() { mkdir -p "$PHASES"; grep -qxF "$2" "$(phase_file "$1")" 2>/dev/ phase_done() { grep -qxF "$2" "$(phase_file "$1")" 2>/dev/null; } clear_phases() { rm -f "$(phase_file "$1")"; } -# which phases does this package's script actually define? -script_phases() { - local f; f="$(script_for "$1")" - [ -f "$f" ] || return 0 +# which phases does this FILE define? The runner tracks phases for scripts +# it is handed by path (a packagemanager install runs the script from the +# package's home, not from $SCRIPTS), so the answer must come from the file, +# not from the name. +script_phases_of_file() { + [ -f "$1" ] || return 0 echo unpack echo build echo install - grep -q '^#### CONFIGURE ####' "$f" && echo configure + # a phase is a function (format v18+); older scripts carried a marker + grep -qE '^configure_pkg\(\)|^#### CONFIGURE ####' "$1" && echo configure return 0 } +# which phases does this package's script actually define? +script_phases() { + script_phases_of_file "$(script_for "$1")" +} + # The phase to run next: the one AFTER the furthest phase already finished. # (Not simply the first unfinished one -- after running only `install`, the # earlier unpack/build had clearly happened too, and suggesting "unpack" would @@ -1280,7 +1360,12 @@ clean_stale_source() { # replaced by a chapter-8 package user, and the build dies with # .../.cc-build-ncurses: Permission denied # Clear them here, as root; the build recreates them. - rm -f "$blddir/.cc-build-$name" "$blddir/.cc-dir-$name" 2>/dev/null + # Markers are named for the PACKAGE (name_version) since 1.14.5, and this + # build root belongs to one package, so clear whatever is in it -- under + # the old name, the older name, or a name_version this step no longer has. + rm -f "$blddir"/.pkgusr-*.dir "$blddir"/.pkgusr-*.cwd \ + "$blddir/.pkgusr-$name.dir" "$blddir/.pkgusr-$name.cwd" \ + "$blddir/.cc-build-$name" "$blddir/.cc-dir-$name" 2>/dev/null local dir dir="$(tar tf "$srcdir/$pkg" 2>/dev/null | head -n1 | cut -d/ -f1)" @@ -1343,18 +1428,30 @@ COLLECTOR_PREFIX="${LFS_COLLECTOR_PREFIX:-$(_read_shared_prefix || true)}" # # Resolved in the same order and for the same reason as COLLECTOR_PREFIX -- # two tools disagreeing about it would build two parallel sets of accounts. +# PRESENT-AND-EMPTY IS AN ANSWER. +# +# `"pkgusr_prefix": ""` means NO PREFIX -- the comment two lines below has said +# so since prefixes were introduced, and the reader ignored it: an empty value +# failed `[ -n "$v" ]`, fell through to the default `p`, and lfs-helper went +# on calling the account p_mako while packagemanager (which honours empty) +# asked for mako. Two tools, two different names for one account. +# +# So: does the KEY EXIST? If it does, its value -- empty or not -- is the +# answer. _read_pkgusr_prefix() { - local v="" + local f v for f in /usr/share/lfs/config.json "$HOME/.local/share/lfs/config.json"; do [ -r "$f" ] || continue + grep -q '"pkgusr_prefix"[[:space:]]*:' "$f" || continue v="$(sed -n 's/.*"pkgusr_prefix"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ "$f" | head -n1)" - [ -n "$v" ] && { printf '%s' "$v"; return 0; } + printf '%s' "$v"; return 0 done for f in /etc/pkgusr/packagemanager.conf; do [ -r "$f" ] || continue + grep -q '^pkgusr_prefix=' "$f" || continue v="$(sed -n 's/^pkgusr_prefix=\(.*\)$/\1/p' "$f" | head -n1)" - [ -n "$v" ] && { printf '%s' "$v"; return 0; } + printf '%s' "$v"; return 0 done return 1 } @@ -1363,7 +1460,25 @@ _read_pkgusr_prefix() { # which is how a tree built before prefixes existed keeps working. With `:-` # an empty value would silently fall through to the default and rename every # package user out from under that tree. -PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" +# THE CONFIG IS CURRENT; THE ENVIRONMENT MAY BE OLD. +# +# Both sources can legitimately say "no prefix" -- a tree built without one +# keeps working only because empty is meaningful. But they can disagree, and +# then age decides: the chroot scripts export LFS_PKGUSR_PREFIX from whatever +# the prefix was WHEN THEY WERE WRITTEN, while the config file is what the +# system says now. The user's chroot exported an empty value from an earlier +# experiment; both config files said "p"; lfs-helper believed the export and +# announced a disagreement that did not exist. +# +# So: a config that NAMES a prefix wins. Otherwise the environment is +# honoured, empty included. Otherwise the default. +_cfg_prefix="$(_read_pkgusr_prefix || true)" +if [ -n "$_cfg_prefix" ]; then + PKGUSR_PREFIX="$_cfg_prefix" +else + PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" +fi +unset _cfg_prefix PKGUSR_PREFIX="${PKGUSR_PREFIX%_}" [ -n "$PKGUSR_PREFIX" ] && PKGUSR_PREFIX="${PKGUSR_PREFIX}_" @@ -1497,6 +1612,257 @@ cmd_owner_name() { pkg_owner_name "$1" } +# --------------------------------------------------------------------------- # +# A home that never met its files. +# +# The hint's add_package_user takes no home argument and creates +# /usr/src/<account>. Every tool here computes the home through +# pkgusr_home_for -- /usr/src/pkgusr/<account> -- and the python side's repair +# only ran `usermod -d`: it repointed the passwd entry and LEFT THE DIRECTORY +# BEHIND. A real bootstrap showed the result: +# +# /usr/src/p_wget /usr/src/p_requests /usr/src/p_make-ca ... +# +# husks beside the roots, each holding the skeleton add_package_user copied in, +# while the real homes lived under /usr/src/pkgusr/. Three tools each had +# part of the repair; this is the whole of it, in one place, and the others +# call it. +# --------------------------------------------------------------------------- # + +# The home the user DATABASE says -- $ETC/passwd first, for the same reason +# user_exists reads it: inside the chroot the file IS the database. +_recorded_home_for() { + if [ -r "$ETC/passwd" ]; then + awk -F: -v u="$1" '$1==u{print $6; exit}' "$ETC/passwd" + else + getent passwd "$1" 2>/dev/null | cut -d: -f6 + fi +} + +_set_recorded_home() { + local acct="$1" home="$2" + # CHECK THAT IT TOOK. + # + # `usermod -d` returned 0 and /etc/passwd still said the old home, so + # fix-home printed "# p_n76310: home -> ..." and changed nothing -- three + # times in a row, each run reporting success. A write that cannot be + # read back did not happen: try usermod, verify, and fall through to the + # file rewrite when it silently did nothing. + if have_shadow_tools && command -v usermod >/dev/null 2>&1; then + usermod -d "$home" "$acct" 2>/dev/null + [ "$(_recorded_home_for "$acct")" = "$home" ] && return 0 + fi + # No usermod (pre-Shadow chroot): edit the file, the way raw_useradd wrote + # it. Rewrite to a temp file and cat back, preserving the inode. + [ -w "$ETC/passwd" ] || return 1 + local tmp; tmp="$(mktemp)" || return 1 + awk -F: -v OFS=: -v u="$acct" -v h="$home" '$1==u{$6=h}1' "$ETC/passwd" > "$tmp" && cat "$tmp" > "$ETC/passwd" + rm -f "$tmp" + # ...and read it back here too: a rewrite that left the line alone (a + # name that does not match, a passwd file somewhere else) is a failure, + # not a success with an exit code of 0 from `cat`. + [ "$(_recorded_home_for "$acct")" = "$home" ] +} + +# Move src's contents into dst, entry by entry: +# * missing in dst -> moved +# * identical regular file -> the stray copy is deleted +# * anything else -> KEPT, so no file is ever destroyed +# Removes src if that empties it; returns 1 if anything had to be kept. +# What was kept lands in _MERGE_KEPT -- "compare and remove by hand" without +# saying WHICH files sends the user diffing eleven directory pairs to find +# them. The report must name the work it hands over. +# Is $1 under directory $2? An EMPTY $2 means "/" (the whole tree), never +# the wildcard that `case "$p" in "$2"/*)` silently becomes -- which is how +# cmake's missing directory stayed invisible in every real chroot while +# being found in every sandbox. See missing_dirs_from_log. +under_dir() { + local p="$1" d="${2%/}" + case "$p" in /*) ;; *) return 1 ;; esac + [ -z "$d" ] && return 0 + [ "${p#"$d"/}" != "$p" ] +} + +_MERGE_KEPT="" +# The files every package-user home gets when it is created. They carry no +# state: whoever made the account wrote them, and the real home's copies are +# the ones in use. +_is_skeleton_file() { + case "$1" in + .bashrc|.bash_profile|.project|build|build.conf|.wget-hsts|.bash_history) + return 0 ;; + esac + return 1 +} + +_merge_dir_into() { + local src="$1" dst="$2" f rel kept=0 + _MERGE_KEPT="" + [ -d "$src" ] || return 0 + [ "$src" = "$dst" ] && return 0 + while IFS= read -r -d '' f; do + rel="${f#"$src"/}" + if [ ! -e "$dst/$rel" ] && [ ! -L "$dst/$rel" ]; then + mkdir -p "$dst/$(dirname "$rel")" 2>/dev/null + mv "$f" "$dst/$rel" 2>/dev/null || { kept=$((kept+1)); _MERGE_KEPT="$_MERGE_KEPT$f"$'\n'; } + elif [ -f "$f" ] && [ ! -L "$f" ] && [ -f "$dst/$rel" ] && cmp -s "$f" "$dst/$rel"; then + rm -f "$f" + elif _is_skeleton_file "$rel"; then + # THE SKELETON IS NOT DATA. + # + # .bashrc, .bash_profile, build, build.conf and .project are + # created by whoever made the account, so a husk and a real home + # have five files that "differ" -- and the merge refused, the + # home was never repaired, and `fix-home` reported success while + # changing nothing that mattered. The destination's copies are + # the current ones; the husk's are a duplicate of the same idea. + rm -f "$f" 2>/dev/null || { kept=$((kept+1)); _MERGE_KEPT="$_MERGE_KEPT$f"$'"'"'\n'"'"'; } + else + kept=$((kept+1)); _MERGE_KEPT="$_MERGE_KEPT$f"$'\n' + fi + done < <(find "$src" -mindepth 1 \( -type f -o -type l \) -print0 2>/dev/null) + find "$src" -depth -mindepth 1 -type d -empty -delete 2>/dev/null + rmdir "$src" 2>/dev/null || true + [ "$kept" = 0 ] && [ ! -d "$src" ] +} + +# Report what a merge kept: up to five paths, then a count. +_report_merge_kept() { + local n shown=0 f + n="$(printf '%s' "$_MERGE_KEPT" | grep -c .)" + [ "$n" -gt 0 ] || return 0 + while IFS= read -r f; do + [ -n "$f" ] || continue + warn " differs: $f" + shown=$((shown+1)); [ "$shown" -ge 5 ] && break + done <<< "$_MERGE_KEPT" + [ "$n" -gt 5 ] && warn " ... and $((n-5)) more" + return 0 +} + +# The whole repair for ONE account: recorded home vs computed home, plus the +# husk add_package_user leaves beside the roots. Reports always; acts when +# fix=1. Returns 0 when nothing was wrong. +# $3=1: this account was JUST created, so add_package_user's own home is +# expected and not news. It takes no home argument, so EVERY new account is +# briefly "misplaced" -- reporting that as a warning meant a normal install +# opened with two lines of alarm about a condition the next line repairs. A +# misplaced home found any OTHER way is still a warning: then it is a real +# account that has been pointing at the wrong directory for who knows how long. +# The BLFS package that provides a command, for the handful the books use as +# build tools without always listing them. Not a database -- just the ones +# that actually bite. +_package_providing() { + case "$1" in + cmake|ccmake|cpack|ctest) echo "cmake" ;; + ninja) echo "ninja" ;; + meson) echo "meson" ;; + cargo|rustc) echo "rust" ;; + go|gofmt) echo "go" ;; + git) echo "git" ;; + nasm) echo "nasm" ;; + yasm) echo "yasm" ;; + doxygen) echo "doxygen" ;; + gperf) echo "gperf" ;; + itstool) echo "itstool" ;; + xsltproc|xmlcatalog|xmllint) echo "libxslt" ;; + pkg-config|pkgconf) echo "pkgconf" ;; + wget) echo "wget" ;; + curl) echo "curl" ;; + unzip|zipinfo) echo "unzip" ;; + java|javac) echo "openjdk" ;; + ruby) echo "ruby" ;; + valac) echo "vala" ;; + sassc) echo "sassc" ;; + asciidoc|asciidoctor) echo "asciidoc" ;; + docbook2*|db2*) echo "docbook-utils" ;; + *) echo "" ;; + esac +} + +_repair_misplaced_home() { + local acct="$1" fix="${2:-0}" fresh="${3:-0}" want got stray rc=0 husk + user_exists "$acct" || return 0 + want="$(pkgusr_home_for "$acct")" + got="$(_recorded_home_for "$acct")" + husk="$SRCROOT/$acct" # the same place the husk check below uses + + if [ -n "$got" ] && [ "$got" != "$want" ]; then + rc=1 + if [ "$fresh" = 1 ] && [ "$got" = "$husk" ]; then + detail " add_package_user put the home at $got; moving it to $want" + else + warn "misplaced home: $acct is recorded at $got" + warn " every tool looks at $want" + fi + if [ "$fix" = 1 ]; then + if [ -d "$got" ] && [ ! -e "$want" ]; then + mkdir -p "$(dirname "$want")" 2>/dev/null + mv "$got" "$want" 2>/dev/null || cp -a "$got" "$want" + elif [ -d "$got" ]; then + _merge_dir_into "$got" "$want" \ + || { warn " $got kept: these files differ from $want's; compare and remove by hand" + _report_merge_kept; } + else + mkdir -p "$want" 2>/dev/null + fi + if _set_recorded_home "$acct" "$want"; then + soft "could not chown the repaired home to $acct" -- \ + real_chown "$acct:$acct" "$want" + ok "# $acct: home -> $want" + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + else + warn " could not record $want as $acct's home" + fi + fi + fi + + # The husk: add_package_user's default location, beside the roots. + stray="$SRCROOT/$acct" + if [ "$stray" != "$want" ] && [ -d "$stray" ] && [ ! -L "$stray" ]; then + rc=1 + warn "stray home: $stray (the real home is $want)" + if [ "$fix" = 1 ]; then + mkdir -p "$want" 2>/dev/null + if _merge_dir_into "$stray" "$want"; then + soft "could not chown the repaired home to $acct" -- \ + real_chown "$acct:$acct" "$want" + ok "# merged $stray into $want" + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + else + warn " $stray kept: these files differ from $want's; compare and remove by hand" + _report_merge_kept + fi + fi + fi + return "$rc" +} + +# `lfs-helper fix-home` -- the one door the other tools call. +cmd_fix_home() { + need_root + local name="" fix=0 + while [ $# -gt 0 ]; do + case "$1" in + --run|--fix) fix=1; shift ;; + -h|--help) cmd_help_for fix-home; return 0 ;; + -*) die "unknown option: $1 (usage: lfs-helper fix-home <name> [--run])" ;; + *) name="$1"; shift ;; + esac + done + [ -n "$name" ] || die "usage: lfs-helper fix-home <name> [--run]" + local acct; acct="$(pkg_owner_name "$name")" + if _repair_misplaced_home "$acct" "$fix"; then + ok "# $acct: home is where every tool looks ($(pkgusr_home_for "$acct"))" + return 0 + fi + if [ "$fix" != 1 ]; then + say " nothing changed. Repair with: lfs-helper fix-home $name --run" + return 1 + fi + return 0 +} + # Every account root, for the passes that walk all of them. pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } @@ -1566,7 +1932,12 @@ require_collector_prefix() { # `nimgnu_p_perl` -- two prefixes stacked, and the wrong two: `nimgnu_` says # "collector group" and `p_` says "package user", which it is not. The prompt # offered it as a choice: -# 1) nimgnu_p_perl (everything owned by 'p_perl' -- fewer groups) +# 1) nimgnu_p_perl (named after the owner -- reusable, fewer groups) +# +# NOT "everything owned by p_perl", which is what this said and it was simply +# untrue: a collector group grants the directories it is applied to, one at a +# time. Choosing the owner-named group only means the NAME can be reused on +# the next directory of that owner; it grants nothing extra by itself. # Strip the account's prefix first; the collector prefix is the only one a # collector group carries. Same rule as everywhere else: one name, one prefix. collector_group_for() { echo "${COLLECTOR_PREFIX}_$(unprefix_pkg_user "$1")"; } @@ -1604,6 +1975,33 @@ add_to_group() { # Group names may only sensibly contain [a-z0-9_-]; keep them short too, since # many tools cap group names at 32 characters. +# Is $1 a near miss for $2 -- same string but for one insertion, deletion or +# substitution? Used to catch a mistyped collector prefix without second- +# guessing a name that was chosen on purpose. +_near_miss() { + local a="$1" b="$2" + [ "$a" = "$b" ] && return 1 + local la=${#a} lb=${#b} + case $((la - lb)) in + 0) local d=0 i c1 c2 + for ((i = 0; i < la; i++)); do + c1="${a:i:1}"; c2="${b:i:1}" + [ "$c1" = "$c2" ] || d=$((d + 1)) + [ "$d" -gt 1 ] && return 1 + done + [ "$d" = 1 ] && return 0 ;; + 1|-1) + local long short + if [ "$la" -gt "$lb" ]; then long="$a"; short="$b" + else long="$b"; short="$a"; fi + local i + for ((i = 0; i < ${#long}; i++)); do + [ "${long:0:i}${long:i+1}" = "$short" ] && return 0 + done ;; + esac + return 1 +} + sanitise_group_name() { # `tr -c` turns EVERY other character into a dash, including spaces -- so # an answer typed with a leading space became "-name", and one with two @@ -1655,6 +2053,39 @@ choose_collector_group() { "${COLLECTOR_PREFIX}_"*) echo "$cur"; return 0 ;; esac + # AN ANCESTOR'S GROUP ALREADY ANSWERED THIS. + # + # /usr/lib/python3.14/site-packages is nimgnu_python; a package needing + # .../site-packages/gi/overrides was asked to name a group for it, when + # the enclosing tree had been settled long ago. A nested directory + # inherits the nearest ancestor's collector group -- the decision to + # share python's tree was made when that group was created. + # + # Only ancestors, and only collector groups: an unrelated group on a + # parent (root, install) says nothing about sharing. + local _anc="$dir" _ag + while [ "$_anc" != "/" ] && [ -n "$_anc" ]; do + _anc="$(dirname "$_anc")" + [ "$_anc" = "/" ] && break + _ag="$(stat -c %G "$_anc" 2>/dev/null)" + case "$_ag" in + "${COLLECTOR_PREFIX}_"*) + # STDOUT IS THE RETURN VALUE HERE. + # + # This function ANSWERS on stdout, and these two lines were + # written with plain `say` -- so the caller captured the + # explanation as part of the group name: + # chgrp "# ... is inside ... nimgnu_gst\n# -- using + # that group\nnimgnu_gst" /usr/share/... + # which failed, silently, and looked like "nothing to grant". + # Everything informational in here goes to stderr; the + # prompt body already goes to /dev/tty for the same reason. + say "# $dir is inside $_anc, which is shared through $_ag" >&2 + say "# -- using that group (no new one needed)" >&2 + echo "$_ag"; return 0 ;; + esac + done + # Already a member of the owner's collector group? Then the decision was # made when that group was created: this package may install into the # owner's directories. Asking again for every directory is just noise -- @@ -1693,20 +2124,46 @@ choose_collector_group() { # Ask on /dev/tty, not stdin: this runs inside a `while ... done <<< list` # loop, so stdin is the directory list and a plain `read` would silently # consume that instead of waiting for an answer. - if [ ! -r /dev/tty ] || [ ! -w /dev/tty ] || [ "${LFS_ASSUME_YES:-0}" = 1 ]; then + # --yes ANSWERS A QUESTION; IT DOES NOT NAME THINGS. + # + # An EXISTING group is a yes/no ("share this directory through it?") and + # --yes may answer that. Creating a NEW one is a choice of name that + # will sit on the system for good, and the user asked to keep making it: + # a run with --yes still stops here when the group does not exist yet and + # a terminal is available. With no terminal there is nobody to ask, so + # the owner-named choice stands -- loudly. + if [ ! -r /dev/tty ] || [ ! -w /dev/tty ]; then + warn "granting $dir to group $by_owner (no terminal to ask on)" + group_exists "$by_owner" || warn " (this CREATES the group $by_owner)" echo "$by_owner"; return 0 fi + if assume_yes && group_exists "$by_owner"; then + warn "granting $dir to the existing group $by_owner (--yes)" + echo "$by_owner"; return 0 + fi + if assume_yes; then + warn "--yes does not name a NEW collector group -- asking:" + fi { say "" say " '$user' needs to install into: $dir" say " that directory belongs to package '$owner'" say " contents: $(ls -A "$dir" 2>/dev/null | wc -l) entry(ies)" - say " Which group should share it?" - say " 1) $by_owner (everything owned by '$owner' -- fewer groups)" - say " 2) $by_dir (just this directory -- more precise)" + say " Which group should share THIS DIRECTORY?" + # The two NAMES are the answer to the question; everything around them is + # explanation. Unhighlighted, the reader has to hunt for them in eight + # lines of prose -- and this prompt appears over and over during a stack. + say " 1) ${C_B}${by_owner}${C_OFF} (named after the owner, so the same group can be" + say " reused on other directories of '$owner' -- fewer groups overall)" + say " 2) ${C_B}${by_dir}${C_OFF} (named after the directory -- clearer, more groups)" say " or type another name" + say " Either way it grants $dir and nothing else: a group applies to the" + say " directories it is put on, never to everything a package owns." } > /dev/tty + # and on the ordinary output too, so a run behind a pipe does not look + # hung when it is only waiting + warn "WAITING FOR AN ANSWER: which group should share $dir?" # Ask until the answer is usable. Quietly substituting a different group # than the one asked for is worse than asking again -- the whole point of # this prompt is that the choice is the user's. @@ -1728,6 +2185,38 @@ choose_collector_group() { esac picked="$(sanitise_group_name "$ans")" if [ -n "$picked" ]; then + # A TYPO IN A GROUP NAME IS PERMANENT. + # + # Someone typing their own name meant to type a collector group + # and wrote `nimnu_gdb` for `nimgnu_gdb`. It is a valid group + # name, so it was created and the directory handed to it -- and + # nothing that looks for `nimgnu_*` groups will ever see it again. + # A name that is nearly the prefix is almost certainly meant to BE + # the prefix, so say so and offer the correction; a name that is + # nothing like it is a deliberate choice and is left alone. + # ONE DEFAULT, NOT TWO. `lfs` and `packagemanager` both fall back + # to "sysgroup"; only this line said "nimgnu". On a system whose + # config never set the key, groups were CREATED as nimgnu_* while + # packagemanager filtered for sysgroup_* -- so its own collector + # groups came out of `verify` as broken packages. + local _pfx="${COLLECTOR_PREFIX:-sysgroup}" + case "$picked" in + "${_pfx}_"*) ;; + *_*) + local _head="${picked%%_*}" _tail="${picked#*_}" + if _near_miss "$_head" "$_pfx"; then + echo " '$_head' is not '$_pfx' -- collector groups" \ + "start with '${_pfx}_'." > /dev/tty + printf " use '%s_%s' instead? [Y/n]: " \ + "$_pfx" "$_tail" > /dev/tty + local _yn="" + read -e -r _yn < /dev/tty || _yn="" + case "$_yn" in + [Nn]*) ;; + *) picked="${_pfx}_${_tail}" ;; + esac + fi ;; + esac # say so when the name had to be adjusted, so it is never a # surprise which group ends up owning the directory [ "$picked" = "$ans" ] \ @@ -1770,6 +2259,40 @@ grant_dir_access() { [ -d "$dir" ] || return 0 owner="$(stat -c %U "$dir" 2>/dev/null)" || return 0 + # A GROUP ON A LEAF IS NO USE IF THE CHAIN IS CLOSED. + # + # The user put it exactly right: a collector group does not work without + # its parents. Writing into .../gst/audio needs SEARCH (x) on every + # directory above it -- and those belong to other packages too: + # /usr/include/gstreamer-1.0/gst p_gstreamer10:nimgnu_gst drwxrwxr-x + # /usr/include/gstreamer-1.0 p_gstreamer10:p_gstreamer10 drwxr-xr-x + # Search happens to be open here (o+x), but a package that tightens its + # own directory would break every package installing below it. + # + # So walk up first and make sure each ancestor is at least searchable, + # stopping at the first one that already is for everyone. + local _up="$dir" + while :; do + _up="$(dirname "$_up")" + case "$_up" in /|/usr|/etc|/var|"") break ;; esac + [ -d "$_up" ] || break + if su -s /bin/sh -c "test -x '$_up'" "$user" 2>/dev/null; then + break # searchable already: so is everything above + fi + say " $_up is not searchable by '$user' -- opening the path" + soft "could not make $_up searchable" -- real_chmod o+x "$_up" + done + + # ALREADY WRITABLE IS NOTHING TO GRANT. + # + # The caller now offers every destination a meson install announced, not + # only the one that failed -- most of which the package can already write + # (its own directories, and shared ones it is a member of). Skipping + # them keeps that from creating groups nobody needs. + if su -s /bin/sh -c "test -w '$dir'" "$user" 2>/dev/null; then + return 0 + fi + # Grant exactly what was asked for. Promoting the request to "the top of # the owner's tree" turned one directory into a whole hierarchy -- and for # a root-owned tree that meant offering to share /sources. The package @@ -1850,6 +2373,22 @@ grant_dir_access() { fi grp="$(choose_collector_group "$dir" "$owner" "$user")" + # NEVER LEAVE IT EMPTY, AND NEVER FAIL QUIETLY. + # + # choose_collector_group prompts when the group does not exist yet. + # During a build stdin is not a terminal, so the prompt cannot run and it + # returns NOTHING -- then `chgrp "" dir` fails and grant_dir_access + # returns 1 with no message, which the caller reports as "nothing to + # grant". That is what five rounds of debugging on gst-plugins-bad were + # chasing. + # + # A name that cannot be asked for falls back to the owner's, which is the + # recommended answer anyway -- and says so. + if [ -z "$grp" ]; then + grp="$(sanitise_group_name "$(collector_group_for "${owner#p_}")")" + warn " could not ask which group should own $dir" + warn " (no terminal here) -- using $grp, named after its owner" + fi [ -n "$grp" ] || grp="$(collector_group_for "$owner")" if ! ensure_group "$grp"; then # A bad name should not end the build: say what was wrong with it and @@ -1879,7 +2418,11 @@ grant_dir_access() { # Installing into a second directory asks again -- and because the package # is already a member of the group by then, that grant is automatic (see # the fast path at the top of this function). - real_chgrp "$grp" "$dir" 2>/dev/null || return 1 + if ! real_chgrp "$grp" "$dir" 2>/dev/null; then + warn " chgrp $grp $dir FAILED -- the directory is unchanged" + warn " (does the group exist? getent group $grp)" + return 1 + fi soft "$dir is group $grp but not group-writable -- the grant will not take" \ -- real_chmod g+rwx "$dir" say " $dir -> group $grp (owner $owner, + $user)" @@ -1910,16 +2453,71 @@ unwritable_dirs_from_log() { | grep -aE "Permission denied|Operation not permitted|cannot (create|remove|touch|open)|couldn't copy|No such file or directory" \ | grep -avE "^[[:space:]]*(File \"|Traceback|at .* line )" \ | head -n 100)" - { printf '%s\n' "$_tail" | grep -aoE "'[^']+'|\"[^\"]+\"" | tr -d "'\"" + # A RELATIVE PATH NEEDS THE `cd` THAT PRECEDED IT. + # + # git's install pipes into tar: + # (cd '/usr/lib/perl5/5.42/site_perl' && umask 022 && tar xof -) + # /usr/bin/tar: ./Git: Cannot mkdir: Permission denied + # The path in the error is RELATIVE, so nothing above matched and the + # repair found nothing to grant -- while the directory that actually + # needs granting is named one line earlier, in the subshell's cd. + # + # Scan the tail in order, remembering the last absolute `cd`, and emit it + # whenever a relative complaint follows. + local _cdpaths + # MESON SAYS "OSError" AND NAMES NOTHING. + # + # gst-plugins-bad ended with + # Installing .../gstnonstreamaudiodecoder.h to /usr/include/gstreamer-1.0/gst/audio + # ERROR: Unhandled python OSError. + # -- the directory it could not write is on the line BEFORE the error, + # which is the only place meson puts it. Two repair rounds ran with + # nothing to grant. + # EVERY DESTINATION IT PRINTED, not just the last one. + # + # Taking only the line before the error granted one directory per round + # and the next round failed on the next directory -- and meson installs + # through PYTHON, not through our wrappers, so nothing on our side sees + # the EACCES to record it. + # + # What meson does give us is every destination it announced before dying. + # Granting all of them that exist and are not already writable costs + # nothing (a directory the package can already write is skipped) and + # settles the package in ONE round instead of one round per header + # directory. + local _mesondirs="" + if grep -qE "Unhandled python OSError|\[Errno 13\]" "$log" 2>/dev/null; then + _mesondirs="$(tail -c 2097152 "$log" 2>/dev/null \ + | awk '/^Installing .* to \// { print $NF }' \ + | sort -u)" + fi + + # (plain awk: match() with a capture array is a gawk extension) + _cdpaths="$(tail -c 524288 "$log" 2>/dev/null | awk ' + /cd .\/|cd \x27\// { + line = $0 + sub(/^.*cd \x27?/, "", line) + sub(/\x27.*$/, "", line) + sub(/ .*$/, "", line) + if (line ~ /^\//) last = line + } + /Cannot mkdir|Cannot open|Permission denied|cannot create/ { + if (last != "" && $0 ~ /(^|[ :])\.\//) print last + }' 2>/dev/null | sort -u)" + + { printf '%s\n' "$_mesondirs" + printf '%s\n' "$_cdpaths" + printf '%s\n' "$_tail" | grep -aoE "'[^']+'|\"[^\"]+\"" | tr -d "'\"" printf '%s\n' "$_tail" | grep -aoE "/[A-Za-z0-9._/+-]{6,}"; } \ | while IFS= read -r p; do case "$p" in /*) ;; *) continue ;; esac - case "$p" in - "${SNAP_ROOT%/}"/*) ;; - /usr/*|/etc/*|/var/*|/opt/*|/lib/*|/lib64/*|/bin/*|/sbin/*|\ - /home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*|/build/*) ;; - *) continue ;; - esac + if ! under_dir "$p" "$SNAP_ROOT"; then + case "$p" in + /usr/*|/etc/*|/var/*|/opt/*|/lib/*|/lib64/*|/bin/*|/sbin/*|\ + /home/*|/root/*|/srv/*|/boot/*|/tools/*|/sources/*|/build/*) ;; + *) continue ;; + esac + fi if [ -d "$p" ]; then echo "$p"; else echo "${p%/*}"; fi done | sort -u \ | while IFS= read -r d; do @@ -1931,6 +2529,395 @@ unwritable_dirs_from_log() { # Grant <user> access to everything in the log it could not write, and say # which groups that took. Returns 0 if anything was granted. + +# A directory that does not exist yet, inside somebody else's tree. +# +# cmake installs into /usr/share/vim/vimfiles/indent. /usr/share/vim belongs +# to vim; vimfiles has never existed; cmake's installer does one mkdir and +# reports +# file INSTALL cannot make directory ".../vimfiles/indent": +# No such file or directory +# A package user cannot create it either, because the parent is another +# package's directory. Granting does not help -- the problem is not that the +# directory is unwritable, it is that it is NOT THERE. +# +# The model already has the answer for a directory two packages share: root +# owns it, group install, group-writable. Nothing was applying that rule ON +# DEMAND, so every package that installs into a new subdirectory of another +# package's tree hits this. Create the missing directories that way, as +# root, and let the retry proceed. +missing_dirs_from_log() { + local log="$1" d parent + [ -f "$log" ] || return 0 + # THE PATTERN, NOT A LIST OF WORDINGS. + # + # First version matched cmake's `cannot make directory "X"` and + # coreutils' `cannot create directory 'X'` -- and then missed + # install_libpaper: line 150: /etc/profile.d/libpaper.sh: + # No such file or directory + # which is bash failing to redirect into a directory that is not there. + # Enumerating messages is the same mistake as enumerating verbs was in + # the book guard: every tool words it differently. The pattern is + # "No such file or directory" plus an absolute path. + # + # Two classes, one rule each: + # a directory the tool tried to CREATE -> make that path + # anything else that could not be written -> make its PARENT + # and in both cases only when what is missing really is the directory + # chain: if the parent already exists, the error is about something else + # (a source file the build never produced) and inventing a directory + # would turn a clear message into a mystery. + { + { + grep -oE 'cannot make directory "[^"]+"' "$log" 2>/dev/null | cut -d'"' -f2 + grep -oE "cannot create directory .[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + } | while IFS= read -r d; do + case "$d" in /*) [ -e "$d" ] || printf '%s\n' "$d" ;; esac + done + + grep "No such file or directory" "$log" 2>/dev/null \ + | grep -oE "/[^ '\"()]+" \ + | while IFS= read -r d; do + d="${d%:}" + [ -e "$d" ] && continue + parent="$(dirname "$d")" + # the directory chain is what is missing -- not a file inside a + # directory that exists + [ -d "$parent" ] && continue + printf '%s\n' "$parent" + done + } | while IFS= read -r d; do + [ -n "$d" ] || continue + # AN EMPTY VARIABLE IS NOT A PREFIX, IT IS A WILDCARD. + # + # This ran as `case "$d" in "$SRCROOT"/*)` and + # `case "$d" in "${SNAP_ROOT%/}"/*)`. In the chroot SNAP_ROOT is "/", + # so "${SNAP_ROOT%/}" is EMPTY: the "is it under the root" test then + # failed to match anything, while the SRCROOT exclusion became "/*" + # and excluded everything. cmake's missing directory was therefore + # invisible on the only system that matters -- and perfectly visible + # in every sandbox, where the root is a subdirectory. Three sessions + # of "the repair does not run" came from those two patterns. + # + # Prefix tests by string removal, with the empty root meaning "/". + _r="${SNAP_ROOT%/}" + [ -n "$_r" ] && [ "${d#"$_r"/}" = "$d" ] && continue # not under the root + case "$d" in /*) ;; *) continue ;; esac # absolute only + # never invent directories in a build tree, the source store or /tmp: + # a path that failed THERE is the build's own problem to report + for _skip in "$SRCROOT" "$_r/sources" "$_r/tmp" "$_r/build" /build; do + [ -n "$_skip" ] || continue + [ "${d#"$_skip"/}" = "$d" ] || continue 2 + done + printf '%s\n' "$d" + done | sort -u +} + +# A file the installer may not replace, because another package owns it. +# +# install: cannot remove '/usr/lib/pkgconfig/freetype2.pc': +# Operation not permitted +# /usr/lib/pkgconfig/freetype2.pc -rw-r--r-- p_cairo:p_cairo +# in /usr/lib/pkgconfig drwxrwxr-t root:install +# +# freetype2's OWN file, owned by cairo -- the over-claiming manifests of +# 1.12.11 and earlier, made real by a `verify --fix`. The directory is +# sticky (sealed), so only the owner may replace the file, and EPERM is not +# something a collector group can grant away. +# +# In this model the package that installs a file owns it: that is what +# verify's ownership rule says, and the installer is by definition the one +# installing it now. So hand the file over and let the install proceed -- +# loudly, naming the previous owner, because a package taking another's file +# is worth seeing. +takeover_files_from_log() { + local log="$1" user="$2" f owner n=0 + [ -f "$log" ] || return 1 + while IFS= read -r f; do + [ -n "$f" ] || continue + under_dir "$f" "$SNAP_ROOT" || continue + for _skip in "$SRCROOT" "${SNAP_ROOT%/}/tmp" "${SNAP_ROOT%/}/sources"; do + [ -n "$_skip" ] && under_dir "$f" "$_skip" && continue 2 + done + [ -e "$f" ] || [ -L "$f" ] || continue + owner="$(stat -c %U "$f" 2>/dev/null)" + [ -n "$owner" ] || continue + [ "$owner" = "$user" ] && continue + # root's files are not another package's mistake -- leave them; only + # a PACKAGE USER's claim is transferable. + case "$owner" in root|UNKNOWN|"") continue ;; esac + user_exists "$owner" || continue + if real_chown -h "$user:$user" "$f" 2>/dev/null; then + warn "!! $f belonged to $owner -- given to $user, which is installing it" + n=$((n + 1)) + fi + done <<< "$( + { grep -oE "cannot remove '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + grep -oE "cannot create regular file '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + grep -oE "cannot unlink '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + grep -oE "cannot overwrite '[^']+'" "$log" 2>/dev/null | cut -d"'" -f2 + } | sort -u)" + [ "$n" -gt 0 ] +} + +# Create the missing directories as shared install directories. Returns 0 if +# anything was created, so the caller knows a retry is worth it. +# A new directory in another package's tree: which collector group? +# +# grant_dir_access names the group after the DIRECTORY'S owner, which here is +# the package that just created it -- so vim's tree got nimgnu_cmake: +# /usr/share/vim/vimfiles p_cmake nimgnu_cmake +# The group belongs to the TREE, not to whoever happened to install first. +# And when no such group exists yet, naming it is a decision, not a yes/no: +# --yes means "do not ask me to confirm", it does not mean "pick a name for +# me". So an existing group is reused silently, and a new one is ASKED for +# whenever there is a terminal; without one, the tree owner's name is used -- +# never the creator's. +share_new_dir_with_tree_owner() { + local dir="$1" powner="$2" user="$3" grp by_tree by_creator ans + # ITS OWN TREE NEEDS NO GROUP. + # + # p_rust created /opt/rustc-1.93.1/share -- inside p_rust's own tree -- + # and was asked which collector group should own it, offering the same + # name twice ("named after the tree" and "named after the package that + # made it" are one name when they are the same package). A question with + # one answer is not a question. + if [ "$powner" = "$user" ]; then + ok "# $dir belongs to $user, which created it -- no group needed" + return 0 + fi + # NAME THE PACKAGE, NOT THE ACCOUNT. collector_group_for unprefixes with + # PKGUSR_PREFIX, so when that is empty (or the account was made under a + # different prefix) "p_rust" survived intact and the group came out + # nimgnu_p_rust. Strip the tools' own default prefix as well. + by_tree="$(sanitise_group_name "$(collector_group_for "${powner#p_}")")" + by_creator="$(sanitise_group_name "$(collector_group_for "${user#p_}")")" + + # THE PARENT DIRECTORY DECIDES, not the existence of a group somewhere. + # + # If the directory we are creating sits in a directory that already + # carries a collector group, that group is the answer -- the decision was + # made when it was put there, and a subdirectory of a shared directory is + # shared the same way. Only when the parent carries no such group is + # there a new decision to make, and then it is ASKED, like every other + # collector-group question on this system. + local _pgrp + _pgrp="$(stat -c %G "$(dirname "$dir")" 2>/dev/null)" + case "$_pgrp" in + "${COLLECTOR_PREFIX}_"*) + grp="$_pgrp" + ok "# $dir -> group $_pgrp (inherited from its parent directory)" + soft "could not give $dir to group $grp" -- real_chgrp "$grp" "$dir" + soft "$dir is group $grp but not group-writable -- the grant will not take" \ + -- real_chmod g+rwx "$dir" + add_to_group "$user" "$grp" >/dev/null 2>&1 || true + return 0 ;; + esac + + if [ -r /dev/tty ] && [ -w /dev/tty ]; then + warn "WAITING FOR AN ANSWER: which collector group should own ${C_B}$dir${C_OFF}?" + { + say "" + say " package : ${C_B}$user${C_OFF} (it is doing the installing)" + say " directory: ${C_B}$dir${C_OFF}" + say " in the tree of: ${C_B}$powner${C_OFF}" + say "" + say " Which collector group should own it?" + say " 1) ${C_B}${by_tree}${C_OFF} ${C_OK}<- recommended${C_OFF}:" + say " named after the tree, which is what other packages" + say " installing into $powner's directories will look for" + say " 2) ${C_B}${by_creator}${C_OFF} (named after the package that made it)" + say " or type another name" + } > /dev/tty + printf ' [1]: ' > /dev/tty + read -e -r ans < /dev/tty || ans="" + case "$ans" in + ""|1) grp="$by_tree" ;; + 2) grp="$by_creator" ;; + *) grp="$(sanitise_group_name "$ans")" ;; + esac + else + grp="$by_tree" + warn "no terminal to ask -- using $grp (named after the tree, not the creator)" + fi + + group_exists "$grp" || create_collector_group "$grp" || return 1 + soft "could not give $dir to group $grp" -- real_chgrp "$grp" "$dir" + soft "$dir is group $grp but not group-writable -- the grant will not take" \ + -- real_chmod g+rwx "$dir" + add_to_group "$powner" "$grp" >/dev/null 2>&1 || true + add_to_group "$user" "$grp" >/dev/null 2>&1 || true + ok "# $dir -> group $grp (owner $user, + $powner)" +} + +# Ask before changing the system. Creating a directory the book never +# mentions is a change to someone else's tree, and it must be the user's +# call: the shared-directory rule is right, but applying it unasked is not. +# --yes (assume_yes) is that permission given in advance; without a terminal +# the answer is NO, because silence is not consent. +confirm_change() { + local what="$1" ans + if assume_yes; then + warn "$what (--yes)" + return 0 + fi + [ -r /dev/tty ] && [ -w /dev/tty ] || { warn "$what -- NOT done (no terminal to ask)"; return 1; } + # ASK WHERE THE OUTPUT IS. The question goes to /dev/tty so it survives + # the pipes, but the build's output goes through tee -- so a run that + # looked hung was a prompt nobody could see. Say it on BOTH: the log + # shows what is being waited for, the terminal takes the answer. + warn "WAITING FOR AN ANSWER: $what [y/N]" + { say ""; say " $what"; } > /dev/tty + printf ' do it? [y/N]: ' > /dev/tty + read -e -r ans < /dev/tty || return 1 + case "$ans" in y|Y|yes) return 0 ;; *) warn "declined"; return 1 ;; esac +} + +create_missing_dirs_from_log() { + local log="$1" user="$2" d made=0 target + while IFS= read -r d; do + [ -n "$d" ] || continue + # missing_dirs_from_log already decided WHICH directory is missing + target="$d" + [ -e "$target" ] && continue + case "$target" in "${SNAP_ROOT%/}"|/|"") continue ;; esac + # WHO SHOULD OWN A DIRECTORY WE CREATE? + # + # It was root:install, and that broke cmake a second way: + # cannot set permissions on "/usr/share/vim/vimfiles/indent": + # Operation not permitted + # chmod requires OWNERSHIP, and an installer sets modes on the + # directories it creates. root:install is the rule for the SHARED + # INSTALL DIRECTORIES (/usr/lib, /usr/share/man/...), which is a + # different thing: those are the system's, listed in installdirs.lst, + # and no package owns them. + # + # This is a new directory inside ANOTHER PACKAGE'S tree + # (/usr/share/vim belongs to p_vim). The model already answers that: + # the package that creates it owns it, and a COLLECTOR GROUP grants + # the other package access -- the same nimgnu_* mechanism, and the + # same question, that every other shared directory goes through. + # Inventing a second rule here is what made vimfiles different from + # everything else on the system. + # WHOSE TREE IS THIS, REALLY? + # + # Asking the nearest existing ancestor was not enough: `mkdir -p` + # creates INTERMEDIATE directories as root:root, so round 1 left + # /usr/share/vim/vimfiles root:root + # and round 2 read that artifact of our own making as "root's tree" + # and applied the system rule to /usr/share/vim/vimfiles/syntax. A + # directory we created a moment ago is not evidence about ownership. + # + # So walk UP until something actually decides: a real install + # directory (the system's rule) or a directory a PACKAGE owns (the + # collector-group rule). A root:root directory that is neither is + # exactly the artifact above -- keep walking. + local _parent _powner + _parent="$(dirname "$target")" + while [ "$_parent" != / ] && [ -n "$_parent" ]; do + if [ -d "$_parent" ]; then + is_install_dir "$_parent" && break + _powner="$(stat -c %U "$_parent" 2>/dev/null)" + [ -n "$_powner" ] && [ "$_powner" != root ] && break + fi + _parent="$(dirname "$_parent")" + done + _powner="$(stat -c %U "$_parent" 2>/dev/null)" + if is_install_dir "$_parent" || [ "$_powner" = root ] || [ -z "$_powner" ]; then + # under the system's own directories: the system's rule applies + confirm_change "create $target (root:install, group-writable) so '$user' can install into it" \ + || continue + if mkdir -p "$target" 2>/dev/null; then + set_install_dir_owner "$target" 2>/dev/null || true + soft "created $target but could not make it group-writable -- the package user still cannot install into it" \ + -- real_chmod g+w "$target" + ok "# created shared directory $target (root:install, group-writable)" + made=1 + else + warn "could not create $target" + fi + else + confirm_change "create $target inside '$_powner''s tree, owned by '$user', and share it with a collector group" \ + || continue + # EVERY component we create, not just the last one. mkdir -p + # would leave the intermediates root:root -- the artifact that + # misled the ancestor walk above, and equally unusable by the + # package that has to install into them. + local _need="" _p2="$target" + while [ "$_p2" != "$_parent" ] && [ "$_p2" != / ]; do + [ -d "$_p2" ] || _need="$_p2 $_need" + _p2="$(dirname "$_p2")" + done + if mkdir -p "$target" 2>/dev/null; then + local _nd + for _nd in $_need; do + soft "created $_nd but could not give it to $user -- the package cannot set its mode" \ + -- real_chown "$user:$user" "$_nd" + ok "# created $_nd (owned by $user, which is installing into it)" + # and let the OWNER of the tree keep access + share_new_dir_with_tree_owner "$_nd" "$_powner" "$user" + done + made=1 + else + warn "could not create $target" + fi + fi + done <<< "$(missing_dirs_from_log "$log")" + [ "$made" = 1 ] +} + +meson_takeover_from_log() { + # THE DIRECTORY WAS NEVER THE PROBLEM. + # + # /usr/include/gstreamer-1.0/gst/audio -- nothing to grant (already writable) + # .../gst/audio drwxrwxr-x p_gst10-plugins-base:p_gst10-plugins-base + # + # The package CAN write there. What it cannot do is finish installing + # over a FILE that belongs to another package: meson copies the contents + # (allowed -- the directory is writable) and then chmod/chown/utime the + # destination, which only its owner may do. Python raises EPERM and + # meson reports "Unhandled python OSError" with no path, which is how + # this looked like a directory problem for five rounds. + # + # Ownership follows the installer -- the rule this whole system is built + # on. So the files this package is installing over become its own, and + # meson's chmod succeeds. Only files it is actually installing (named on + # meson's own "Installing SRC to DIR" lines) and only where the current + # owner is another PACKAGE USER: root's files are never taken. + local log="$1" user="$2" took=0 src dest f owner + grep -qE "Unhandled python OSError|\[Errno 13\]|\[Errno 1\]" "$log" 2>/dev/null || return 1 + while IFS= read -r line; do + src="${line#Installing }"; src="${src%% to *}" + dest="${line##* to }" + [ -n "$src" ] && [ -d "$dest" ] || continue + f="$dest/$(basename "$src")" + [ -e "$f" ] || continue + owner="$(stat -c %U "$f" 2>/dev/null)" || continue + [ "$owner" = "$user" ] && continue + case "$owner" in + root|"") continue ;; # never take root's files + esac + pkgusr_kind "$owner" >/dev/null 2>&1 || true + say " $f belongs to $owner -- handing it to $user (the installer owns what it installs)" + soft "could not hand $f to $user" -- real_chown "$user:$user" "$f" && took=$((took+1)) + done <<< "$(tail -c 2097152 "$log" 2>/dev/null | grep '^Installing .* to /')" + # ONE FILE PER ROUND IS A LOOP, NOT PROGRESS. + # + # meson stops at the FIRST file it may not replace, so its log names only + # that one -- and the next round learns exactly one more. libdisplay-info + # went twelve rounds, re-cloning a subproject each time, handing over one + # .mo file per round. When a package keeps taking a single file from the + # same other package, it is installing a BUNDLED COPY of it, and the fix + # is upstream of the ownership question. + if [ "$took" = 1 ] && [ -n "${owner:-}" ]; then + LAST_SINGLE_HANDOVER="$owner" + else + LAST_SINGLE_HANDOVER="" + fi + [ "$took" -ge 1 ] +} + + auto_grant_from_log() { local log="$1" user="$2" need d granted=0 need="$(unwritable_dirs_from_log "$log" "$user")" @@ -1939,15 +2926,36 @@ auto_grant_from_log() { say "" note "# '$user' could not write into some directories -- granting" note "# access via collector groups and retrying:" - # Many failing paths walk up to the SAME tree top, so without this the same - # directory is granted once per path -- nine identical grants in one round. + # DEDUP BY DIRECTORY, NOT BY TREE TOP. + # + # The dedup collapsed every failing path to the top of its tree, so once + # ONE directory under /usr/include/gstreamer-1.0 had been granted, every + # other directory under it was skipped -- including the one that actually + # failed: + # /usr/include/gstreamer-1.0/gst/audio drwxrwxr-x p_gst10-plugins-base + # Two repair rounds ran, granted a sibling each time, and the build failed + # on the same directory. + # + # A group applies to the directories it is put on (the prompt says so), + # so each distinct directory needs its own grant. The original problem + # the tree-top dedup solved -- the same directory named by nine paths -- + # is solved by deduping on the directory itself. local seen="" while IFS= read -r d; do [ -n "$d" ] || continue - local top; top="$(collector_tree_top "$d")" - case "$seen" in *"|$top|"*) continue ;; esac - seen="$seen|$top|" - grant_dir_access "$d" "$user" && granted=1 + case "$seen" in *"|$d|"*) continue ;; esac + seen="$seen|$d|" + # SAY WHAT HAPPENED TO EACH ONE. + # + # This loop printed a header and then, when every candidate was + # skipped, nothing at all -- so a failing package looked like a + # repair that had not run, and four fixes were made by guessing. + # Every candidate now reports its outcome. + if grant_dir_access "$d" "$user"; then + granted=1 + else + say " $d -- nothing to grant (already writable, or not a directory)" + fi done <<< "$need" [ "$granted" = 1 ] } @@ -2011,6 +3019,164 @@ cmd_prune_locales() { say " (packages will recreate them on reinstall; re-run this when you like)" } +# Strip debug symbols -- the book's 8.85, done the package-user way. +# +# `strip` does not edit a file: it writes a stripped copy beside it and +# renames the copy into place. Two things follow, and both shaped this: +# +# * The NEW file belongs to whoever ran strip. Run as root, every stripped +# binary would become root's -- and here the owner IS the record of which +# package installed it. So the files are grouped by owner and each group +# is stripped AS that owner, via su. That is also why the shared install +# directories can take it: they are group-writable for `install` and +# sticky, so an owner may replace its own file and nobody else's. +# * A running process keeps its open inode across the rename, so stripping +# the binaries in use right now (bash, strip itself, libc) is safe; the +# book's copy-to-/tmp dance guards against in-place edits, which GNU +# strip does not do. +# +# Scope is the book's: shared objects under /usr/lib and /usr/libexec, and +# every ELF file under /usr/bin, /usr/sbin and /usr/libexec. Libtool .la +# files go too (the book's cleanup) unless --keep-la: left behind, they make +# later BLFS builds link against paths that no longer exist. +cmd_strip() { + need_root + local run=0 keep_la=0 + while [ $# -gt 0 ]; do + case "$1" in + --run) run=1 ;; + --keep-la) keep_la=1 ;; + *) die "unknown option: $1 (usage: lfs-helper strip [--run] [--keep-la])" ;; + esac + shift + done + + local r="${SNAP_ROOT%/}" + # benign: the directory usually exists already + mkdir -p "$STATE/tmp" 2>/dev/null || true + local all="$STATE/tmp/strip-files.$$" + : > "$all" || die "cannot write to $STATE/tmp" + + say "Collecting what can be stripped (this reads every candidate's header) ..." + # The book's candidate set, filtered down to what really is ELF -- scripts, + # data files and already-removed symlink targets fall out here, so nothing + # below has to tolerate strip failing on a non-binary. + local f magic + { + find "$r/usr/lib" "$r/usr/libexec" -xdev -type f -name '*.so*' \ + ! -name '*.dbg' 2>/dev/null + find "$r/usr/bin" "$r/usr/sbin" "$r/usr/libexec" -xdev -type f \ + 2>/dev/null + } | LC_ALL=C sort -u | while IFS= read -r f; do + magic="" + IFS= LC_ALL=C read -r -n 4 magic < "$f" 2>/dev/null || true + [ "$magic" = $'\x7fELF' ] && printf '%s\n' "$f" + done > "$all" + + local total; total="$(count_stdin_lines < "$all")" + + # group by owner: one list per owner, stripped in one su each + local bylist="$STATE/tmp/strip-owners.$$" + xargs -d '\n' -r stat -c '%U|%n' < "$all" > "$bylist" \ + || die "cannot stat the candidate files" + local -a owners=() + local o + while IFS= read -r o; do + [ -n "$o" ] && owners+=("$o") + done < <(cut -d'|' -f1 "$bylist" | LC_ALL=C sort -u) + + # libtool archives, the book's cleanup + local lalist="$STATE/tmp/strip-la.$$" + find "$r/usr/lib" "$r/usr/libexec" -xdev -type f -name '*.la' \ + 2>/dev/null > "$lalist" || true + local nla; nla="$(count_stdin_lines < "$lalist")" + + local before + before="$(xargs -d '\n' -r stat -c '%s' < "$all" \ + | awk '{s+=$1} END{printf "%d", s/1024/1024}')" + + say "" + say " $total ELF file(s), ${before:-0} MB, owned by ${#owners[@]} account(s):" + for o in "${owners[@]}"; do + say " $(grep -c "^$o|" "$bylist") owned by $o" + done | LC_ALL=C sort -rn -k1 | head -8 + [ "${#owners[@]}" -gt 8 ] && say " ... and $((${#owners[@]} - 8)) more" + if [ "$keep_la" = 1 ]; then + say " $nla libtool .la file(s) -- kept (--keep-la)" + else + say " $nla libtool .la file(s) to remove" + fi + + if [ "$run" = 0 ]; then + warn "(dry run -- nothing was changed. Add --run to strip them.)" + rm -f "$all" "$bylist" "$lalist" + return 0 + fi + + # The runner each owner executes. A generated file, not an inline -c + # string, so the quoting cannot go wrong and the loop is the same for + # root and for a package user. Failures are printed as marked lines and + # collected by the parent -- never swallowed, never fatal per file. + local runner="$STATE/tmp/strip-run.$$.sh" + cat > "$runner" <<'RUNNER' +#!/bin/bash +# generated by: lfs-helper strip +set -u +while IFS= read -r f; do + out="$(strip --strip-unneeded -- "$f" 2>&1)" \ + || printf 'FAILSTRIP\t%s\t%s\n' "$f" "$out" +done < "$1" +exit 0 +RUNNER + chmod 755 "$runner" + + local failed="$STATE/tmp/strip-failed.$$" + : > "$failed" + local list n + for o in "${owners[@]}"; do + list="$STATE/tmp/strip-list.$o.$$" + grep "^$o|" "$bylist" | cut -d'|' -f2- > "$list" + chmod 644 "$list" + n="$(count_stdin_lines < "$list")" + if [ "$o" = root ]; then + say " stripping $n file(s) as root ..." + bash "$runner" "$list" | grep '^FAILSTRIP' >> "$failed" || true + elif ! id -u "$o" >/dev/null 2>&1; then + # an owner the user database does not know cannot run anything; + # its files stay exactly as they are, and that is said, not hidden + warn " ${n} file(s) owned by unknown account '$o' -- left alone" + elif have_su; then + say " stripping $n file(s) as $o ..." + su -s /bin/bash "$o" -c "bash '$runner' '$list'" \ + | grep '^FAILSTRIP' >> "$failed" || true + else + warn " no 'su' available -- ${n} file(s) owned by '$o' left alone" + fi + rm -f "$list" + done + + local nfail; nfail="$(count_stdin_lines < "$failed")" + if [ "$nfail" != 0 ]; then + warn "$nfail file(s) could not be stripped:" + cut -f2,3 "$failed" | head -5 | sed 's/^/ /' >&2 + [ "$nfail" -gt 5 ] && warn " ... and $((nfail - 5)) more" + fi + + if [ "$keep_la" != 1 ] && [ "$nla" != 0 ]; then + # deleting is a directory operation, not a rewrite: it changes no + # file's owner, so root may do it directly + xargs -d '\n' -r rm -f -- < "$lalist" + say " removed $nla libtool .la file(s)" + fi + + local after + after="$(xargs -d '\n' -r stat -c '%s' < "$all" 2>/dev/null \ + | awk '{s+=$1} END{printf "%d", s/1024/1024}')" + ok "stripped: ${before:-0} MB -> ${after:-0} MB across $total file(s)" + rm -f "$all" "$bylist" "$lalist" "$runner" "$failed" + return 0 +} + # Find (and optionally remove) per-user Python installs. # # pip falls back to ~/.local when the system site-packages is not writable. The @@ -2185,7 +3351,7 @@ cmd_clean_sources() { say " nothing here, and one above 10000" say " collides with a real package user)" say " $blddir -- build scratch" - say " * leftover build markers (.cc-build-*, .cc-dir-*) -> removed" + say " * leftover build markers (.pkgusr-*, .cc-*) -> removed" say " (unpacked trees are left alone: a build may be using one. To" say " drop them all, delete $blddir -- it is rebuilt on demand.)" say "" @@ -2196,7 +3362,7 @@ cmd_clean_sources() { # on top of a package user and report a plausible, wrong name. orphans="$(find "$srcdir" -maxdepth 1 -mindepth 1 -not -type d \ \( -not -user root -o -not -group root \) 2>/dev/null | sort)" - markers="$(find "$srcdir" "$blddir" -maxdepth 1 -name '.cc-*' \ + markers="$(find "$srcdir" "$blddir" -maxdepth 1 \( -name '.cc-*' -o -name '.pkgusr-*' \) \ 2>/dev/null | sort)" local n_o n_m @@ -2293,6 +3459,19 @@ cmd_fix_perms() { # otherwise behave normally. make_wrappers() { mkdir -p "$WRAPPERS" + # STAMP THEM. A wrapper is code, and code goes stale: a snapshot restored + # the wrapper from before the bundled-option fix, it existed and was + # executable, and the caller's check -- "missing or incomplete" -- was + # satisfied. A fixed wrapper on the tools side never reached the tree, and + # the exact failure the fix was for came back on the exact same line. + # Same species as a stale chroot copy of the tools, which is why the stamp + # is the same fingerprint --version prints. + # ${VAR:-} and the `command -v` guard: the test suite lifts this function + # out with sed and runs it under `set -u` in a context where neither the + # version variable nor _build_id exists -- and a stamp that kills the + # function it stamps is worse than no stamp. + echo "lfs-helper ${LFS_HELPER_VERSION:-unknown} (build $(command -v _build_id >/dev/null 2>&1 && _build_id || echo unknown))" \ + > "$WRAPPERS/.written-by" 2>/dev/null cat > "$WRAPPERS/chown" <<'EOF' #!/bin/bash @@ -2308,6 +3487,24 @@ echo "*** chgrp $* (skipped: package users cannot change group)" >&2 exit 0 EOF + # ldconfig: the LIBRARY CACHE is root's, refreshed after the install. + # + # /etc/ld.so.cache is rewritten by every package that installs a library. + # ldconfig writes ld.so.cache~ and RENAMES it over the old file, and in a + # sticky /etc only the file's owner may do that -- so a package user got + # Renaming of /etc/ld.so.cache~ to /etc/ld.so.cache failed: + # Operation not permitted + # Unsealing /etc to fix that would let every package delete every other + # package's config. The cache is not the package's to write: it is a + # cache, and the tools refresh it as root when the install finishes -- + # the same treatment mandb gets. + cat > "$WRAPPERS/ldconfig" <<'EOF' +#!/bin/bash +echo "*** ldconfig: skipped -- the library cache is refreshed as root after this install" >&2 +exit 0 +EOF + chmod 755 "$WRAPPERS/ldconfig" + cat > "$WRAPPERS/install" <<'EOF' #!/bin/bash # Drop -o/-g (ownership), refuse setuid/setgid modes, and never re-create or @@ -2319,6 +3516,45 @@ EOF real=/usr/bin/install args=(); skipped="" +# BUNDLED SHORT OPTIONS FIRST, or none of the rules below see what they are +# looking for. Everything here matched `-d` only as a whole argument, and +# make-ca's Makefile writes +# install -vdm755 /usr/sbin +# -- one token. The directory rule never fired, the real install ran, and it +# died on the exact message this wrapper's own comment quotes: +# install: cannot change permissions of '/usr/sbin': Operation not permitted +# `install -d` on an EXISTING directory still applies the mode, and chmod needs +# ownership, not write permission -- so no group membership could ever fix it. +# +# Split `-vdm755` into `-v -d -m 755`, so one spelling reaches every rule. +# These are install(1)'s short options that take a value; for them the rest of +# the token is the value (or the next argument, if the token ends there). +_expanded=() +for _a in "$@"; do + case "$_a" in + -[!-]*) + _rest="${_a#-}" + while [ -n "$_rest" ]; do + _c="${_rest%"${_rest#?}"}" # first character + _rest="${_rest#?}" + case "$_c" in + m|o|g|t|S) # takes a value + _expanded+=("-$_c") + # `-m755`: the value is the rest of this token. + # `-m 755`: it is the next argument, which reaches the + # loop on its own and is passed through as a word. + if [ -n "$_rest" ]; then + _expanded+=("$_rest"); _rest="" + fi ;; + *) _expanded+=("-$_c") ;; + esac + done ;; + *) + _expanded+=("$_a") ;; + esac +done +set -- "${_expanded[@]}" + # -d mode: create directories. Skip the ones that already exist. want_dirs=0 for a in "$@"; do case "$a" in -d|--directory) want_dirs=1 ;; esac; done @@ -2346,6 +3582,9 @@ fi while [ $# -gt 0 ]; do case "$1" in -o|--owner|-g|--group) skipped="$skipped $1 $2"; shift 2 ;; + # Still reachable for the long forms (--owner=root) -- the expansion + # above only splits SHORT clusters, so `-oroot` arrives here as + # `-o root` and is caught by the line before this one. -o*|-g*) skipped="$skipped $1"; shift ;; -m|--mode) case "$2" in @@ -2358,7 +3597,109 @@ while [ $# -gt 0 ]; do esac done [ -n "$skipped" ] && echo "*** install:$skipped (skipped: ownership is the package user's)" >&2 + +# AN ARTIFACT THAT WAS NEVER BUILT IS NOT A FAILED INSTALL. +# +# The book installs optional extras with commands the package's own +# `make install` does not run: +# install -v -m644 doc/doxygen/html/*.* (alsa-lib, needs Doxygen) +# install -Dm644 docs/build/man/libuv.1 (libuv, needs sphinx) +# The tool that BUILDS them is optional, so on a system without it the +# sources do not exist and install dies -- taking the whole package with it, +# after a successful build and a successful `make install`. +# +# If every SOURCE argument is missing, there is nothing to install: say so +# and succeed. A partial set still goes to the real install, so a genuinely +# half-built package is not hidden. Sources are relative paths in the build +# tree here; an absolute destination is never treated as a source. +_srcs=(); _n=${#args[@]} +if [ "$_n" -ge 2 ]; then + for ((_i = 0; _i < _n - 1; _i++)); do + case "${args[$_i]}" in + -*) continue ;; + esac + # the value of a value-taking option is not a source + if [ "$_i" -gt 0 ]; then + case "${args[$((_i - 1))]}" in -m|--mode|-o|--owner|-g|--group|-t|-S) continue ;; esac + fi + _srcs+=("${args[$_i]}") + done + if [ "${#_srcs[@]}" -gt 0 ]; then + _have=0 + for _s in "${_srcs[@]}"; do + [ -e "$_s" ] && _have=1 && break + done + if [ "$_have" = 0 ]; then + echo "*** install: none of these exist, nothing to install: ${_srcs[*]}" >&2 + echo "*** (an optional artifact that was never built -- not a failure)" >&2 + exit 0 + fi + fi +fi +# A SYSTEMD UNIT ON A SYSTEM WITH NO SYSTEMD. +# +# The systemd book's wpa_supplicant page ends with +# install -m644 ... /usr/lib/systemd/system/ +# and on a SysV system that directory does not exist, so install fails and +# takes the package with it -- after wpa_supplicant, wpa_cli and the man +# pages had all installed correctly. This is the same situation the +# systemctl shim was written for: the book assumes an init this machine does +# not run. Say so and succeed, rather than creating a directory of units +# nothing will ever read. +_dest="${args[$((${#args[@]} - 1))]}" +case "$_dest" in + /usr/lib/systemd/*|/lib/systemd/*|/usr/lib/systemd|/lib/systemd) + if [ ! -d /usr/lib/systemd/system ] && ! command -v systemctl >/dev/null 2>&1; then + echo "*** install: skipped $_dest -- this system does not run systemd" >&2 + echo "*** (a unit file from a systemd-book page; nothing would read it)" >&2 + exit 0 + fi ;; +esac exec "$real" "${args[@]}" +EOF + + # A SERVICE ACCOUNT IS ROOT'S JOB. + # + # fcron's page starts with + # groupadd -g 22 fcron && useradd ... -u 22 fcron + # and the phase runs as p_fcron, which cannot write /etc/group: + # groupadd: cannot lock /etc/group; try again later + # cmd_build creates these as root BEFORE the phase (see + # create_service_accounts_from_script). These wrappers make the package + # user's own attempt a no-op when the account is already there, so the + # book's command can stay in the script where a reader expects it. + cat > "$WRAPPERS/groupadd" <<'EOF' +#!/bin/bash +real=/usr/sbin/groupadd +_name="" +for a in "$@"; do case "$a" in -*) ;; *) _name="$a" ;; esac; done +if [ -n "$_name" ] && getent group "$_name" >/dev/null 2>&1; then + echo "*** groupadd: group '$_name' already exists -- nothing to do" >&2 + exit 0 +fi +if [ "$(id -u)" != 0 ]; then + echo "*** groupadd: '$_name' does not exist and a package user cannot" >&2 + echo "*** create it. As root: groupadd $*" >&2 + exit 1 +fi +exec "$real" "$@" +EOF + + cat > "$WRAPPERS/useradd" <<'EOF' +#!/bin/bash +real=/usr/sbin/useradd +_name="" +for a in "$@"; do case "$a" in -*) ;; *) _name="$a" ;; esac; done +if [ -n "$_name" ] && getent passwd "$_name" >/dev/null 2>&1; then + echo "*** useradd: user '$_name' already exists -- nothing to do" >&2 + exit 0 +fi +if [ "$(id -u)" != 0 ]; then + echo "*** useradd: '$_name' does not exist and a package user cannot" >&2 + echo "*** create it. As root: useradd $*" >&2 + exit 1 +fi +exec "$real" "$@" EOF cat > "$WRAPPERS/chmod" <<'EOF' @@ -2369,7 +3710,17 @@ me="$(id -un)" for a in "$@"; do case "$a" in [2467][0-7][0-7][0-7]|*[ug]+s*|*+s*) + # WRITE DOWN WHAT WAS REFUSED. + # + # fcron installs fcrontab setgid `fcron`; the bit was refused + # here, correctly, and then nothing said so again. Much later + # `fcrontab -z -u systab` failed with + # ERROR could not change egid to 22: Operation not permitted + # -- as ROOT, which makes no sense until you know the binary lost + # its setgid bit at install time. A refusal nobody records is a + # decision that has to be re-derived from its consequences. echo "*** chmod $* (skipped: refuses to set setuid/setgid; do it as root if needed)" >&2 + printf 'chmod %s\n' "$*" >> "$HOME/.pkgusr-setid.requests" 2>/dev/null exit 0 ;; esac done @@ -2398,6 +3749,36 @@ done exec "$real" "${args[@]}" EOF + # tar, for the same reason as cp. + # + # Git installs its translations with `tar ... | (cd /usr/share/locale && tar + # -x)`, and tar then tries to stamp the destination DIRECTORIES -- which + # belong to other packages and are sticky because we sealed them: + # tar: ./zh_TW: Cannot utime: Operation not permitted + # tar: ./zh_TW: Cannot change mode to rwxr-xr-t: Operation not permitted + # tar: Exiting with failure status due to previous errors + # 104 files had installed correctly. As with cp, the metadata complaint + # about a directory we do not own is harmless; anything else is not. + cat > "$WRAPPERS/tar" <<'EOF' +#!/bin/bash +real=/usr/bin/tar +err="$(mktemp)" +"$real" "$@" 2>"$err" +rc=$? +cat "$err" >&2 +if [ $rc -ne 0 ] && [ -s "$err" ]; then + # every complaint must be about metadata on something we do not own + if ! grep -vE "Cannot (utime|change mode|change ownership)|Exiting with failure status" "$err" \ + | grep -q .; then + echo "*** tar: could not stamp existing directories (harmless: the files extracted)" >&2 + rc=0 + fi +fi +rm -f "$err" +exit $rc +EOF + chmod 755 "$WRAPPERS/tar" + cat > "$WRAPPERS/cp" <<'EOF' #!/bin/bash # `cp -a dest/* /` is how several packages install a staged tree. As a package @@ -2451,7 +3832,7 @@ EOF soft "could not set $WRAPPERS to 0755 -- a package user may be able to rewrite the wrappers" \ -- chmod 0755 "$WRAPPERS" soft "could not give $WRAPPERS to root -- the wrappers are writable by whoever owns them now" \ - -- chown root:root "$WRAPPERS" "$WRAPPERS"/* + -- chown 0:0 "$WRAPPERS" "$WRAPPERS"/* } # Re-apply the install-directory permissions. @@ -2714,6 +4095,40 @@ NCLIST # neither pass should touch it. printf '%s\n' "${SNAP_ROOT%/}/var/mail" + # root's own home. Files appear there as side effects of root BEING IN A + # SHELL while a package builds -- .bash_history, .lesshst, .wget-hsts -- + # and the timestamp scan swept them into whichever manifest was open: + # [62/221] sqlite /root/.bash_history root -> p_sqlite + # --fix would have handed root's shell history to p_sqlite. Nothing in + # /root is installed by a package, and anything a cfg step writes there + # must stay root's anyway. (Both snapshot scans now skip /root and /home + # too; this line is what protects the manifests already polluted.) + printf '%s\n' "${SNAP_ROOT%/}/root" + + # Generated trust stores and our own configuration. + # + # A real verify found 711 wrong-owner claims of this species: p_libogg + # claiming every /etc/ssl/certs file, p_make-ca claiming /etc/pki + # anchors, p_docutils claiming /etc/pkgusr/stacks and + # /usr/share/lfs-pkgusr. None of them installed those files -- make-ca's + # detached background regeneration and `make install` of these tools ran + # DURING someone else's build, and the timestamp scan sweeps whatever is + # newer than the stamp. --fix would have chowned the system's certificate + # store to a package that has nothing to do with it. + # + # These are all regenerated-on-demand or ours: same class as + # /usr/share/info/dir and the state directory. + printf '%s\n' "${SNAP_ROOT%/}/etc/ssl/certs" "${SNAP_ROOT%/}/etc/pki" \ + "${SNAP_ROOT%/}/etc/pkgusr" "${SNAP_ROOT%/}/usr/share/lfs-pkgusr" \ + "${SNAP_ROOT%/}/tools" + + # The tools' own stores -- same reasoning as $STATE below: books the user + # downloaded, parse caches, tool stamps. Generated at runtime, regrown on + # demand, nobody's install: + # unclaimed /usr/share/blfs/books/BLFS-BOOK-13.0-systemd-nochunks.html + # unclaimed /usr/share/lfs/tool-stamps.json + printf '%s\n' "${SNAP_ROOT%/}/usr/share/lfs" "${SNAP_ROOT%/}/usr/share/blfs" + # The state directory itself. # # It moved under /usr/src in 1.9.0, and /usr/src IS an install directory -- @@ -2978,8 +4393,43 @@ _merge_stage() { # Same cause, different chapter. So before building X as user X, give X every # path recorded under X, X-tmp, X-pass1 and X-pass2 -- but only paths still # owned by root or the build user, never one belonging to another package. +# One stat and one chown for the WHOLE manifest, not per path. +# +# The loop this replaces forked three or four times per line +# (strip_host_prefix in a command substitution, stat, chown), which is +# invisible work: nothing is printed between "# user : p_gcc" and +# "# building gcc ...", so a package with big earlier stages looked HUNG for +# ten minutes with a bash process quietly burning a core. GCC is the worst +# case -- gcc-pass1, gcc-pass2 and libstdcpp all claim into the same user. +# +# Prints the paths that are eligible: they exist, and they still belong to +# whoever the build left them to (root, lfs, or a uid with no name) rather +# than to another package. +_claim_candidates() { # <owner> <manifest> ; paths on stdout + local owner="$1" man="$2" m="${LFS_HOST_MOUNT:-}" r="${SNAP_ROOT%/}" + local tmp; tmp="$(mktemp)" || return 0 + local p + while IFS= read -r p; do + [ -n "$p" ] || continue + # strip_host_prefix, inline: a command substitution per path is a fork + [ -n "$m" ] && [ "${p#$m/}" != "$p" ] && p="/${p#$m/}" + printf '%s\n' "$r$p" + done < "$man" > "$tmp" + # stat in batches: a path that does not exist simply produces no line + xargs -d '\n' -r -a "$tmp" stat -c '%U %n' -- 2>/dev/null | awk -v o="$owner" ' + { + u = $1 + i = index($0, " ") + path = substr($0, i + 1) + if (u == o) next + if (u != "root" && u != "lfs" && u != "UNKNOWN") next + print path + }' + rm -f "$tmp" +} + _claim_earlier_stages() { - local name="$1" owner="$2" man base n=0 p cur + local name="$1" owner="$2" man base n=0 p [ -n "$owner" ] || return 0 user_exists "$owner" || return 0 @@ -2998,45 +4448,51 @@ _claim_earlier_stages() { [ -n "$stem" ] || continue bases+=("$stem" "$stem-tmp" "$stem-pass1" "$stem-pass2") done + # GCC's first stage is not called gcc: chapter 5 builds Libstdc++ from + # GCC, and pkg_owner_name maps libstdcpp to this same user. Without it + # those files stayed root-owned and chapter 8 could not replace them. + case "$owname" in + *gcc) bases+=("libstdcpp" "libstdc++") ;; + esac # de-duplicate, so a manifest is not walked twice local seen="" b for b in "${bases[@]}"; do case " $seen " in *" $b "*) continue ;; esac seen="$seen $b" done + + # SAY IT STARTED, not only that it finished. This runs between + # "# user : p_gcc" and "# building gcc ...", and for a package with big + # earlier stages it is the only thing happening for a while. + detail "# claiming files from earlier stages of $owner ..." + local chownbin; chownbin="$(_real_tool chown)" + local batch; batch="$(mktemp)" || return 0 for base in $seen; do man="$MANIFESTS/$base.files" - [ -s "$man" ] || continue - while IFS= read -r p; do - [ -n "$p" ] || continue - p="$(strip_host_prefix "$p")" - [ -e "$p" ] || [ -L "$p" ] || continue - cur="$(stat -c %U "$p" 2>/dev/null)" || continue - [ "$cur" = "$owner" ] && continue - # only what the build itself left behind -- never another package's - case "$cur" in - root|lfs|UNKNOWN) ;; - *) continue ;; - esac - real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) - done < "$man" + if [ -s "$man" ]; then + _claim_candidates "$owner" "$man" > "$batch" + if [ -s "$batch" ]; then + n=$((n + $(count_stdin_lines < "$batch"))) + xargs -d '\n' -r -a "$batch" "$chownbin" -h "$owner:$owner" -- \ + 2>/dev/null || true + fi + fi # the directories too, or the package cannot create files in them man="$MANIFESTS/$base.dirs" [ -s "$man" ] || continue + : > "$batch" while IFS= read -r p; do - [ -n "$p" ] || continue - p="$(strip_host_prefix "$p")" [ -d "$p" ] || continue - cur="$(stat -c %U "$p" 2>/dev/null)" || continue - [ "$cur" = "$owner" ] && continue is_install_dir "$p" && continue # shared, leave it alone - case "$cur" in - root|lfs|UNKNOWN) ;; - *) continue ;; - esac - real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) - done < "$man" + printf '%s\n' "$p" >> "$batch" + done < <(_claim_candidates "$owner" "$man") + if [ -s "$batch" ]; then + n=$((n + $(count_stdin_lines < "$batch"))) + xargs -d '\n' -r -a "$batch" "$chownbin" -h "$owner:$owner" -- \ + 2>/dev/null || true + fi done + rm -f "$batch" [ "$n" -gt 0 ] && say "# gave '$owner' $n path(s) from its earlier build stages" return 0 } @@ -3095,16 +4551,319 @@ _restore_user_db() { _vfy_user_order 1 >/dev/null 2>&1 || true } +# --------------------------------------------------------------------------- # +# The runner -- ONE way to execute a phase as the package user (item 6, 3b) +# --------------------------------------------------------------------------- # +# +# Both drivers ran the install script, and spelled it differently: +# `cmd_build` used `su - "$owner" -c "env ... bash SCRIPT PHASE" | tee`, while +# packagemanager_install used `su - -c "set -o pipefail; PATH=...; bash +# ~/install_<user> ... | tee" <user>`. Same act, two spellings, and the +# differences were exactly where the bugs lived: which PATH the build sees +# (no wrappers => "install: cannot change permissions of '/usr/sbin'"), and +# whether the exit code came from the script or from tee (a failing build +# reporting success cost a day). +# +# Three privilege cases, and they are not interchangeable: +# * as root -- chapter-7 temp tools, and root config steps. NO +# wrappers: root legitimately chowns, and the wrapper +# would silently skip it. +# * as the owner -- the real package-user way, wrappers first in PATH. +# * root, then hand over -- `su` comes from Shadow (chapter 8), so for the +# first packages there is no way to drop privileges; the +# caller chowns what appeared afterwards. +# +# Returns the SCRIPT's status, never tee's. +# Variables the BOOK expects to be exported, kept in one file. +# +# Several pages need one (TEXLIVE_PREFIX, XORG_CONFIG, ...) and the build +# runs as a package user through `su -`, which drops the caller's +# environment -- so exporting them in your shell does not reach the build. +# `packagemanager env` writes them here; every phase run passes them through. +BUILD_ENV_FILE="${LFS_BUILD_ENV:-$ETC/pkgusr/build.env}" + +# A VALUE WITH SPACES IS ONE VALUE. +# +# These pairs are pasted into `env $envpass bash ...`, which the shell then +# splits on whitespace -- so XORG_CONFIG's flag list became several words and +# the second one was run as a command: +# env: '--disable-static': No such file or directory +# Each pair is single-quoted, with the one escape single quoting needs. +build_env_pairs() { + [ -r "$BUILD_ENV_FILE" ] || return 0 + local line name val + while IFS= read -r line; do + case "$line" in ''|\#*) continue ;; esac + case "$line" in *=*) ;; *) continue ;; esac + name="${line%%=*}" + val="${line#*=}" + printf "%s='%s' " "$name" "$(printf '%s' "$val" | sed "s/'/'\\\\''/g")" + done < "$BUILD_ENV_FILE" +} + +LAST_FAILED_PHASE="" # set by run_phase_as: which phase a failed + # `all` run stopped in (read by cmd_build's retry) + +create_service_accounts_from_script() { + # A book page that needs a system account says so in its own commands: + # groupadd -g 22 fcron && + # useradd -d /dev/null -c "Fcron User" -g fcron -s /bin/false -u 22 fcron + # Those run in the install phase, as the PACKAGE user, which cannot write + # /etc/group -- so fcron failed after a complete build with + # groupadd: cannot lock /etc/group; try again later + # Root is running this function; do it here, once, before the phase, and + # let the wrappers turn the package user's own attempt into a no-op. + local script="${1:-}" + [ -n "$script" ] && [ -f "$script" ] || return 0 + [ "$(id -u)" = 0 ] || return 0 + local line name + while IFS= read -r line; do + case "$line" in \#*) continue ;; esac + # the NAME is the last argument -- stripping "-x value" pairs by hand + # tripped over the quoted comment in + # useradd -d /dev/null -c "Fcron User" -g fcron ... fcron + # and produced 'User" fcron' + name="$(printf '%s' "${line%%&&*}" | awk '{print $NF}')" + [ -n "$name" ] || continue + case "$line" in + *groupadd*) + getent group "$name" >/dev/null 2>&1 && continue + say "# creating system group '$name' (the page asks for it; a package user cannot)" + eval "${line%%&&*}" >/dev/null 2>&1 || warn "could not create group '$name' -- the build will say so" ;; + *useradd*) + getent passwd "$name" >/dev/null 2>&1 && continue + say "# creating system user '$name' (the page asks for it; a package user cannot)" + eval "${line%%&&*}" >/dev/null 2>&1 || warn "could not create user '$name' -- the build will say so" ;; + esac + done <<EOF +$(grep -E "^[[:space:]]*(group|user)add[[:space:]]" "$script" 2>/dev/null) +EOF + return 0 +} + +report_setid_requests() { + # Printed after the phase, by root, because only root can act on it. + local owner="${1:-}" home f + [ -n "$owner" ] || return 0 + home="$(pkgusr_home_for "$owner" 2>/dev/null)" + f="$home/.pkgusr-setid.requests" + [ -s "$f" ] || return 0 + warn "" + warn "This package asked for setuid/setgid bits, which a package user" + warn "cannot set. Nothing else refuses them; the binaries are installed" + warn "and will misbehave without the bit (fcrontab, for one, cannot change" + warn "group and reports 'Operation not permitted' even when run by root)." + warn "Apply as root, if you want the package to work as upstream intends:" + while IFS= read -r line; do + [ -n "$line" ] && warn " $line" + done < "$f" + : > "$f" 2>/dev/null || true +} + +run_phase_as() { + local owner="$1" script="$2" phase="$3" log="$4" envpass="${5:-}" + LAST_FAILED_PHASE="" + envpass="$(build_env_pairs)$envpass" + local cwd="${6:-}" as_root="${7:-0}" track="${8:-}" rc=0 + # Phase tracking lives HERE (item 6, step 4), not in the callers. It was + # cmd_build's private bookkeeping, so a packagemanager install that died + # during `make install` left no record -- next_phase_for knew nothing, and + # "resume" meant starting over. The runner is the one place that KNOWS + # which phase ran and how it ended, whichever tool asked for it. + # + # $track is the record key (the step name). Empty = no tracking, and a + # script without the phased dispatcher is never tracked: `bash script + # unpack` on an old-style script runs everything, so a record of "unpack" + # would claim something the run did not do. + if [ -n "$track" ] && ! grep -q '^install_pkg() {' "$script" 2>/dev/null; then + track="" + fi + # A script written for the shared runner (format v18+) sources lfs-phases + # off $PATH or $PKGUSR_LIB. Say so here, in one place, instead of + # letting `su -` report "lfs-phases: No such file or directory" from + # inside the package user's shell. + if grep -q 'pkgusr_run "\$@"' "$script" 2>/dev/null; then + local _lib="${PKGUSR_LIB:-}" + [ -n "$_lib" ] || _lib="$(command -v lfs-phases 2>/dev/null || true)" + [ -n "$_lib" ] || [ ! -r "${SNAP_ROOT%/}/usr/bin/lfs-phases" ] \ + || _lib="${SNAP_ROOT%/}/usr/bin/lfs-phases" + if [ -z "$_lib" ] || [ ! -r "$_lib" ]; then + die "$script needs the phase runner 'lfs-phases', which is not on PATH -- install lfs-pkgusr (make install) or set PKGUSR_LIB=/path/to/lfs-phases" + fi + envpass="PKGUSR_LIB='$_lib' $envpass" + fi + if [ -n "$track" ]; then + case "$phase" in all|update) clear_phases "$track" ;; esac + fi + mkdir -p "$(dirname "$log")" 2>/dev/null || true + if [ "$as_root" = 1 ] || ! have_su || ! user_exists "$owner"; then + ( [ -n "$cwd" ] && cd "$cwd"; eval "env $envpass bash '$script' '$phase'" ) \ + 2>&1 | tee "$log" + rc=${PIPESTATUS[0]} + else + # `su -` resets the environment, so the vars go on the command line + su - "$owner" -c "env $envpass bash '$script' '$phase'" 2>&1 | tee "$log" + rc=${PIPESTATUS[0]} + fi + if [ -n "$track" ]; then + local _p + case "$phase" in + all|update) + if [ "$rc" = 0 ]; then + while read -r _p; do record_phase "$track" "$_p"; done \ + < <(script_phases_of_file "$script") + else + # The phases run in order and stop at the first failure, + # so everything before the failing one did succeed. + # Record that, or the next run wrongly reports + # unpack/build as "never completed". + local _failed + _failed="$(grep -oE "phase '[a-z]+' FAILED" "$log" 2>/dev/null \ + | head -n1 | cut -d"'" -f2)" + # ONE READER OF THE LOG. cmd_build needs to know which + # phase failed, so it can retry THAT one instead of the + # whole build (1.14.70). It must not grep for itself: + # phase bookkeeping lives here, and a second parser is a + # second thing to keep right. + LAST_FAILED_PHASE="$_failed" + if [ -n "$_failed" ]; then + for _p in unpack build install configure; do + [ "$_p" = "$_failed" ] && break + record_phase "$track" "$_p" + done + fi + fi ;; + *) [ "$rc" = 0 ] && record_phase "$track" "$phase" ;; + esac + fi + # A ZERO EXIT IS NOT PROOF THAT NOTHING FAILED. + # + # alsa-lib's install ended with + # install: cannot create regular file '/usr/include/sys/asoundlib.h': + # Permission denied + # make[4]: *** [Makefile:786: install-data-hook] Error 1 + # ... make: *** [Makefile:408: install-recursive] Error 1 + # and the run reported "alsa-lib: done. 1 package(s) installed." Whatever + # swallowed the status -- an older script format, a trailing command that + # succeeded -- the LOG said plainly that make had failed, and every + # "installed but not really" in this session began exactly here. So the + # log is checked as well as the status: cheap, and it cannot be bypassed by + # a script that forgets to propagate an exit code. + # THE LOG BELONGS BESIDE THE PACKAGE. + # + # Every failure box ends with "logs: /usr/src/pkgusr/p_<name>/log", and + # for a modern script that directory held one empty .diff: the phase log + # went to the central $LOGS/<name>-<phase>.log and nowhere else. The + # central copy is overwritten by the next package; the package's own + # history is what someone wants three days later, and the box had been + # pointing at an empty directory to find it. + if [ -n "$log" ] && [ -f "$log" ] && [ -n "$owner" ]; then + local _pkghome _pkglog + _pkghome="$(pkgusr_home_for "$owner" 2>/dev/null)" + if [ -n "$_pkghome" ] && [ -d "$_pkghome" ]; then + _pkglog="$_pkghome/log/$(basename "${log%.log}")-$(date +%Y-%m-%d_%H-%M-%S).log" + mkdir -p "$_pkghome/log" 2>/dev/null + if cp -f "$log" "$_pkglog" 2>/dev/null; then + chown "$owner:$owner" "$_pkglog" 2>/dev/null || true + # keep the last 10: a webkitgtk log is megabytes, and a + # package that fails twenty times should not fill the disk + ls -1t "$_pkghome/log"/*.log 2>/dev/null | tail -n +11 \ + | while IFS= read -r _old; do rm -f "$_old"; done + fi + fi + fi + if [ "$rc" = 0 ] && [ -n "$log" ] && [ -f "$log" ]; then + local _swallowed="" + _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^install: cannot create|: Permission denied$" \ + "$log" 2>/dev/null)" + if [ -n "$_swallowed" ]; then + warn "!! this phase exited 0, but its log says otherwise:" + warn " $_swallowed" + warn " Treating it as a FAILURE -- a package that could not write" + warn " its files is not installed, whatever the exit status said." + rc=1 + fi + fi + return "$rc" +} + +# `lfs-helper run-script` -- the same runner, for a caller that is not this +# script. packagemanager_install uses it instead of its own su line. +cmd_run_script() { + need_root + local owner="" script="" phase="all" log="" envpass="" cwd="" as_root=0 + local track="" _track_set=0 + while [ $# -gt 0 ]; do + case "$1" in + -h|--help) + cat <<'EOF' +lfs-helper run-script --user U --script S [--phase P] [--log F] [--env VARS] + [--cwd DIR] [--as-root] + +Runs one phase of an install script as the package user -- the ONE runner +both drivers use, so a build behaves the same whichever tool started it. + + --user the account to run as (package name or account name) + --script the install script to run + --phase all (default) | unpack | build | install | configure | test + --log where to tee the output (default: the account's log/ directory) + --name the step name to record phases under (default: derived from + --user; --name '' disables tracking) + --as-root run as root and WITHOUT the wrappers -- for root config steps, + which legitimately chown + +Phases that finish are RECORDED (item 6, step 4), so an install that dies +during `make install` can be resumed with `lfs-helper build <name> --phase +install` instead of starting over -- whichever tool started it. + +Reports the SCRIPT's exit status, never tee's. +EOF + return 0 ;; + --user) owner="${2:-}"; shift 2 ;; + --script) script="${2:-}"; shift 2 ;; + --phase) phase="${2:-}"; shift 2 ;; + --log) log="${2:-}"; shift 2 ;; + --env) envpass="${2:-}"; shift 2 ;; + --cwd) cwd="${2:-}"; shift 2 ;; + --name) track="${2-}"; _track_set=1; shift 2 ;; + --as-root) as_root=1; shift ;; + *) die "unknown option: $1 (usage: lfs-helper run-script --user U --script S [--phase P] [--log F])" ;; + esac + done + [ -n "$owner" ] || die "run-script: --user is required" + [ -n "$script" ] || die "run-script: --script is required" + [ -f "$script" ] || die "run-script: no such script: $script" + # The record key is the STEP name. The caller hands us the ACCOUNT + # (p_wget); cmd_build records under the step (wget), and two keys for one + # package is the disease this project keeps treating. pkg_owner_name + + # unprefix_pkg_user is the same round trip every other name makes. + if [ "${_track_set:-0}" != 1 ]; then + track="$(unprefix_pkg_user "$(pkg_owner_name "$owner")")" + fi + if [ -z "$log" ]; then + log="$(pkgusr_home_for "$owner")/log/run-$phase-$(date +%Y%m%d_%H%M%S).log" + fi + # wrappers first in PATH, unless this is a root step -- see run_phase_as + if [ -z "$envpass" ] && [ "$as_root" != 1 ]; then + envpass="PATH='$WRAPPERS:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin'" + fi + run_phase_as "$owner" "$script" "$phase" "$log" "$envpass" "$cwd" "$as_root" "$track" +} + + # Does this step rewrite the user database? _step_rewrites_user_db() { # any truncating redirect into a passwd or group file, wherever $ETC points grep -qE '>[[:space:]]*[^>|&]*/(passwd|group)([[:space:]]|$)' "$1" 2>/dev/null } -# The generated scripts carry "# package: <name>-<version>" in their header. +# The version, from the one header line every generator writes. This read +# a "# package:" comment no generator ever wrote, so every pkgusr.info +# version was empty -- while its caller read name_version= for the VERSION +# file twenty lines earlier. One fact, one parser. _version_from_script() { [ -r "${1:-}" ] || return 0 - sed -n 's/^# *package *: *//p' "$1" 2>/dev/null | head -1 \ + sed -n "s/^name_version=[\"']\(.*\)[\"']\$/\1/p" "$1" 2>/dev/null | head -1 \ | sed 's/^.*-\([0-9][0-9.]*\)$/\1/' } @@ -3238,21 +4997,41 @@ _package_already_owns_files() { cmd_build() { need_root local name="" phase="all" force=0 run_tests=0 jobs="" auto_fix=1 - local _force_stage=auto + local _force_stage=auto script_override="" as_root=0 while [ $# -gt 0 ]; do case "$1" in --phase) phase="$2"; shift 2 ;; + # The loop, for a script that is not in $SCRIPTS (item 6, step 5). + # packagemanager_install prepares the script in the package's home + # and used to run its own copy of everything below -- runner, + # grant-and-retry, recorder, version. Now it hands the file over + # and THIS is the one loop. + --script) script_override="${2:?--script needs a path}"; shift 2 ;; --force) force=1; shift ;; --stage) _force_stage=yes; shift ;; --no-stage) _force_stage=no; shift ;; --tests) run_tests=1; shift ;; --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; --no-auto-fix) auto_fix=0; shift ;; + # A CONFIGURATION STEP THAT NEEDS ROOT (1.14.79) told people to run + # lfs-helper build fcron --phase configure --force --as-root + # and cmd_build did not take --as-root: the flag existed on + # cmd_run_script, one function away, so the advice was confidently + # wrong. BLFS's "Configuring" sections are root's, so `build` has + # to accept it too. + --as-root) as_root=1; shift ;; -*) die "unknown option: $1" ;; *) name="$1"; shift ;; esac done - [ -n "$name" ] || die "usage: lfs-helper build <package> [--phase P] [--force]" + [ -n "$name" ] || die "usage: lfs-helper build <package> [--phase P] [--force] [--as-root]" + # Records are kept under the STEP name. Callers hand over a mixture -- + # build-all passes step names, packagemanager passes the ACCOUNT it just + # resolved -- and recording p_wget's build under "p_wget" while lfs + # records it under "wget" is two keys for one package, the disease this + # project keeps treating. Stripping the package prefix is safe on names + # that never carry it (gcc-pass1, cfg_bootscripts come back unchanged). + name="$(unprefix_pkg_user "$name")" # A step this tool performs itself, with no script in the book. # @@ -3279,25 +5058,92 @@ cmd_build() { return 0 fi - local s; s="$(script_for "$name")" + local s + if [ -n "$script_override" ]; then + s="$script_override" + [ -f "$s" ] || die "no such script: $s" + else + s="$(script_for "$name")" + fi + if [ ! -f "$s" ]; then + # NOT EVERY NAME IS AN LFS STEP. + # + # glib2 is a BLFS package: it has no $SCRIPTS/glib2.sh and never will. + # Its script lives in its own package user's home, which is where the + # failure box's own retry hint points -- but `lfs-helper build glib2` + # looked only at the step scripts, announced "no script for 'glib2'" + # and advised regenerating the CHROOT scripts, a command that does not + # even run in here. Advice that cannot work is worse than none. + local _own _ownhome _cand + _own="$(pkg_owner_name "$name" 2>/dev/null)" + if [ -n "$_own" ] && user_exists "$_own"; then + _ownhome="$(pkgusr_home_for "$_own")" + # the canonical install_<Name-Version>, newest first; install_last + # is the record of what ran and is the fallback + _cand="$(ls -1t "$_ownhome"/install_*-[0-9]* 2>/dev/null \ + | grep -v '\.edited$\|\.bak$' | head -1)" + [ -n "$_cand" ] || _cand="$(ls -1 "$_ownhome/install_last" 2>/dev/null)" + if [ -n "$_cand" ] && [ -f "$_cand" ]; then + say "# '$name' is not an LFS step; using its own script" + say "# $_cand" + s="$_cand" + fi + fi + fi if [ ! -f "$s" ]; then # A step in the order with no script and no built-in handler almost # always means the ORDER is older than the tools -- the step was # removed and the tree still lists it. Say that, rather than only # offering to generate scripts that may already be there. _warn_if_scripts_are_stale || true - die "no script for '$name' at $s - Either the step order is older than these tools, or the scripts were never - generated. From outside the chroot: + # ONE OF THREE THINGS, AND THEY NEED DIFFERENT ANSWERS. + # + # This assumed the name was an LFS step whose script was missing, and + # said "regenerate the chroot scripts" -- from inside the chroot, + # where that command refuses to run. A name can also be a BLFS + # package (its script lives in its package user's home) or simply a + # typo, and both were told to regenerate the wrong thing. + local _msg="no script for '$name' at $s" + if in_step_order "$name"; then + _msg="$_msg + '$name' IS in the step order, so its script was never generated or was + removed. From OUTSIDE the chroot (type 'exit' first): lfs build-system gen-chroot-scripts --run --overwrite If '$name' no longer exists in this version, clear it from the order: lfs-helper undone $name" + else + _msg="$_msg + '$name' is not an LFS step, and no package user of that name has a script + either. If it is a BLFS package, build it through packagemanager: + packagemanager install $name --run --recursive + or, to re-run one phase of something already unpacked: + packagemanager script install $name + If you meant an LFS step, see the list: + lfs-helper list" + fi + die "$_msg" fi # Clear a stale source tree first, as root -- independent of everything # below, and it must happen even if staging later has trouble. + # DO NOT THROW AWAY A BUILD THAT SUCCEEDED. + # + # `--phase all` cleans the source tree first, which is right for a fresh + # build and wrong for a RETRY: LLVM compiled all 4343 targets, failed on + # a `systemd-run` line at the very end, and the next run wiped the tree + # and compiled it again from scratch. The phase records already say what + # finished (1.11.55), so a retry that has 'build' on record keeps the + # tree and the phases pick up from there. + local _done_phases="" + _done_phases="$(cat "$(phase_file "$name")" 2>/dev/null | tr '\n' ' ')" case "$phase" in - all|unpack) clean_stale_source "$name" "$s" ;; + all|unpack|update) + case " $_done_phases " in + *" build "*) + say "# keeping the source tree: 'build' already finished here" + say "# (rebuild from scratch with: lfs-helper build $name --phase unpack --force)" ;; + *) clean_stale_source "$name" "$s" ;; + esac ;; esac if is_done "$name" && [ "$force" = 0 ]; then @@ -3315,13 +5161,26 @@ cmd_build() { # One derivation of the staging directory, held in a variable. The script # is installed there, the build cd's there, and the failure banner names # it -- five call sites that used to re-derive it, and so could disagree. - local as_root=0 staged stagedir owner="$name" + # NOT `local as_root=0`: that shadowed the --as-root the caller passed, + # so the flag parsed cleanly and then did nothing. A root STEP still sets + # it; an explicit --as-root is honoured as well. + local staged stagedir owner="$name" if is_root_step "$name"; then as_root=1 stagedir="$(step_stage_dir "$name")" staged="$stagedir/install_$name" mkdir -p "$stagedir" - install -m 0755 "$s" "$staged" || die "could not stage the install script" + # STAGING A FILE ONTO ITSELF IS NOT AN ERROR, IT IS A NO-OP. + # + # pm-install copies the script into the account's home and then the + # phase runner stages it again to the same name. That was invisible + # while the account's home was MISPLACED (the two paths differed); + # once fix-home repointed the home mid-run they became one file and + # install(1) refused: + # install: '.../install_p_nss' and '.../install_p_nss' are the same file + if ! [ "$s" -ef "$staged" ]; then + install -m 0755 "$s" "$staged" || die "could not stage the install script" + fi # A *-tmp temporary tool is the SAME package chapter 8 rebuilds, so its # files should belong to that package user from the outset -- otherwise # chapter 8 inherits a root-owned tree it cannot write into. The user @@ -3344,8 +5203,28 @@ cmd_build() { cmd_add_user "$owner" stagedir="$(pkgusr_home_for "$owner")" staged="$stagedir/install_$name" - install -m 0755 -o "$owner" -g "$owner" "$s" "$staged" \ - || die "could not stage the install script" + # ALREADY THERE IS ALREADY STAGED. + # + # A script that lives in the owner's own home needs no second copy + # under a second name. Copying it anyway is what produced + # script: install_p_hatchling + # staged: install_hatchling + # edit : install_Hatchling-1.28.0 + # -- three names for one file, and the failure box had to spend three + # lines telling the user which was which. + if [ "${s%/*}" = "${stagedir%/}" ] && [ -f "$s" ]; then + staged="$s" + soft "the install script is not owned by $owner" \ + -- real_chown "$owner:$owner" "$staged" + chmod 0755 "$staged" 2>/dev/null || true + elif ! [ "$s" -ef "$staged" ]; then + install -m 0755 -o "$owner" -g "$owner" "$s" "$staged" \ + || die "could not stage the install script" + else + # already in place: make sure it still belongs to the package user + soft "the staged script is not owned by $owner" \ + -- real_chown "$owner:$owner" "$staged" + fi fi mkstate @@ -3365,7 +5244,8 @@ cmd_build() { else detail "# user : $owner (package user) phase: $phase" fi - [ "$phase" = "all" ] && clear_phases "$name" + # Phase records are the runner's job now (item 6, step 4): run_phase_as + # clears on an "all" run and records what finished, for EVERY caller. # Record that this package is MID-INSTALL. Cancelling here (Ctrl-C during # `make install`) leaves the package half on disk: gcc, interrupted during # its install, replaces /usr/bin/gcc with a native compiler but has not yet @@ -3393,6 +5273,17 @@ cmd_build() { _protect_user_db fi + # This build's grant record starts EMPTY. The file is only cleared by + # the report at the end, so a build that failed (or any path that granted + # without reporting) left its entries behind -- and the next successful + # build printed them under "Collector groups used during this run". + # Building m4 announced that p_urllib3, make-ca and p11-kit had been + # granted access, which they had, weeks earlier. A record whose contents + # outlive the run it describes cannot answer the question it is asked. + mkdir -p "$(dirname "$GRANTED")" 2>/dev/null || true + : > "$GRANTED" 2>/dev/null || true + create_service_accounts_from_script "$staged" + : > "$(pkgusr_home_for "$owner" 2>/dev/null)/.pkgusr-setid.requests" 2>/dev/null || true say "# building $name ..." local rc=0 make_wrappers @@ -3456,7 +5347,16 @@ cmd_build() { soft "$_bldroot is not 755 -- the build may not be able to enter its own tree" \ -- real_chmod 755 "$_bldroot" - local envpass="LFS_CC_PHASE='$phase' BUILD_ROOT='$_bldroot'" + # A GENERIC NAME IN THE ENVIRONMENT IS A COLLISION WAITING TO HAPPEN. + # + # libvpx's build/make/Makefile says + # BUILD_ROOT?=. + # CFLAGS+=-I$(BUILD_PFX)$(BUILD_ROOT) -I$(SRC_PATH) + # and `?=` yields to the ENVIRONMENT -- so our BUILD_ROOT (the package's + # source directory) replaced libvpx's own "." and every compile looked + # for vpx_config.h in the wrong place. The variable is ours, so it gets + # our prefix; the scripts still see BUILD_ROOT, set from it locally. + local envpass="LFS_CC_PHASE='$phase' LFS_BUILD_ROOT='$_bldroot'" [ -n "$_stage" ] && envpass="$envpass DESTDIR='$_stage'" # The wrappers exist to stop a PACKAGE USER doing things it may not: chown, # setuid modes, re-moding shared directories. A ROOT step must not get @@ -3464,21 +5364,57 @@ cmd_build() { # to hand a source tree to a package user, and the wrapper silently skipped # it -- leaving the tree root-owned, so the build then failed with # ./configure: line 4389: config.log: Permission denied + # A FIXED PATH LOSES EVERYTHING INSTALLED UNDER /opt. + # + # This replaced PATH with a hard-coded list, so a package that lives in + # /opt -- rust, and everything the book puts there -- was invisible to + # every later build: + # install_p_cargo-c: line 161: cargo: command not found + # while `su - p_cargo-c -c 'command -v cargo'` found it perfectly, because + # a login shell reads /etc/profile.d and this did not. + # + # The extra directories are collected from the profile snippets the + # packages themselves installed (that is what they are for), appended + # after the fixed list so the wrappers and the system tools still come + # first. + # Read the snippets directly rather than starting another shell to ask: + # a login shell here would be slower, and would inherit whatever else the + # profile does. A profile.d snippet that adds a directory names it in a + # pathprepend/pathappend line; that is all we need. + local _extra="" _d + if [ -d "${SNAP_ROOT%/}/etc/profile.d" ]; then + for _d in $(grep -hoE 'path(prepend|append)[[:space:]]+(/[^[:space:]"]+)' \ + "${SNAP_ROOT%/}"/etc/profile.d/*.sh 2>/dev/null \ + | awk '{print $2}' | sort -u); do + case "$_d" in + /opt/*) [ -d "$_d" ] && _extra="${_extra:+$_extra:}$_d" ;; + esac + done + fi + local _base="/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin" + [ -n "$_extra" ] && _base="$_base:$_extra" if [ "$as_root" = 1 ]; then - envpass="$envpass PATH='/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin'" + envpass="$envpass PATH='$_base'" else # wrappers first in PATH -- see make_wrappers() for why - envpass="$envpass PATH='$WRAPPERS:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin'" + envpass="$envpass PATH='$WRAPPERS:$_base'" fi # the books expect some test failures, so suites only run when asked - [ "$run_tests" = 1 ] && envpass="$envpass LFS_RUN_TESTS=1" + [ "$run_tests" = 1 ] && envpass="$envpass PKGUSR_TESTS=1 LFS_RUN_TESTS=1" [ -n "${LFS_TGT:-}" ] && envpass="$envpass LFS_TGT='$LFS_TGT'" # MAKEFLAGS: --jobs wins over the configured value (it must be appended # LAST -- `env A=1 A=2` keeps the last one, so ordering decides). A package # that races under -j12 usually builds fine with --jobs 1. + # PRECEDENCE: --jobs, then build.env, then whatever this shell happens to + # have. The last assignment on an `env` line wins, and build.env is added + # FIRST, so passing the caller's MAKEFLAGS unconditionally overrode the + # per-machine setting with an ambient one -- a `MAKEFLAGS=-j16` inherited + # from the chroot's config beat the `MAKEFLAGS=-j13` someone had put in + # /etc/pkgusr/build.env, which is the file meant to hold exactly this. if [ -n "$jobs" ]; then envpass="$envpass MAKEFLAGS='-j$jobs'" - elif [ -n "${MAKEFLAGS:-}" ]; then + elif [ -n "${MAKEFLAGS:-}" ] \ + && ! printf '%s' "$envpass" | grep -q "MAKEFLAGS="; then envpass="$envpass MAKEFLAGS='$MAKEFLAGS'" fi # Keep pip out of the package user's ~/.local. PIP_USER=0 stops it @@ -3510,6 +5446,33 @@ cmd_build() { # mktemp: failed to create file via template '/tmp/tmp.XXXXXXXXXX' # and the empty path that follows silently disables file tracking for the # whole step. $STATE always exists -- it is where the scripts came from. + # WAIT FOR THE TREE TO GO QUIET FIRST. + # + # Attribution by timestamp rests on one assumption -- that nothing else + # writes to the tree during a build -- and a real run proved it false. + # make-ca (package 98) was still generating /etc/ssl/certs and + # /etc/pki/anchors at 19:45, after its own scan had closed, while libogg + # (99) was building; libogg's manifest claimed 400 certificates, and + # `verify --fix` would have chowned the system trust store to it. We + # cause part of it ourselves: mandb and record-install are detached with + # setsid so Ctrl-C cannot interrupt them, which puts them squarely inside + # the NEXT package's window. + # + # So the window does not open until the known writers are done. Bounded, + # because a hung mandb must not stop a build -- if it is still going, say + # so, since the manifest that follows may then over-claim. + # Share the indexes BEFORE the build, not after it fails. + # + # init-pkgusr does this too, but /usr/share/info/dir usually does not + # exist yet when the package-user system is set up: it is created by the + # first package that ships an info page, owned by THAT package user, and + # the next package to run install-info cannot replace it. On a fresh LFS + # base build that is a guaranteed failure partway through chapter 8 -- + # repaired by the retry loop, but a failure the user still has to watch. + # Idempotent and cheap, so doing it every build costs nothing and removes + # the failure entirely. + pkgusr_ready && share_shared_files + _settle_tree_before_stamp local stamp mkdir -p "$STATE/tmp" 2>/dev/null || true stamp="$STATE/tmp/stamp.$$" @@ -3518,20 +5481,18 @@ cmd_build() { if [ -z "$stamp" ] || [ ! -e "$stamp" ]; then warn "# cannot create a build stamp -- file tracking is off for this step" fi - if [ "$as_root" = 1 ]; then - ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} - elif have_su; then - # `su -` resets the environment, so the vars go on the command line - su - "$owner" -c "env $envpass bash '$staged' '$phase'" \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} - else + # ONE runner (item 6, step 3b). The three privilege cases live in + # run_phase_as now; what stays here is the decision that belongs to the + # BUILD: when there is no `su` yet, the files have to be handed over + # afterwards. + if [ "$as_root" != 1 ] && ! have_su; then detail "# no 'su' yet (Shadow not built) -- building as root, then" detail "# giving the installed files to '$owner'" chown_after=1 - ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} fi + run_phase_as "$owner" "$staged" "$phase" "$log" "$envpass" \ + "$stagedir" "$as_root" "$name" + rc=$? # Move the staged tree into place BEFORE looking at what changed, so the # file tracking sees the finished system. Only on success: a failed @@ -3607,6 +5568,7 @@ cmd_build() { # Scratch belongs to no package. The two snapshot scans above already # knew that; this one did not. Same rule, one more place. local _bld="${BUILD_ROOT:-$r/build}" + say "# scanning the tree for what this step touched (a fresh tree takes a moment) ..." touched_dirs="$(find "${r:-/}" -xdev -type d -newer "$stamp" \ -not -path "$SRCROOT/*" \ -not -path "$r/sources/*" -not -path "$_bld/*" \ @@ -3637,6 +5599,16 @@ cmd_build() { | grep -v '^$' | sort -u > "$mand.tmp" && mv "$mand.tmp" "$mand" say "# tracked $(wc -l < "$man") file(s) for '$name'" + # ...and the owner-scanned record too, once ownership makes it meaningful. + # One door: cmd_record_install decides which strategy applies (item 6). + # ON SUCCESS ONLY. A failed libxml2 run recorded its 4 half-built files + # as pkg.lst, the planner read pkg.lst as "installed", and every later + # plan skipped the package -- docbook then died on the xmlcatalog that + # was never installed. The manifest above is the record of what a step + # TOUCHED and rightly survives failure; pkg.lst and VERSION are the claim + # that it is INSTALLED, and a failed build may not make that claim. + [ "${rc:-1}" = 0 ] && { cmd_record_install "$name" >/dev/null 2>&1 || true; } + # A FAILED build must not take ownership of anything. # @@ -3712,8 +5684,20 @@ cmd_build() { # that stops because the USER must do something -- "mount your EFI System # Partition first" -- must never be answered by handing directories away. local _perm_ish=0 - grep -qiE "permission denied|operation not permitted|cannot (create|remove|change|touch)|couldn't copy|read-only file system" \ + grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system" \ "$log" 2>/dev/null && _perm_ish=1 + # "cannot make directory ...: No such file or directory" is not a + # permission problem and granting will never fix it -- but it IS + # automatically fixable, so it must reach the retry loop. + [ -n "$(missing_dirs_from_log "$log")" ] && _perm_ish=1 + # SAY WHY THERE WAS NO RETRY. cmake failed twice on the same missing + # directory with no attempt and no explanation, which left the only + # evidence -- "did the loop run?" -- invisible. A decision not to act is + # a decision, and it has to be reviewable. + if [ "$rc" != 0 ] && [ "$auto_fix" = 1 ] && [ "$_perm_ish" != 1 ]; then + detail "# no automatic repair attempted: the failure does not look like" + detail "# a permission or missing-directory problem" + fi # Grants are per-directory now (a collector group is permission to install # into a directory, not ownership of a tree), so a package installing into @@ -3731,19 +5715,82 @@ cmd_build() { break fi _last_need="$_need_now" - auto_grant_from_log "$log" "$owner" || break # nothing left to grant + # create what is missing FIRST: a directory that is not there cannot + # be granted, and one round often needs both + _fixed_something=0 + # a shared index the package must rewrite (install-info on + # /usr/share/info/dir) is not a grant problem: give it the shared + # treatment and let the retry proceed + if grep -qE "Operation not permitted for|Permission denied" "$log" 2>/dev/null; then + _before_share="$(stat -c %U:%G:%a "$(shared_files_list | head -n1)" 2>/dev/null)" + share_shared_files + [ "$_before_share" != "$(stat -c %U:%G:%a "$(shared_files_list | head -n1)" 2>/dev/null)" ] \ + && _fixed_something=1 + fi + create_missing_dirs_from_log "$log" "$owner" && _fixed_something=1 + # a file another package owns, which THIS package is installing: + # ownership follows the installer (granting cannot fix EPERM) + takeover_files_from_log "$log" "$owner" && _fixed_something=1 + auto_grant_from_log "$log" "$owner" && _fixed_something=1 + # ...and the other half of a meson install failure: a FILE owned by + # another package that this one is installing over (see + # meson_takeover_from_log). The directory grant does nothing for it. + meson_takeover_from_log "$log" "$owner" && _fixed_something=1 + [ "$_fixed_something" = 1 ] || break # nothing left to fix _round=$((_round + 1)) + # a package taking one file per round from the same other package is + # building a bundled copy of it; more rounds will not help + if [ -n "${LAST_SINGLE_HANDOVER:-}" ]; then + _same_owner_rounds=$((${_same_owner_rounds:-0} + 1)) + else + _same_owner_rounds=0 + fi + if [ "${_same_owner_rounds:-0}" -ge 3 ]; then + fail "!" + fail "! $name has now taken three files, one per round, from" + fail "! '$LAST_SINGLE_HANDOVER'. It is building a BUNDLED COPY of that" + fail "! package -- meson clones it as a subproject and installs its" + fail "! files over the real ones. Handing them over one at a time" + fail "! cannot end: stop the subproject instead." + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$(unprefix_pkg_user "$name")]" + fail "! wrap_mode = nodownload" + fail "! then: packagemanager install $(unprefix_pkg_user "$name") --run --regenerate" + break + fi + # RETRY THE PHASE THAT FAILED, NOT THE WHOLE BUILD. + # + # foot took seven rounds to collect its directory grants, and every + # round re-cloned the repository, re-ran meson and recompiled all 150 + # targets -- to redo an install that had failed on one completion + # file. Granting a directory cannot invalidate a compile. When a + # `--phase all` run failed in install or configure, the retry runs + # THAT phase; the sources and the build tree are untouched and still + # there, which is exactly why the grant is all that was missing. + local _retry_phase="$phase" + if [ "$phase" = all ]; then + case "${LAST_FAILED_PHASE:-}" in + install|configure) _retry_phase="$LAST_FAILED_PHASE" ;; + esac + fi say "" - say "# retrying $name (phase $phase), round $_round ..." - if [ "$as_root" = 1 ]; then - ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} - elif have_su; then - su - "$owner" -c "env $envpass bash '$staged' '$phase'" \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} + if [ "$_retry_phase" != "$phase" ]; then + say "# retrying $name (phase $_retry_phase only -- the build is done), round $_round ..." else - ( cd "$stagedir" && eval "env $envpass bash '$staged' '$phase'" ) \ - 2>&1 | tee "$log"; rc=${PIPESTATUS[0]} + say "# retrying $name (phase $_retry_phase), round $_round ..." + fi + # the same runner as the first attempt -- this was a third copy of + # the three-branch block, and a retry that ran the build differently + # from the attempt it is retrying is a bug waiting for its day + run_phase_as "$owner" "$staged" "$_retry_phase" "$log" "$envpass" \ + "$stagedir" "$as_root" "$name" + rc=$? + # `all` also runs configure after install; when only install was + # retried, that step still owes to be run once it succeeds + if [ "$rc" = 0 ] && [ "$phase" = all ] && [ "$_retry_phase" = install ]; then + run_phase_as "$owner" "$staged" configure "$log" "$envpass" \ + "$stagedir" "$as_root" "$name" + rc=$? fi after="$(snapshot)" done @@ -3751,27 +5798,17 @@ cmd_build() { warn "gave up after $_max_rounds permission grants -- something else is wrong" fi - # An 'all' run executes the phases in order and stops at the first failure, - # so everything before the failing phase did succeed. Record that, or the - # next run wrongly reports unpack/build as "never completed". - if [ "$phase" = "all" ]; then - local _failed _p - _failed="$(grep -oE "phase '[a-z]+' FAILED" "$log" 2>/dev/null \ - | head -n1 | cut -d"'" -f2)" - if [ -n "$_failed" ]; then - for _p in unpack build install configure; do - [ "$_p" = "$_failed" ] && break - record_phase "$name" "$_p" - done - fi - fi - # Exit 4 means "there was deliberately nothing to do" -- e.g. rEFInd is # already installed and already has an LFS entry. That is a success, and # the "installed no files" check below must not treat it as suspicious. local _nothing_to_do=0 if [ "$rc" = 4 ]; then _nothing_to_do=1; rc=0; fi + # Whatever happened, say what the chmod wrapper refused: only root can + # act on it, root is running this, and the consequence otherwise arrives + # much later as an unexplained "Operation not permitted". + report_setid_requests "$owner" + # Exit 3 means "this needs the user to do something", not a build error. # Report it plainly: no failure banner, no permission fixing, no retrying. if [ "$rc" = 3 ]; then @@ -3792,13 +5829,721 @@ cmd_build() { fail "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" fail "! $name FAILED (phase '$phase', exit $rc)" fail "! script: $s" - fail "! staged: $staged" + # only when it IS a second file; usually the script is the staged one + [ "$staged" = "$s" ] || fail "! staged: $staged" + # WHAT THE AUTOMATIC REPAIR SAW. cmake failed three times on the + # same missing directory; whether the repair even ran was invisible + # from the outside, and two sessions went into guessing. The box + # now states it: what the gate decided, and what the detector found. + # A SCRIPT OLDER THAN THE GENERATOR CARRIES THE OLD BUGS. + # + # packagemanager warns about this when it RESOLVES a script, at the + # top of a long plan; by the time a package fails, that line is far + # out of sight. The failure is where it matters, so say it here too. + local _sgen _cgen + _sgen="$(sed -n 's/^### generated by blfs \([^ ]*\).*/\1/p' "$s" 2>/dev/null | head -n1)" + _cgen="$(blfs --version 2>/dev/null | sed -n 's/^blfs \([^ ]*\).*/\1/p' | head -n1)" + if [ -n "$_sgen" ] && [ -n "$_cgen" ] && [ "$_sgen" != "$_cgen" ]; then + fail "!" + fail "! This script was generated by blfs $_sgen; the generator is now" + fail "! $_cgen. Fixes made since then are NOT in it -- including any" + fail "! for this very failure. Rebuild it from the book first:" + # The suggestion needs the ANCHOR (mesa), which is what + # `packagemanager install` takes -- not the account (p_mesa). + # unprefix_pkg_user does that from PKGUSR_PREFIX; it printed p_mesa + # anyway, which means the prefix was EMPTY here while packagemanager + # had it set. That is a config disagreement worth naming rather than + # papering over, so say so when it happens. + local _sugg; _sugg="$(unprefix_pkg_user "$name")" + case "$_sugg" in + p_*) fail "! packagemanager install ${_sugg#p_} --run --yes --regenerate" + fail "! (note: this tool read an EMPTY package-user prefix, but the" + fail "! account is '$name'. /usr/share/lfs/config.json and" + fail "! /etc/pkgusr/packagemanager.conf should agree.)" ;; + *) fail "! packagemanager install $_sugg --run --yes --regenerate" ;; + esac + fi + fail "! auto-repair: $([ "$auto_fix" = 1 ] && echo enabled || echo DISABLED)," \ + "gate: $([ "$_perm_ish" = 1 ] && echo "recognised" || echo "did not recognise this failure")," \ + "rounds run: ${_round:-0}" + local _md_now + _md_now="$(missing_dirs_from_log "$log" 2>/dev/null | tr '\n' ' ')" + if [ -n "$_md_now" ]; then + fail "! directories it could create: $_md_now" + fail "! (they are still missing, so creating them did not work --" + fail "! say so with the output above and this line)" + fi + # "Program 'x' not found" from meson/cmake usually means a DEPENDENCY + # that is recorded as installed but is not really there -- a record + # written by a pre-1.12.7 failed build still lies on disk, and the + # planner believes records. Name the missing program and the cure, + # or the user chases the wrong package. + # A COMPILE error is not ours to repair -- but the place to record the + # cure is, and it is easy to miss. cairo 1.18.4 declares its own + # static ctime_r when it thinks glibc lacks one, then collides with + # the real declaration; the answer is one line in machine.conf. + if grep -qE "^[^ ]+:[0-9]+:[0-9]+: error:" "$log" 2>/dev/null; then + fail "!" + fail "! This is a COMPILE error in the package's own source, not a" + fail "! permission or layout problem -- the tools cannot repair it." + fail "! If it needs a build flag, put it where it survives a" + fail "! regenerated script, per package:" + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$name]" + fail "! c_args = -DSOMETHING # merged into meson setup" + # A WARNING THAT IS ONLY FATAL BECAUSE OF -Werror. + # + # wlroots 0.19 against libinput 1.31: + # error: enumeration value 'LIBINPUT_SWITCH_KEYPAD_SLIDE' not + # handled in switch [-Werror=switch] + # cc1: all warnings being treated as errors + # The code is fine; a dependency grew a new enum value since it was + # written. Nothing can be "fixed" in the package, and the cure is + # always the same flag -- worth naming, since version skew like this + # is the normal state of a rolling book against pinned git sources. + if grep -q "all warnings being treated as errors" "$log" 2>/dev/null; then + local _werr + _werr="$(grep -oE "\[-Werror=[a-z0-9-]+\]" "$log" 2>/dev/null \ + | head -1 | tr -d '[]')" + fail "!" + fail "! This is a WARNING that -Werror turned into an error${_werr:+ ($_werr)}." + fail "! The package builds fine without it; the usual cause is a" + fail "! dependency that grew something new since this version was" + fail "! written. Turn -Werror off for this package:" + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$(unprefix_pkg_user "$name")]" + fail "! werror = false # meson" + if [ -n "$_werr" ]; then + fail "! c_args = -Wno-${_werr#-Werror=} # cmake or autotools" + else + fail "! c_args = -Wno-error # cmake or autotools" + fi + fail "! then: packagemanager install $(unprefix_pkg_user "$name") --run --regenerate" + fi + # THE COMMONEST COMPILE ERROR OF THIS DECADE. + # + # error: 'uint8_t' does not name a type + # is not a broken package: GCC 13 stopped including <cstdint> from + # other headers, so code written before that no longer compiles. The + # cure is one flag, and it is the same flag every time -- worth saying + # outright rather than leaving "put a build flag somewhere". + local _stdint + _stdint="$(sed -n "s/.*error: [\`'‘]\(u\?int[0-9_a-z]*_t\|uintptr_t\|intptr_t\|size_t\)['’\`].*does not name a type.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + if [ -n "$_stdint" ]; then + local _lang="c_args" _key + grep -qE "\.(cc|cpp|cxx|hpp|hh):" "$log" 2>/dev/null && _lang="cpp_args" + fail "!" + fail "! '$_stdint' does not name a type: this is the GCC 13+ change" + fail "! -- <cstdint> is no longer pulled in by other headers, and" + fail "! code written before it needs the include added. One flag," + fail "! and the same one every time:" + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$(unprefix_pkg_user "$name")]" + if [ "$_lang" = cpp_args ]; then + fail "! cpp_args = -include cstdint" + else + fail "! c_args = -include stdint.h" + fi + fail "! then: packagemanager install $(unprefix_pkg_user "$name") --run --regenerate" + fail "! (an autotools package takes the same thing as" + fail "! configure_args = CXXFLAGS=-include\\ cstdint)" + fi + # AN UNDECLARED SYMBOL USUALLY BELONGS TO SOMEONE ELSE. + # + # cairo stopped on FC_HINT_NONE, FC_RGBA_RGB and ten more, all of them + # fontconfig's -- a DEPENDENCY whose headers are missing or too old, + # not a bug in cairo. The box said "the tools cannot repair it" and + # left the person to work out whose symbols those were. We can look: + # if some installed header declares it, the include path is wrong; if + # none does, the package that should have provided it is not properly + # installed. + local _sym _owner_hdr + _sym="$(sed -n "s/.*error: ['\`‘]\([A-Za-z_][A-Za-z0-9_]*\)['\`’] undeclared.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + if [ -n "$_sym" ]; then + _owner_hdr="$(timeout 20 grep -rlm1 -w "$_sym" \ + "${SNAP_ROOT%/}/usr/include" 2>/dev/null | head -1)" + fail "!" + if [ -n "$_owner_hdr" ]; then + fail "! '$_sym' IS declared, in:" + fail "! $_owner_hdr" + fail "! so the header is installed but this build did not see" + fail "! it -- an include path or a missing .pc file." + # /usr/include/<pkg>/<file>.h names the package often enough + # to be worth offering; a bare /usr/include/<file>.h does not. + local _hpkg="${_owner_hdr#${SNAP_ROOT%/}/usr/include/}" + case "$_hpkg" in + */*) _hpkg="${_hpkg%%/*}" + fail "! pkg-config --cflags $_hpkg" + fail "! packagemanager info $_hpkg" ;; + esac + else + fail "! Nothing under /usr/include declares '$_sym', so the" + fail "! package that provides it is missing or out of date." + fail "! Find and rebuild it:" + fail "! packagemanager which-package $_sym" + fail "! grep -rl $_sym /usr/include # nothing = not installed" + fi + fi + fi + # "cannot find -lfoo" is the linker saying a DEPENDENCY did not leave + # its library behind -- the same species as "Program 'x' not found": + # a package the planner believes is installed, is not. Name the + # libraries and the cure, or the user goes looking inside the wrong + # package. + local _missing_libs="" _l _libname + while IFS= read -r _l; do + [ -n "$_l" ] || continue + _libname="lib${_l}" + # Only report the ones that really are absent. One glob per + # directory: `ls a* b* c*` fails when ANY pattern misses, so a + # library that IS installed in the first directory would still be + # reported missing because the other two do not have it. + # WHAT THE LINKER ACTUALLY NEEDS: libfoo.so (or libfoo.a), + # exactly. A glob for libfoo.so* matches libfoo.so.6.20.1 and + # calls the library present -- which is precisely the case that + # fails, since libfoo.so.N without libfoo.so is an install that + # never finished. + _found=0 + for _d in /usr/lib /usr/lib64 /lib /lib64 /usr/local/lib; do + if [ -e "${SNAP_ROOT%/}$_d/$_libname.so" ] \ + || [ -e "${SNAP_ROOT%/}$_d/$_libname.a" ]; then + _found=1; break + fi + done + [ "$_found" = 1 ] || case " $_missing_libs " in + *" $_l "*) ;; *) _missing_libs="$_missing_libs $_l" ;; + esac + done < <(grep -oE "cannot find -l[A-Za-z0-9_.+-]+" "$log" 2>/dev/null \ + | sed 's/^cannot find -l//' | sort -u) + if [ -n "$_missing_libs" ]; then + fail "!" + fail "! The linker could not find:$(for _l in $_missing_libs; do printf ' lib%s.so' "$_l"; done)" + # TWO DIFFERENT ILLNESSES, TWO DIFFERENT CURES. + # + # libfoo.so.N present, libfoo.so absent + # -> the package IS installed but its install did not finish: + # libfoo.so is the DEVELOPMENT symlink, made by the package's + # own `make install`. (libfoo.so.N can appear without it -- + # ldconfig creates the SONAME link from any library it finds.) + # Rebuilding the owner is the cure; its record is not lying. + # nothing at all + # -> a dependency recorded as installed that is not there. + local _l _owner _real + for _l in $_missing_libs; do + _real="" + for _d in /usr/lib /usr/lib64 /lib /lib64 /usr/local/lib; do + set -- "${SNAP_ROOT%/}$_d/lib$_l".so.* + [ -e "$1" ] && { _real="$1"; break; } + done + if [ -n "$_real" ]; then + # The SONAME link (libfoo.so.1) is made by ldconfig, under + # whatever account happened to run it -- it blamed p_libpaper + # for fontconfig. Follow it to the real file, and prefer the + # manifest's answer to any ownership at all. + # WHO OWNS THE FILE beats who CLAIMS it. Asking the + # manifests first told the user to rebuild cairo for a + # fontconfig library, because a failed cairo round had swept + # the file into its manifest. The file's owner is a fact; + # a manifest entry is a claim, and claims have been wrong + # before (see the over-claiming manifests of 1.12.13). + _owner="" + [ -L "$_real" ] && _real="$(dirname "$_real")/$(readlink "$_real")" + _owner="$(stat -c %U "$_real" 2>/dev/null)" + case "$_owner" in + root|UNKNOWN|"") + _owner="" + if [ -d "$MANIFESTS" ]; then + _owner="$(grep -lxF "$_real" "$MANIFESTS"/*.files 2>/dev/null \ + | head -n1 | xargs -r basename 2>/dev/null)" + _owner="${_owner%.files}" + fi ;; + esac + _owner="$(unprefix_pkg_user "$_owner")" + fail "! lib$_l: $(basename "$_real") exists, lib$_l.so does not." + fail "! The development symlink is made by the package's own" + fail "! install, so that install did not finish. Rebuild it:" + fail "! lfs-helper build ${_owner:-<pkg>} --phase all --force" + fail "! (from $(basename "$_real"); if that is the wrong" + fail "! package, ask: packagemanager search lib$_l)" + else + fail "! lib$_l: nothing on this system. A dependency recorded as" + fail "! installed is not -- the same species as a missing program:" + fail "! packagemanager search lib$_l" + fail "! lfs-helper verify # names every lying record" + fail "! rm /usr/src/pkgusr/p_<pkg>/pkg.lst /usr/src/pkgusr/p_<pkg>/VERSION" + fail "! packagemanager install <pkg> --run" + fi + done + fi + # THE WRONG BOOK, FOUND BY THE PACKAGE INSTEAD OF BY US. + # + # BlueZ stopped with + # configure: error: systemd system unit directory is required + # on a SysV system. Nothing is broken: the SYSTEMD book's page was + # used, and its configure line omits --disable-systemd because on a + # systemd machine there is nothing to disable. Every run already + # prints "no book matched 'stable-sysv'; using the cached + # BLFS-BOOK-...-systemd", and this is what that note costs when it is + # ignored -- so say it here, where it stopped, with both ways out. + if grep -qiE "systemd (system )?unit (directory|dir) is required|--enable-systemd|systemd.*is required" \ + "$log" 2>/dev/null \ + && ! command -v systemctl >/dev/null 2>&1; then + fail "!" + fail "! This is a SysV system, and the page that produced this" + fail "! script came from the SYSTEMD book -- which never passes" + fail "! --disable-systemd, because on a systemd machine there is" + fail "! nothing to disable. Get the right book:" + fail "! blfs fetch stable-sysv" + fail "! packagemanager install $(unprefix_pkg_user "$name") --run --regenerate" + fail "! That fixes every later package too, not just this one." + fail "! If this machine cannot reach the network, disable it per" + fail "! package instead:" + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$(unprefix_pkg_user "$name")]" + fail "! configure_args = --disable-systemd" + fi + # A PACKAGE THAT CHECKS ITS OWN FILE OWNERSHIP. + # + # fcron refuses to run unless its files belong to the accounts it + # expects: + # ERROR Conf file (/etc/fcron.conf) must be owned by root:fcron + # ERROR Could not chdir to /var/spool/fcron: Permission denied + # A package-user install cannot produce that: everything it writes is + # owned by p_fcron, and the chown wrapper skips the package's own + # attempts on purpose -- that skipping is the whole model. For a + # daemon that validates ownership before it will start, the last step + # is root's, and the package says exactly what it wants. + local _ownwant _ownfile + _ownwant="$(sed -n 's/.*must be owned by \([A-Za-z0-9_.:-]*\).*/\1/p' \ + "$log" 2>/dev/null | head -1)" + _ownfile="$(sed -n 's/.*Conf file (\([^)]*\)).*must be owned.*/\1/p' \ + "$log" 2>/dev/null | head -1)" + # ...and the SAME demand, one file at a time. fcron names only the + # file it tripped over: first /etc/fcron.conf, then /etc/fcron.allow, + # and /etc/fcron.deny would have been third. The package created a + # system group for itself (groupadd, in its own script), and every + # config file it installed under /etc wants root:<that group>. Say so + # once, for all of them, instead of one round per file. + if [ -z "$_ownwant" ] \ + && grep -qE "could not open [^:]*: Permission denied|Permission denied" \ + "$log" 2>/dev/null; then + local _svcgrp="" + _svcgrp="$(grep -E "^[[:space:]]*groupadd[[:space:]]" "$s" 2>/dev/null \ + | head -1 | sed 's/&&.*//' | awk '{print $NF}')" + if [ -n "$_svcgrp" ] && getent group "$_svcgrp" >/dev/null 2>&1; then + # from the MANIFEST, not from the log: the log names the one + # file it tripped over, and the package installed several + local _cfgs _mf + _mf="${SRCROOT:-/usr/src}/$(pkg_owner_name "$name")/pkg.lst" + _cfgs="$( { [ -f "$_mf" ] && grep -E "^/etc/" "$_mf" 2>/dev/null + grep -oE "/etc/[A-Za-z0-9_.+-]+" "$log" 2>/dev/null; } \ + | sort -u | while IFS= read -r _f; do + [ -f "$_f" ] || continue + [ "$(stat -c %G "$_f" 2>/dev/null)" = "$_svcgrp" ] && continue + printf '%s ' "$_f" + done)" + if [ -n "$_cfgs" ]; then + fail "!" + fail "! '$(unprefix_pkg_user "$name")' runs under the group '$_svcgrp'" + fail "! (its own script creates it), and reads these as that group:" + fail "! chown root:$_svcgrp $_cfgs" + fail "! chmod 640 $_cfgs" + fail "! It reports ONE file at a time, so fix them together" + fail "! or you will be back here for the next one." + fi + fi + fi + if [ -n "$_ownwant" ]; then + fail "!" + fail "! This package CHECKS who owns its files, and a package-user" + fail "! install cannot satisfy that: everything it wrote belongs to" + fail "! $(pkg_owner_name "$name"), and the chown wrapper skips the package's own" + fail "! chown on purpose. Set it as root, once:" + [ -n "$_ownfile" ] && fail "! chown $_ownwant $_ownfile" + local _spool + _spool="$(sed -n 's/.*Could not chdir to \([^:]*\).*/\1/p' \ + "$log" 2>/dev/null | head -1)" + [ -n "$_spool" ] && fail "! chown -R ${_ownwant#*:}:${_ownwant#*:} $_spool" + fail "! (the package's own page says which; these come from its" + fail "! error messages above)" + fail "! Then re-run this phase." + fi + # A CONFIGURATION STEP THAT NEEDS ROOT. + # + # fcron built, installed 74 files, and then its configure phase said + # must be privileged to use -u + # from `fcrontab -u systab`. BLFS's "Configuring" sections say "as + # the root user" -- they are system configuration, not part of + # installing a package, and no directory grant can help: the command + # itself refuses to run unprivileged. --as-root already exists for + # precisely this; say so. + if [ "${LAST_FAILED_PHASE:-$phase}" = configure ] \ + && grep -qiE "must be privileged|only root|Operation not permitted|are you root" \ + "$log" 2>/dev/null; then + fail "!" + fail "! This is the CONFIGURATION step, and it needs root: the" + fail "! book's \"Configuring\" section is written for the root user," + fail "! and no directory grant helps -- the command itself refuses" + fail "! to run unprivileged. The package is already installed;" + fail "! run just this step:" + fail "! lfs-helper build $(unprefix_pkg_user "$name") --phase configure --force --as-root" + fi + # A LIBRARY THAT WILL NOT LINK. + # + # alsa-utils found alsa-lib's headers and then: + # checking for snd_ctl_open in -lasound... no + # configure: error: No linkable libasound was found. + # Headers present but the library unlinkable is a different fault from + # a missing package, and the three ways it happens are told apart by + # what is on disk: no libfoo.so DEVELOPMENT symlink (the runtime + # .so.N is there but nothing to link against), no library at all, or + # both present but absent from the ldconfig cache. + local _lnklib="" + _lnklib="$(sed -n 's/.*No linkable lib\([A-Za-z0-9_+-]*\) was found.*/\1/p' \ + "$log" 2>/dev/null | head -1)" + [ -n "$_lnklib" ] || _lnklib="$(sed -n 's/.*cannot find -l\([A-Za-z0-9_+-]*\).*/\1/p' \ + "$log" 2>/dev/null | head -1)" + if [ -n "$_lnklib" ]; then + local _libdir="${SNAP_ROOT%/}/usr/lib" + local _dev="" _run="" + [ -e "$_libdir/lib$_lnklib.so" ] && _dev=1 + ls "$_libdir/lib$_lnklib".so.* >/dev/null 2>&1 && _run=1 + fail "!" + if [ -n "$_dev" ] && [ -n "$_run" ]; then + fail "! lib$_lnklib IS installed, and links should work." + fail "! The usual cause is a stale linker cache:" + fail "! ldconfig && ldconfig -p | grep $_lnklib" + fail "! If that lists it, look in the package's config.log for" + fail "! the REAL link error (a missing second library, or an" + fail "! ABI mismatch) -- configure only reports the last one." + elif [ -n "$_run" ]; then + fail "! lib$_lnklib.so.N is installed but lib$_lnklib.so is NOT --" + fail "! that symlink is what a build links against, and without" + fail "! it the library exists at run time and not at build time." + fail "! The package that owns it installed incompletely:" + fail "! ls -l $_libdir/lib$_lnklib.so*" + fail "! packagemanager which-package lib$_lnklib.so" + fail "! packagemanager install <that package> --run --reinstall" + else + # ASK THE MANIFESTS, AS 1.14.33 DOES FOR A MISSING PROGRAM. + # + # p_alsa-lib had a 1711-byte pkg.lst while /usr/lib had no + # libasound at all. "Nothing provides it" is true and stops + # one step short: SOMEBODY recorded installing it, and that + # package is the one to rebuild. + local _lclaim="" + _lclaim="$(timeout 20 grep -l -- "/lib$_lnklib\.so" \ + "${SRCROOT:-/usr/src}"/*/pkg.lst 2>/dev/null | head -1)" + if [ -n "$_lclaim" ]; then + local _lpkg; _lpkg="$(basename "$(dirname "$_lclaim")")" + fail "! No lib$_lnklib on this system -- but '$_lpkg'" + fail "! RECORDS having installed it, so its record is wrong:" + fail "! grep $_lnklib $_lclaim" + fail "! packagemanager install $(unprefix_pkg_user "$_lpkg") --run --reinstall" + else + fail "! Nothing here provides lib$_lnklib at all, and no" + fail "! package records having installed it:" + fail "! packagemanager which-package lib$_lnklib.so" + fail "! packagemanager search $_lnklib" + fi + fi + fi + # A DEPENDENCY THE BUILD SYSTEM COULD NOT FIND. + # + # meson says only + # ERROR: Dependency "libdrm_intel" not found, tried pkgconfig and cmake + # and there are three quite different reasons, which the .pc files on + # disk tell apart. mesa hit the third: libdrm WAS installed, but built + # before libpciaccess existed, so it has no libdrm_intel.pc and never + # will until it is rebuilt. The book only RECOMMENDS Xorg-Libraries + # for libdrm, so nothing had ordered them. + local _dep _pcdir _pcbase + _dep="$(sed -n 's/.*ERROR: Dependency "\([A-Za-z0-9_.+-]*\)" not found.*/\1/p' \ + "$log" 2>/dev/null | head -1)" + # meson has a SECOND wording, with single quotes, for a dependency the + # project marks required: + # ERROR: Dependency 'libtiff-4' is required but not found. + # gtk4 and gegl both stop with that one, and it matched none of the + # patterns here -- so the entire .pc diagnosis was skipped and only + # the fallback note (which was the wrong advice for gtk4) was printed. + [ -n "$_dep" ] || _dep="$(sed -n "s/.*ERROR: Dependency '\([A-Za-z0-9_.+-]*\)' is required but not found.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + # autoconf's own wording, which is NOT meson's and not pkg-config's: + # configure: error: Package requirements (libgvc >= 2.16) were not met: + # Package 'libgvc' not found + # vala stopped exactly there and matched neither pattern, so the whole + # diagnosis was skipped for every autotools package in the book. + [ -n "$_dep" ] || _dep="$(sed -n "s/.*No package '\([A-Za-z0-9_.+-]*\)' found.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + [ -n "$_dep" ] || _dep="$(sed -n "s/.*Package '\([A-Za-z0-9_.+-]*\)' not found.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + [ -n "$_dep" ] || _dep="$(sed -n 's/.*Package requirements (\([A-Za-z0-9_.+-]*\)[^)]*) were not met.*/\1/p' \ + "$log" 2>/dev/null | head -1)" + local _dep_provider="" + if [ -n "$_dep" ]; then + # WHICH PACKAGE OWNS THIS MODULE NAME. + # + # gtk4 asked for `libtiff-4`; that is libtiff's pkg-config module, + # and libtiff is a BLFS package installed on that system. A + # pkg-config name often carries an API version the package name + # does not (libtiff-4, libxml-2.0, ncursesw6), so try the name as + # given and then with a trailing version dropped. + local _dcand + for _dcand in "$_dep" "${_dep%-[0-9]*}" "${_dep%%[0-9]*}"; do + [ -n "$_dcand" ] || continue + if user_exists "$(pkg_owner_name "$_dcand")" 2>/dev/null; then + _dep_provider="$_dcand" + break + fi + done + # NOT INSTALLED IS NOT THE SAME AS NOT A PACKAGE. + # + # gtk4 wanted `libtiff-4` on a system with no p_libtiff at all, so + # the account check found nothing and the box said "nothing + # provides it -- packagemanager install libtiff-4", naming a + # package the book does not have, and then offered to vendor it. + # A trailing API version that strips to a DIFFERENT name is the + # signature of a pkg-config module belonging to a real package: + # libtiff-4 -> libtiff libxml-2.0 -> libxml + # so name that instead, and let the fallback note stay quiet. + local _dep_guess="" + if [ -z "$_dep_provider" ]; then + _dep_guess="${_dep%-[0-9]*}" + [ "$_dep_guess" = "$_dep" ] && _dep_guess="${_dep%%[0-9]*}" + [ "$_dep_guess" = "$_dep" ] && _dep_guess="" + fi + fail "!" + if [ -n "$_dep_provider" ]; then + fail "! '$_dep' is the pkg-config name of '$_dep_provider'," + fail "! which IS installed here. So this is a pkg-config" + fail "! problem, not a missing package:" + fail "! pkg-config --exists --print-errors '$_dep'" + fail "! ls /usr/lib/pkgconfig/$_dep.pc" + fail "! If the .pc is absent, $_dep_provider installed without it" + fail "! -- rebuild it:" + fail "! packagemanager install $_dep_provider --run --reinstall" + fi + _pcdir="$(timeout 10 find "${SNAP_ROOT%/}/usr/lib" "${SNAP_ROOT%/}/usr/share" \ + -maxdepth 3 -name "$_dep.pc" 2>/dev/null | head -1)" + if [ -n "$_pcdir" ]; then + fail "! '$_dep.pc' EXISTS, at $_pcdir --" + fail "! so the build did not look where it is. Check:" + fail "! pkg-config --variable pc_path pkg-config" + fail "! PKG_CONFIG_PATH in /etc/pkgusr/build.env" + else + # libdrm_intel -> libdrm: the provider is usually the name up + # to the first separator, and its .pc being present while the + # asked-for one is not is the tell. + _pcbase="${_dep%%[-_.]*}" + if [ -n "$_pcbase" ] && [ "$_pcbase" != "$_dep" ] \ + && timeout 10 find "${SNAP_ROOT%/}/usr/lib" -maxdepth 3 \ + -name "$_pcbase.pc" 2>/dev/null | grep -q .; then + fail "! '$_dep' is missing, but '$_pcbase' IS installed." + fail "! So $_pcbase was built WITHOUT the part that provides" + fail "! it -- usually because one of ITS dependencies was not" + fail "! installed at the time. Rebuild it, then this:" + fail "! packagemanager install $_pcbase --run --recursive --reinstall" + elif [ -n "$_dep_guess" ]; then + fail "! Nothing here provides '$_dep', and no package of" + fail "! that name exists -- it is the pkg-config module of" + fail "! '$_dep_guess', which is NOT installed:" + fail "! packagemanager install $_dep_guess --run --recursive" + fail "! (if that name is wrong: packagemanager search $_dep_guess)" + elif [ -z "$_dep_provider" ]; then + # (silent when the provider is known -- saying both + # "libtiff is installed" and "nothing provides it" in one + # box is worse than saying neither) + fail "! Nothing on this system provides '$_dep'." + fail "! packagemanager which-package $_dep" + fail "! packagemanager install $_dep --run --recursive" + fi + fi + fi + # A FALLBACK *WE* DISABLED. + # + # blfs adds --wrap-mode=nofallback to every meson setup, so a package + # cannot silently vendor a dependency (cairo once installed its own + # freetype that way). When the subproject has no BLFS page -- gegl + # needs libnsgif and poly2tri-c, which the book does not carry -- that + # protection is what breaks the build, and meson's message says only + # "Use of fallback dependencies is disabled", never who disabled it. + # ...and ONLY when nothing on this system could have supplied it. + # gtk4 wanted `libtiff-4`, which is the pkg-config module of libtiff -- + # a BLFS package, installed on that system. Telling the person to + # allow a vendored copy would have built a second libtiff inside gtk4 + # and hidden a pkg-config problem. `_dep_provider` (set above) is the + # package the book has for it; when there is one, the advice above is + # the right advice and this must stay quiet. + if grep -q "Use of fallback dependencies is disabled" "$log" 2>/dev/null \ + && [ -z "${_dep_provider:-}" ] && [ -z "${_dep_guess:-}" ]; then + local _fbdep + _fbdep="$(sed -n "s/.*ERROR: Dependency '\([A-Za-z0-9_.+-]*\)' is required but not found.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + fail "!" + fail "! The fallback that would have supplied${_fbdep:+ '$_fbdep'} is" + fail "! disabled BY US: blfs adds --wrap-mode=nofallback so a package" + fail "! cannot quietly install its own copy of a library another" + fail "! package owns. When the subproject has no BLFS page, that is" + fail "! the only way to build it. Allow it for this package:" + fail "! /etc/pkgusr/stacks/machine.conf" + fail "! [$(unprefix_pkg_user "$name")]" + fail "! wrap_mode = default" + fail "! then: packagemanager install $(unprefix_pkg_user "$name") --run --regenerate" + fi + # A FEATURE MISSING FROM AN INSTALLED PACKAGE. + # + # pango died on "No Cairo font backends found" with cairo installed + # and working -- because cairo had been built BEFORE freetype2 + # existed (an old ordering) and silently has no ft backend. The + # package is fine; the feature is not. Nothing detects that, so say + # what it looks like when a `<pkg>-<feature> found: NO` sits next to + # a `<pkg> found: YES`. + local _featpkg + _featpkg="$(grep -oE "dependency [a-z0-9]+-[a-z0-9]+ found: NO" "$log" 2>/dev/null \ + | head -n1 | awk '{print $2}' | cut -d- -f1)" + if [ -n "$_featpkg" ] && \ + grep -qE "dependency ${_featpkg} found: YES" "$log" 2>/dev/null; then + fail "!" + fail "! '$_featpkg' is installed, but the feature this package needs" + fail "! is missing from it -- it was probably built before its own" + fail "! optional dependency existed. Rebuild it, then retry:" + fail "! packagemanager install $_featpkg --run --yes --reinstall" + fi + # A VARIABLE THE BUILD ASKED FOR. The generated script stops with + # "this package needs NAME to be set", which says WHAT but not HOW -- + # and the answer is one command, so print it. + local _envname + _envname="$(grep -oE "needs [A-Z][A-Z0-9_]{2,} to be set" "$log" 2>/dev/null \ + | head -n1 | awk '{print $2}')" + if [ -n "$_envname" ]; then + fail "!" + fail "! $_envname is not set for builds. Set it once and it applies to" + fail "! every package (builds run as a package user, so exporting it in" + fail "! your shell does not reach them):" + fail "! packagemanager env --set $_envname=... # or: packagemanager env --ask" + fi + # A PYTHON MODULE THE BUILD NEEDED. + # + # pip prints a whole traceback and ends with "metadata-generation- + # failed", and the log scanner's first-errors picked exactly the two + # lines that say nothing. The name is in the traceback: + # ModuleNotFoundError: No module named 'pathspec' + # and it is almost always a BLFS package of the same name. + local _pymod + _pymod="$(sed -n "s/.*ModuleNotFoundError: No module named '\([A-Za-z0-9_.-]*\)'.*/\1/p" \ + "$log" 2>/dev/null | head -1)" + if [ -n "$_pymod" ]; then + fail "!" + fail "! The build needs the Python module '$_pymod', which is not" + fail "! installed for the Python this system runs." + fail "! packagemanager install $_pymod --run --recursive" + fail "! If that says it is already installed, it was built for an" + fail "! older Python -- check with:" + fail "! python3 -c 'import $_pymod'" + fail "! packagemanager info $_pymod" + fi + local _missing_prog + _missing_prog="$(grep -oE "Program '[A-Za-z0-9._+-]+' not found" "$log" 2>/dev/null \ + | head -n1 | cut -d"'" -f2)" + if [ -n "$_missing_prog" ] && ! command -v "$_missing_prog" >/dev/null 2>&1; then + fail "!" + fail "! '$_missing_prog' does not exist on this system." + # THE MANIFESTS KNOW WHO CLAIMED IT. + # + # This said "find the package, purge its record, rebuild it" and left + # the person to find it -- when every installed package has a pkg.lst + # naming the files it installed, and a grep answers in a second. Two + # quite different situations hide behind one missing program: + # somebody's record CLAIMS the file (the record is wrong), or nobody + # does (the package was never installed). + local _claimer="" + # ${SRCROOT:-} -- a failure report runs under `set -u` and must not + # add a second failure to the one it is describing + _claimer="$(timeout 20 grep -l -- "/$_missing_prog\$" \ + "${SRCROOT:-/usr/src}"/*/pkg.lst 2>/dev/null | head -1)" + if [ -n "$_claimer" ]; then + local _cpkg; _cpkg="$(basename "$(dirname "$_claimer")")" + fail "! But '$_cpkg' RECORDS having installed it, so its record is" + fail "! wrong -- the build failed after the record was written, or" + fail "! the file was removed since. Rebuild it:" + fail "! packagemanager install $(unprefix_pkg_user "$_cpkg") --run --reinstall" + else + fail "! No installed package records that file, so whatever" + fail "! provides it was never installed here. Find it:" + fail "! packagemanager search $_missing_prog" + fail "! packagemanager install <pkg> --run --recursive" + fi + fi + # THE RETRY HINT MUST BE A COMMAND THAT WORKS. + # + # `lfs-helper build` takes the ACCOUNT; the box printed whatever name + # it was called with, and for a BLFS package that is the anchor: + # lfs-helper build gst10-plugins-bad --phase all --force + # !! no script for 'gst10-plugins-bad' at .../scripts/...sh + # -- so it fell through to the LFS base-step path and blamed step + # scripts from 1.12.30, which have nothing to do with the failure. + # Five rounds of debugging ran past this because the hint looked + # plausible. + # + # RESOLVED HERE, ONCE, FOR EVERY HINT BELOW. It used to be computed + # inside the permission-failure branch and used by three hints outside + # it, so any OTHER kind of failure ended the report with + # /usr/bin/lfs-helper: line 5596: _retry: unbound variable + # -- the error reporter crashing on the one line that tells you how to + # retry, in the middle of reporting someone else's error. + local _retry="$name" + if ! id "$_retry" >/dev/null 2>&1; then + local _acctname; _acctname="$(pkg_owner_name "$name" 2>/dev/null)" + id "$_acctname" >/dev/null 2>&1 && _retry="$_acctname" + fi + # A MISSING COMMAND IS A MISSING PACKAGE. + # + # install_libheif: line 33: cmake: command not found + # ! auto-repair: gate: did not recognise this failure + # -- as recognisable a failure as there is, reported as unrecognised. + # The BOOK does not always list the tool: libheif's page uses cmake in + # its commands and lists no Required section at all, so no resolver + # can plan it from the dependency lists. The log says it outright. + # A SHELL WORD IS NOT A PROGRAM. + # + # enchant's configure has a broken probe that runs the word `not`: + # ./configure: line 26491: not: command not found + # and the box announced "'not' is not installed -- the build needs it", + # sending the reader to `packagemanager search not`. The line was also + # harmless -- configure carried on -- while the REAL failure was fifty + # lines further down. Shell keywords and a handful of English words + # are never packages; skip them and take the next candidate. + local _missing_cmd="" + local _cand + while IFS= read -r _cand; do + case "$_cand" in + not|then|else|fi|do|done|elif|esac|in|the|and|or|is|if|"") continue ;; + esac + _missing_cmd="$_cand" + break + done <<EOF +$(sed -n 's/.*: \([a-zA-Z0-9_.+-]*\): command not found.*/\1/p' "$log" 2>/dev/null) +EOF + if [ -n "$_missing_cmd" ]; then + local _prov; _prov="$(_package_providing "$_missing_cmd")" + fail "! '$_missing_cmd' is not installed -- the build needs it and" + fail "! the book's page for $name does not list it as a dependency." + if [ -n "$_prov" ]; then + fail "! packagemanager install $_prov --run --recursive" + else + fail "! find what provides it: packagemanager search $_missing_cmd" + fi + fail "! Then retry:" + fail "! lfs-helper build $_retry --phase $phase --force" + fi if grep -qiE "permission denied|couldn't copy|cannot create|cannot remove" \ "$log" 2>/dev/null; then fail "! The package could not write into a directory owned by another" fail "! package. Grant access (collector groups) and retry:" fail "! lfs-helper fix-perms $name --phase $phase --run" - fail "! lfs-helper build $name --phase $phase --force" + fail "! lfs-helper build $_retry --phase $phase --force" fi # A later phase often fails because an EARLIER one never finished -- # gcc's `mv /usr/lib/*gdb.py` finds nothing when `make install` aborted @@ -3820,12 +6565,12 @@ cmd_build() { fi fi fail "! Edit the script, then retry just this phase:" - fail "! lfs-helper build $name --phase $phase --force" + fail "! lfs-helper build $_retry --phase $phase --force" if [ -z "$jobs" ] && [ "$phase" != "install" ]; then fail "! If the output ends with no obvious error, a parallel build may" fail "! have raced -- the real message is far above the tail. Retry" fail "! serially (slower, but the error appears where it happens):" - fail "! lfs-helper build $name --phase $phase --force --jobs 1" + fail "! lfs-helper build $_retry --phase $phase --force --jobs 1" fi fail "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" fail "" @@ -3926,7 +6671,7 @@ cmd_build() { real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true fi - if [ "$phase" = "all" ]; then + if [ "$phase" = "all" ] || [ "$phase" = "update" ]; then _write_pkgusr_info "$name" "$owner" "${LFS_PKG_SOURCE:-lfs}" \ "$(_version_from_script "$staged")" mark_done "$name" @@ -3935,7 +6680,7 @@ cmd_build() { trap - INT TERM _clear_interrupted "$name" else - record_phase "$name" "$phase" + # the runner recorded the phase; what stays here is the guidance local nxt; nxt="$(next_phase_for "$name" || true)" if [ -n "$nxt" ]; then ok "# $name: phase '$phase' finished" @@ -3967,7 +6712,7 @@ report_granted_groups() { local n; n=$(count_lines "$GRANTED") [ "$n" -gt 0 ] || return 0 say "" - note "Collector groups used during this run:" + note "Collector groups this build needed:" local grp user owner while IFS='|' read -r grp user owner; do [ -n "$grp" ] || continue @@ -4151,6 +6896,43 @@ cmd_grant_dir() { grant_dir_access "$dir" "$user" } +# A collector group created under the wrong name. +# +# The prompt asks for a group name and takes what is typed; someone answering +# it wrote `nimnu_gdb` for `nimgnu_gdb`, and a valid-but-wrong group owned the +# directory from then on -- invisible to everything that looks for +# `${COLLECTOR_PREFIX}_*`. Renaming is the whole repair: the GID does not +# change, so every directory already handed to it and every member still +# apply. 1.14.19 makes the typo far less likely; this fixes the ones already +# made. +cmd_rename_group() { + need_root + local old="${1:?usage: lfs-helper rename-group <old> <new> [--run]}" + local new="${2:?usage: lfs-helper rename-group <old> <new> [--run]}" + shift 2 + local run=0 + [ "${1:-}" = "--run" ] && run=1 + + group_exists "$old" || die "no such group: $old" + group_exists "$new" && die "'$new' already exists -- pick another name" + local gid dirs + gid="$(getent group "$old" | cut -d: -f3)" + say "$old (gid $gid) -> $new" + dirs="$(find "${SNAP_ROOT%/}/" -xdev -group "$old" -type d 2>/dev/null | head -20)" + if [ -n "$dirs" ]; then + say " directories it owns (gid is unchanged, so these stay granted):" + printf ' %s\n' $dirs + fi + local members; members="$(getent group "$old" | cut -d: -f4)" + [ -n "$members" ] && say " members: $members" + if [ "$run" != 1 ]; then + warn "(dry run -- nothing was changed. Add --run to rename.)" + return 0 + fi + "$(_real_tool groupmod)" -n "$new" "$old" || die "could not rename $old" + ok "renamed: $old -> $new" +} + # Which package installed this path? # # The manifests are the record of who installed what, so a file that should @@ -4190,8 +6972,35 @@ cmd_which_package() { if [ "$found" = 0 ]; then say " (none -- nothing recorded installing it)" say "" - say " Nothing recorded means it was installed before file tracking" - say " existed, or outside these tools entirely." + # TWO REGISTRIES, ONE QUESTION. These manifests cover the BUILD + # steps, chapters 5-9. Anything installed afterwards is recorded + # in the package user's own pkg.lst, which this never looked at -- + # so a package installed by these tools, ten minutes ago, was + # reported as "installed outside these tools entirely". + _wp_hit="" + for _r in "$PKGUSR_ROOT" "$CFGUSR_ROOT"; do + [ -d "$_r" ] || continue + for _l in "$_r"/*/pkg.lst; do + [ -f "$_l" ] || continue + if grep -qxF "$target" "$_l" 2>/dev/null; then + _wp_hit="$(basename "$(dirname "$_l")")" + say " Recorded in a package user's own list, not a build" + say " manifest: $_wp_hit ($_l)" + say " That is a package installed after the build, by" + say " packagemanager. Ask it for the detail:" + say " packagemanager which-package $target" + break + fi + done + [ -n "$_wp_hit" ] && break + done + if [ -z "$_wp_hit" ]; then + say " Nothing recorded means it was installed before file tracking" + say " existed, or outside these tools entirely." + say " Packages installed after the build record themselves" + say " separately; check there too:" + say " packagemanager which-package $target" + fi fi fi @@ -4604,10 +7413,18 @@ cmd_seal_install_dirs() { say "earlier will break installs that legitimately overwrite older files." say "" - local d n=0 missing=0 + local d n=0 missing=0 _shared_dirs + # A directory holding a shared index must never be sealed: sticky means + # only a file's owner may replace it, and every package replaces that + # index (install-info on /usr/share/info/dir). + _shared_dirs="|$(shared_file_dirs | tr '\n' '|')" while IFS= read -r d; do [ -n "$d" ] || continue [ -d "$d" ] || { missing=$((missing+1)); continue; } + case "$_shared_dirs" in + *"|$d|"*) say "# not sealing $d -- it holds a shared index every package rewrites" + continue ;; + esac [ -k "$d" ] && continue if [ "$run" = 1 ]; then real_chmod o+t "$d" 2>/dev/null && n=$((n+1)) @@ -4696,7 +7513,7 @@ cmd_adopt_existing() { } _vfy_manifest_ownership() { - local fix="$1" man base name owner p cur kind + local fix="$1" only="${2:-}" man base name owner p cur kind # In BUILD order, not alphabetical. # # This pass creates the package users it finds missing, and a user's uid is @@ -4706,6 +7523,11 @@ _vfy_manifest_ownership() { # useless for reading the build order off the passwd file. local _mans _nman=0 _ntot _mans="$(_manifests_in_build_order) $(_manifests_in_build_order dirs)" + # ...or just the one the caller named + if [ -n "$only" ]; then + _mans="$(printf '%s\n' $_mans | grep -E "/(${only}|$(pkg_owner_name "$only"))[./]" || true)" + [ -n "$_mans" ] || warn "no manifest found for '$only' -- nothing to check" + fi _ntot="$(printf '%s\n' $_mans | grep -c .)" for man in $_mans; do [ -e "$man" ] || continue @@ -4811,7 +7633,8 @@ _vfy_manifest_ownership() { # gcc:gcc 755, losing the install group and g+w -- the entire scheme, undone # by one package. Checked every run now. _vfy_install_dirs() { - local fix="$1" d live owner grp mode want_mode sealed + local fix="$1" d live owner grp mode want_mode sealed _shared_dirs + _shared_dirs="|$(shared_file_dirs | tr '\n' '|')" # Before the ownership epoch there is no user database, so `install` is not # a name and neither is root. Reporting 27 wrong install directories then # is not a finding -- it is the tree being younger than the question. @@ -4836,6 +7659,12 @@ _vfy_install_dirs() { # sticky only after the build is finished -- during it, package users # must be able to replace the temporary system's root-owned files if [ "$sealed" = 1 ]; then want_mode=1775; else want_mode=775; fi + # ...and NEVER for a directory holding a shared index: sticky means + # only a file's owner may replace it, and the whole point of that + # index is that every package replaces it (install-info). + case "$_shared_dirs" in + *"|$live|"*) want_mode=775 ;; + esac if [ "$owner" != 0 ] || [ "$grp" != "$INSTALL_GID" ] \ || [ "$mode" != "$want_mode" ]; then _vfy_report installdir "$live" \ @@ -5085,6 +7914,83 @@ _vfy_user_homes() { done } +# --- rule 6b: every package account's home is where the tools look --------- # +# +# Rule 6 checks the COMPUTED home exists and is owned right; it never asks +# whether the passwd entry agrees, or whether add_package_user's husk is +# sitting beside the roots. A bootstrap on a fresh tree left ten of those +# (/usr/src/p_wget, /usr/src/p_requests, ...) and verify walked past them. +_vfy_misplaced_homes() { + local fix="$1" acct uid rest bad=0 match + # Prefixes are how a package account is recognised; with none configured + # there is no safe way to tell one from a real user. + [ -n "$PKGUSR_PREFIX$CFGUSR_PREFIX" ] || return 0 + while IFS=: read -r acct _ uid rest; do + case "$uid" in ''|*[!0-9]*) continue ;; esac + [ "$uid" -ge "$PKG_UID_MIN" ] || continue + match=0 + [ -n "$PKGUSR_PREFIX" ] && case "$acct" in "$PKGUSR_PREFIX"?*) match=1 ;; esac + [ -n "$CFGUSR_PREFIX" ] && case "$acct" in "$CFGUSR_PREFIX"?*) match=1 ;; esac + [ "$match" = 1 ] || continue + _repair_misplaced_home "$acct" "$fix" || bad=$((bad+1)) + done < <(if [ -r "$ETC/passwd" ]; then cat "$ETC/passwd"; else getent passwd; fi) + if [ "$bad" -gt 0 ]; then + [ "$fix" = 1 ] \ + || warn " repair with: lfs-helper verify --fix (one package: lfs-helper fix-home <name> --run)" + return 1 + fi + return 0 +} + +# --- rule 3c: a record must describe reality ------------------------------- # +# +# pkg.lst and VERSION are the claim "this package IS installed", and the +# planner believes them. Records written by pre-1.12.7 failed builds lie: +# libxml2 claimed 4 half-built files, the planner skipped it, and three +# packages later shared-mime-info died looking for xmllint. This pass reads +# every pkg.lst and counts how many of its paths exist -- a record whose +# files are mostly gone is named a LIAR, with the purge-and-rebuild cure. +# Report-only: rebuilding is a build decision, not a repair. +_vfy_lying_records() { + local fix="${1:-0}" home lst total missing pct acct pkg bad=0 p + for home in "$PKGUSR_ROOT"/*/; do + [ -d "$home" ] || continue + lst="$home/pkg.lst" + [ -s "$lst" ] || continue + total=0; missing=0 + while IFS= read -r p; do + case "$p" in ""|\#*) continue ;; esac + total=$((total+1)) + [ -e "$p" ] || [ -L "$p" ] || missing=$((missing+1)) + done < "$lst" + [ "$total" -gt 0 ] || continue + pct=$(( missing * 100 / total )) + if [ "$pct" -ge 50 ]; then + bad=$((bad+1)) + acct="$(basename "$home")" + pkg="$(unprefix_pkg_user "$acct")" + warn "lying record: $acct claims $total path(s), $missing missing (${pct}%)" + if [ "$fix" = 1 ]; then + # DELETING A LIE IS A REPAIR. The record says the package is + # installed and it is not; while it stands, the planner skips + # the package and every dependent fails somewhere else + # entirely (xmllint, glib, -lfreetype). Rebuilding is a build + # decision and stays the user's; removing the false claim is + # not. + rm -f "$home/pkg.lst" "$home/VERSION" 2>/dev/null + ok "# removed the false record -- the planner will build it again" + warn " rebuild it when convenient: packagemanager install $pkg --run" + _vfy_n_fixed=$((_vfy_n_fixed + 1)) + else + warn " the planner believes this and will SKIP the package. Cure:" + warn " rm $home/pkg.lst $home/VERSION" + warn " packagemanager install $pkg --run" + fi + fi + done + [ "$bad" = 0 ] +} + _vfy_seal_install_dirs() { local d live n=0 while IFS= read -r d; do @@ -5369,13 +8275,31 @@ _vfy_unclaimed() { cmd_verify() { need_root - local fix=0 - [ "${1:-}" = "--fix" ] && fix=1 - [ "${1:-}" = "--run" ] && fix=1 + # ONE PACKAGE, IF ONE IS NAMED. + # + # `lfs-helper verify xorg7-lib --fix` read only $1 for the flag, ignored + # the name, and began walking all 533 packages -- minutes of work to + # answer a question about one of them. + local fix=0 only="" a + for a in "$@"; do + case "$a" in + --fix|--run) fix=1 ;; + -*) ;; + *) [ -z "$only" ] && only="$a" ;; + esac + done + if [ -n "$only" ]; then + local _acct; _acct="$(pkg_owner_name "$only")" + if ! id "$_acct" >/dev/null 2>&1; then + die "no package user for '$only' (looked for '$_acct')" + fi + say "# checking only '$only' (account $_acct)" + fi _vfy_log_open local problems=0 fixed=0 + _vfy_n_fixed=0 detail "# checking for accounts that share an id ..." # ---- 1. two accounts sharing one id ----------------------------------- # @@ -5417,7 +8341,7 @@ cmd_verify() { # One implementation. This calls it. local n_wrong=0 checked=0 _vfy_n_checked=0; _vfy_n_wrong=0 - _vfy_manifest_ownership "$fix" + _vfy_manifest_ownership "$fix" "$only" checked="$_vfy_n_checked"; n_wrong="$_vfy_n_wrong" if [ "$n_wrong" -gt 0 ]; then [ "$n_wrong" -ge 8 ] && warn " ... and $((n_wrong - 8)) more" @@ -5429,7 +8353,15 @@ cmd_verify() { detail "# looking for files no manifest claims ..." # ---- 3b. files no manifest claims ------------------------------------- # + # Refresh our own tools' manifest first (it is rewritten from what is on + # disk, so a file added by a newer `make install` -- the completion file -- + # gets claimed here instead of reported below). Only under --fix: a bare + # verify changes nothing, not even its own records. + [ "$fix" = 1 ] && _adopt_pkgusr_tools 2>/dev/null _vfy_unclaimed || problems=$((problems+1)) + if [ "$fix" = 1 ]; then share_shared_files; fi + detail "# checking that install records describe reality ..." + _vfy_lying_records "$fix" || problems=$((problems+1)) detail "# looking for files with no owner ..." # ---- 4. files with no owner at all ------------------------------------ # @@ -5462,6 +8394,7 @@ cmd_verify() { # discarded. [ "$fix" = 1 ] && ensure_pkgusr_roots 2>/dev/null _vfy_user_homes "$fix" || true + _vfy_misplaced_homes "$fix" || problems=$((problems+1)) _vfy_build_user_leftovers "$fix" || true _vfy_user_order "$fix" || true @@ -5482,8 +8415,9 @@ cmd_verify() { return 0 fi [ -n "$VERIFY_LOG" ] && say "full report: $VERIFY_LOG" + fixed=$((fixed + _vfy_n_fixed)) if [ "$fix" = 1 ]; then - ok "repaired $fixed path(s); $problems area(s) had problems" + ok "repaired $fixed thing(s); $problems area(s) had problems" say "Run verify again to confirm." else warn "$problems area(s) need attention. Repair with:" @@ -5515,7 +8449,7 @@ cmd_verify() { # The command WRAPPERS are deliberately not here: they stay root:root so a # package user cannot rewrite the rule that constrains it. never_claim_list # covers them, so verify stops asking. -PKGUSR_TOOLS="lfs-helper packagemanager packagemanager_install blfs lfs +PKGUSR_TOOLS="lfs-helper packagemanager packagemanager_install blfs lfs lfs-phases add_package_user install_package list_package uninstall_package forall_direntries_from grep_all_regular_files_for list_suspicious_files list_suspicious_files_from" @@ -5531,8 +8465,10 @@ _adopt_pkgusr_tools() { for t in $PKGUSR_TOOLS; do # /usr/bin AND /usr/sbin: the hint's add_package_user and # install_package go to sbin, and looking only in bin is why a finished - # build still reported them as claimed by nobody. - for d in /usr/bin /usr/sbin; do + # build still reported them as claimed by nobody. The completions dir + # for the same reason one release later: `make install` puts the + # bash completion there, and verify reported it as claimed by nobody. + for d in /usr/bin /usr/sbin /usr/share/bash-completion/completions; do p="${SNAP_ROOT%/}$d/$t" [ -e "$p" ] || continue echo "$p" >> "$man.new" @@ -5551,6 +8487,430 @@ _adopt_pkgusr_tools() { return 0 } + +# --------------------------------------------------------------------------- # +# pm-install: what was packagemanager_install (item 6, step 7) +# --------------------------------------------------------------------------- # +# +# The fifth tool is gone. Its build loop moved here in 1.11.57 (cmd_build +# --script); this section is the FRONT-MATTER that survived it, transplanted +# rather than rewritten -- the exact bash that carried a full fresh build. +# packagemanager calls `lfs-helper pm-install <account> <script>` where it +# used to exec /usr/bin/packagemanager_install. +# +# What was deliberately dropped, with the user's sign-off (no compatibility +# required): the old-style installBaseScript/vim editing flow, and the +# engine's private runner/retry/wrapper/recorder block -- the one loop does +# all of that, for every caller. + +# The desktop user that u_ application accounts serve. Was hardcoded in the +# engine; now read from config, same default so an existing system behaves +# identically. +_pm_home_user() { echo "${LFS_HOME_USER:-n76310}"; } + +# Stage an already-downloaded tarball into the account's home, so an install +# never dials out for a file that is sitting in /sources. +_pm_stage_source() { + local _u="$1" _script="$2" _pkg _home _d + [ -f "$_script" ] || return 0 + _pkg="$(sed -n 's/^[[:space:]]*pkg=["'"'"']\{0,1\}\([^"'"'"']*\)["'"'"']\{0,1\}[[:space:]]*$/\1/p' \ + "$_script" | head -n1)" + case "$_pkg" in ""|*'$'*) return 0 ;; esac + _home="$(_recorded_home_for "$_u")" + [ -n "$_home" ] && [ -d "$_home" ] || return 0 + [ -f "$_home/$_pkg" ] && return 0 + for _d in "${LFS_SOURCES_DIR:-/sources}" /sources; do + if [ -f "$_d/$_pkg" ]; then + cp "$_d/$_pkg" "$_home/$_pkg" || return 0 + chown "$_u:$_u" "$_home/$_pkg" 2>/dev/null + ok "staged $_d/$_pkg -> $_home (already downloaded)" + return 0 + fi + done + return 0 +} + +# Create an account the way the engine did: the hint's add_package_user, then +# the ONE home repair, then the ONE sorter. Used for every pm-install +# account -- p_, nimgnu_ and u_ alike -- because this exact sequence is what +# the fresh build proved. +_pm_add_account() { + need_root + add_package_user "$1" "$1" 10000 20000 "$1" 10000 20000 || die "could not create account $1" + _repair_misplaced_home "$1" 1 1 || true + cmd_sort_users --run >/dev/null 2>&1 || true +} + +_pm_user_in_group() { + grep "^$2:" "$ETC/group" 2>/dev/null | sed "s/.*://g;s/,/\n/g" | grep -qw "^$1$" +} + +_pm_add_collector_group() { + local group_name="$1" + if ! echo "$group_name" | grep -qw "audio"; then + if ! echo "$group_name" | grep -q "^u_"; then + echo "$group_name" | grep -q "^nimgnu_" || + group_name="nimgnu_$1" + fi + fi + say "" + say "Create collector group $group_name" + if grep -q "^$group_name:" "$ETC/group" && grep -q "^$group_name:" "$ETC/passwd"; then + ok "group and user exist" + return 0 + fi + if ! assume_yes; then + printf 'create collector account %s? [y/N]: ' "$group_name" + local check; read -e -r check + [ "$check" = "y" ] || die "declined creating collector group $group_name" + fi + _pm_add_account "$group_name" +} + +_pm_join_group() { + local user="$1" group_name="$2" + if ! echo "$group_name" | grep -qw "audio"; then + if ! echo "$group_name" | grep -q "^u_"; then + echo "$group_name" | grep -q "^nimgnu_" || + group_name="nimgnu_$2" + fi + fi + _pm_user_in_group "$user" "$group_name" && { + ok "$user is already in $group_name"; return 0; } + usermod -a -G "$group_name" "$user" \ + || warn "could not add $user to $group_name" + _pm_user_in_group "$user" "$group_name" \ + && ok "$user is now in $group_name" +} + +# install_groups=( ... ) in the script: offer collector-group membership. +_pm_check_groups() { + local script="$1" user="$2" group check + grep -q "^install_groups=(.*)" "$script" || return 0 + [ "${_pm_sourced:-0}" = 1 ] \ + || eval "$(grep --color=never "^install_groups=(.*)" "$script")" + warn "This user may need to be added to collector groups:" + for group in "${install_groups[@]}"; do + [ "$group" = "python-dependencies" ] && group="nimgnu_python-modules" + # resolve to the collector-prefixed group if that is the one that exists + if ! grep -q -E "^$group:" "$ETC/group" \ + && grep -q -E "^nimgnu_$group:" "$ETC/group"; then + group="nimgnu_$group" + fi + if _pm_user_in_group "$user" "$group"; then + ok "$user is already in $group" + continue + fi + grep -q -E "^$group:" "$ETC/group" || _pm_add_collector_group "$group" + if assume_yes; then + warn " usermod -a -G $group $user (--yes)" + _pm_join_group "$user" "$group" + else + warn "By adding to a group the user gets access to all files of the group" + say " usermod -a -G $group $user" + printf '[y/N]: ' + read -e -r check + [ "$check" = "y" ] && _pm_join_group "$user" "$group" + fi + done + return 0 +} + +_pm_groups_of_dir() { + local dir="$1" d + for d in "$dir"/*; do + [ -e "$d" ] && stat -c %G "$d" + done | sort -u +} + +# nimgnu_ collector accounts: take over the named directories. +_pm_install_collector_dirs() { + local user="$1" g f old_group check param dir + say "" + say "Install directories for collector group $user" + for dir in "${nimgnu_groups_dir[@]}"; do + old_group="$(stat -c %G "$dir" 2>/dev/null)" + say "Found groups in $dir:" + _pm_groups_of_dir "$dir" + for g in $(_pm_groups_of_dir "$dir"); do + if [ "$g" = "$old_group" ]; then + say "nothing to change" + else + _pm_join_group "$g" "$old_group" >/dev/null 2>&1 || true + fi + done + for f in "$dir"/*; do + [ -e "$f" ] || continue + # give the group the user's own rights on each entry + chmod "g=$(stat -c %A "$f" | cut -c2-4)" "$f" + done + su -c "mkdir -p '$dir'" "$user" 2>/dev/null || mkdir -p "$dir" + if [ -d "$dir" ]; then + param="" + if [ "${nimgnu_groups_dir_recrusive:-}" = "true" ]; then + if assume_yes; then + param="-R" + else + say "change permission recursive for $user on: $dir" + printf '[y/N]: ' + read -e -r check + [ "$check" = "y" ] && param="-R" + fi + fi + { chgrp $param "$user" "$dir" && chmod $param 775 "$dir"; } \ + || warn "could not hand $dir to group $user" + else + warn "$dir: is not a valid directory" + fi + if [ -n "$old_group" ] && [ "$user" != install ] && [ "$user" != root ] \ + && [ "$old_group" != root ] && [ "$user" != "$old_group" ]; then + _pm_join_group "$user" "$old_group" + fi + done +} + +# u_ application accounts: pam su rule, shared dir, launcher + .desktop entry. +_pm_install_app_user() { + local user="$1" script="$2" home="$3" + local home_user home_user_dir t linenumber s_dir_nam app + home_user="$(_pm_home_user)" + home_user_dir="/home/$home_user" + say "" + say "Install application user $user (for $home_user)" + if [ -f "$ETC/pam.d/su_u_user" ]; then + linenumber="$(wc -l < "$ETC/pam.d/su_u_user")" + if sed "${linenumber}q;d" "$ETC/pam.d/su_u_user" | grep -q "success=1 default=ignore"; then + t="success=ignore default=1" + else + t="success=1 default=ignore" + fi + grep -q "$user).*USER.*\<$user\>.*\<$home_user\>" "$ETC/pam.d/su_u_user" || + sed -i "$((1+$(grep -n 'case "$PAM_USER" in' "$ETC/pam.d/su_u_user" | sed "s/\:.*$//g")))i\ \ $user) USERS=\"$user $home_user\"; ;;" "$ETC/pam.d/su_u_user" + say "modified pam.d/su_u_user" + else + warn "$ETC/pam.d/su_u_user does not exist -- su rules not installed" + fi + chmod -R 0750 "$home" + + if grep -q "^install_groups=(" "$script"; then + eval "$(grep "^install_groups=(" "$script")" + _pm_check_groups "$script" "$user" + if echo "${install_groups[@]}" | grep -wq u_xdg_runtime; then + rm -f "$home/.bash_profile" + ln -s /etc/pkgusr/skel-u_xdg/.bash_profile "$home/.bash_profile" + chmod 755 "$home/.bash_profile" 2>/dev/null + chown "$user:" "$home/.bash_profile" + fi + fi + if grep -q "^shared=true" "$script"; then + s_dir_nam="$(echo "$user" | sed "s/^u_//g")" + mkdir -p "$home/${s_dir_nam^}_shared" + chmod g+sw "$home/${s_dir_nam^}_shared" + chmod 770 "$home/${s_dir_nam^}_shared" + chown -R "$user:" "$home/${s_dir_nam^}_shared" + ln -sf "$home/${s_dir_nam^}_shared" "$home_user_dir/Shared" 2>/dev/null + chown -R "$home_user:" "$home_user_dir/Shared/${s_dir_nam^}_shared" 2>/dev/null + fi + if grep -q "^app=" "$script"; then + eval "$(grep "^app=" "$script")" + if command -v "$app" >/dev/null 2>&1; then + if [ ! -f "$home_user_dir/bin/$app" ]; then + mkdir -p "$home_user_dir/bin" "$home_user_dir/.local/share/applications" 2>/dev/null + cat > "$home_user_dir/bin/$app" << PMEOF +#!/bin/bash + +su - u_$app -c "$app \$(echo "\${@}" | sed "s/\"/\\\\\"/g")" +PMEOF + cat > "$home_user_dir/.local/share/applications/$app.desktop" << PMEOF +[Desktop Entry] +Name=${app^} +Exec=script -c "$home_user_dir/bin/$app" /dev/null +Terminal=false +Type=Application +Icon=${app} +StartupWMClass=${app^} +PMEOF + chown "${home_user}:" "$home_user_dir/bin/$app" + chmod +x "$home_user_dir/bin/$app" + chown "${home_user}:" "$home_user_dir/.local/share/applications/$app.desktop" + ok "created $home_user_dir/bin/$app and its .desktop entry" + fi + else + warn "$app not found -- launcher not created" + fi + fi + usermod -a -G "$user" "$home_user" 2>/dev/null \ + || warn "could not add $home_user to group $user" +} + +cmd_pm_install() { + need_root + local user_name="" install_script="" a + for a in "$@"; do + case "$a" in + -h|--help) cmd_help_for pm-install; return 0 ;; + -*) die "unknown option: $a (usage: lfs-helper pm-install <account> <script>)" ;; + *) if [ -z "$user_name" ]; then user_name="$a" + elif [ -z "$install_script" ]; then install_script="$a" + else die "usage: lfs-helper pm-install <account> <script>"; fi ;; + esac + done + [ -n "$user_name" ] && [ -n "$install_script" ] \ + || die "usage: lfs-helper pm-install <account> <script>" + [ -f "$install_script" ] || die "no such install script: $install_script" + + # The naming rule is ours; asking is idempotent, so a name that already + # carries its prefix comes back unchanged. nimgnu_/u_ names pass through + # pkg_owner_name untouched by construction. + user_name="$(pkg_owner_name "$user_name")" + # AN ACCOUNT THAT ALREADY EXISTS CAN STILL BE POINTING AT THE WRONG HOME. + # + # The repair runs when an account is CREATED (add_package_user puts the + # home in /usr/src/<name>, one line before it is moved). An account made + # by an older version -- or carried over from a previous system -- was + # never re-examined, so /etc/passwd still said /usr/src/p_elogind while + # every tool used /usr/src/pkgusr/p_elogind. `su -` starts in the PASSWD + # home, so the build ran in one directory and everything else looked in + # the other: two half-populated homes, and a build that could not find its + # own sources. + user_exists "$user_name" && _repair_misplaced_home "$user_name" 1 || true + local home + home="$(pkgusr_home_for "$user_name")" + + say "" + say "-------------------------------" + say "Install $user_name" + say "-------------------------------" + say "install file: $install_script" + if ! assume_yes; then + warn "First check the install file for: link, name, wrong commands" + warn "check that \"$user_name\" is the right account" + printf 'Press enter to continue ' + read -r -n1 _ + say "" + fi + + # Phased scripts carry their metadata (install_groups, nimgnu_groups_dir, + # app=, shared=) as top-level assignments; sourcing is how the engine read + # them, so it is how this reads them. + local _pm_sourced=0 + if grep -q "^install_pkg() {" "$install_script"; then + _pm_sourced=1 + # OUR shell options must not govern THEIR file. We source the script + # only to read its metadata, and this shell runs under set -u; a book + # string that merely LOOKS like a variable then killed the install + # before it started: + # install_texlive...: line 25: TEXLIVE_PREFIX: unbound variable + # The generator no longer emits such strings unquoted (1.12.18), but + # the rule stands on its own: sourcing someone else's file must not + # impose our strictness on it, or every future script that trips one + # of our settings fails for a reason that is ours, not theirs. + local _pm_opts="$-" + set +u +e + # shellcheck disable=SC1090 + source "$install_script" + case "$_pm_opts" in *u*) set -u ;; esac + case "$_pm_opts" in *e*) set -e ;; esac + fi + + user_exists "$user_name" || _pm_add_account "$user_name" + user_exists "$user_name" || die "the account $user_name could not be created" + + # The tarball, now that there is a home to put it in. + _pm_stage_source "$user_name" "$install_script" + + # Ask again, now that the account exists -- creating it may have settled + # on a different home than the one computed above. + local _now_home + _now_home="$(_recorded_home_for "$user_name")" + [ -n "$_now_home" ] && home="$_now_home" + mkdir -p "$home/log" + + _pm_check_groups "$install_script" "$user_name" + + # The canonical, editable script (edit THIS to keep changes across + # re-installs), the runtime copy the loop executes, a read-only record of + # what was last installed, and a diff against it. + # ONE FILE TO EDIT, ONE TO RUN, ONE RECORD. + # + # There used to be a fourth: install_p_<user>, a second runtime copy of + # the very same text. Every failure then had to explain itself -- + # "script: install_p_hatchling / staged: install_hatchling / edit: + # install_Hatchling-1.28.0" -- three names for one script, and the one + # that ran was the one the user was told NOT to edit. The canonical + # file is runnable; run it. install_p_<user> is removed when it is a + # copy, and left alone if someone has edited it. + local base; base="$(basename "$install_script")" + diff "$home/install_last" "$install_script" \ + >> "$home/log/install_$user_name-$(date +%Y-%m-%d_%H-%M-%S).diff" 2>/dev/null + [ "$install_script" -ef "$home/$base" ] || cp "$install_script" "$home/$base" + chmod 0755 "$home/$base" 2>/dev/null || true + chown "$user_name:" "$home/$base" 2>/dev/null || true + if [ -f "$home/install_$user_name" ]; then + if cmp -s "$home/install_$user_name" "$home/$base"; then + rm -f "$home/install_$user_name" + else + warn "$home/install_$user_name differs from $base and is no longer" + warn " used -- move your edits into $base (or delete the old file)" + fi + fi + if ! [ "$install_script" -ef "$home/install_last" ]; then + rm -f "$home/install_last" 2>/dev/null + cp "$install_script" "$home/install_last" + chmod 0444 "$home/install_last" 2>/dev/null || true + fi + chown "$user_name:" "$home"/install_* 2>/dev/null + chown -R "$user_name:" "$home/log/" 2>/dev/null + + # The special installers, keyed on what the account IS. + case "$user_name" in + nimgnu_*) _pm_install_collector_dirs "$user_name" ;; + u_*) _pm_install_app_user "$user_name" "$install_script" "$home" ;; + esac + + # THE ONE LOOP. Phased scripts go through cmd_build -- same wrappers, + # same grant-and-retry, same records, same failure box as every other + # install on this system. A rare old-style script (no dispatcher) still + # runs through the one runner, untracked, and is recorded afterwards. + local rc=0 + if [ "$_pm_sourced" = 1 ]; then + local _extra="" + [ "${PM_NO_AUTO_FIX:-0}" = "1" ] && _extra="--no-auto-fix" + cmd_build "$user_name" --script "$home/$base" \ + --phase "${PM_MODE:-all}" --force $_extra + rc=$? + else + run_phase_as "$user_name" "$home/$base" "" \ + "$home/log/pm-install-$(date +%Y-%m-%d_%H-%M-%S).log" + rc=$? + [ "$rc" = 0 ] && [ "${PM_NO_REFRESH:-}" != "1" ] \ + && { cmd_record_install "$user_name" >/dev/null 2>&1 || true; } + fi + [ "$rc" = 0 ] || return "$rc" + + # After-care: the man-page db, detached so Ctrl-C cannot interrupt the + # finished install. It runs into the NEXT package's attribution window, + # which is half of why manifests over-claimed -- _settle_tree_before_stamp + # now waits for exactly this before the next window opens, so detaching + # stays safe. + # the library cache first: a package that just installed a library is not + # usable until ldconfig has seen it, and the wrapper skipped it + # -X: REBUILD THE CACHE, TOUCH NO SYMLINKS. + # + # ldconfig does two jobs: it writes /etc/ld.so.cache, and it creates or + # updates SONAME symlinks (libfoo.so.6 -> libfoo.so.6.2.1) in the library + # directories. The second one, run as root, drops root-owned symlinks + # into package-owned trees -- which is exactly how libfreetype.so.6 came + # to be owned by p_libpaper earlier in this build, and how a package's + # own file stops belonging to it. The cache is root's to rebuild; the + # symlinks belong to the package that installs the library, and its own + # `make install` creates them. + soft "could not refresh the library cache (ldconfig -X)" -- ldconfig -X + warn "Installed. Refreshing mandb in the background..." + setsid mandb >/dev/null 2>&1 </dev/null & disown 2>/dev/null || true + return 0 +} + # --------------------------------------------------------------------------- # # the ownership epoch # --------------------------------------------------------------------------- # @@ -5581,6 +8941,132 @@ OWNERSHIP_MARK="$STATE_PROGRESS/ownership-established" ownership_established() { [ -f "$OWNERSHIP_MARK" ]; } +# --------------------------------------------------------------------------- # +# The install record -- ONE function, two strategies (item 6, step 1) +# --------------------------------------------------------------------------- # +# +# What a package installed was written down twice, by two drivers, in two +# places: lfs-helper's $MANIFESTS/<n>.files (scan by TIMESTAMP) and +# packagemanager_install's ~/pkg.lst (scan by OWNER). Each tool read only its +# own, so `packagemanager update` reported "0 of 97 judged" on a system where +# every one of those packages was recorded -- in the other file. +# +# Both strategies have to stay, because each is the only one that works in its +# own window: +# +# * SCAN BY TIMESTAMP is the build-stage recorder. Early in the LFS build a +# package's files are not owned by its package user yet -- ownership is +# established at one later step -- so an owner scan returns nothing. It is +# also the only one that sees a config step REWRITE an existing file +# without installing a new one. +# * SCAN BY OWNER (list_package) is the default everywhere after that. It is +# self-correcting: it reports the tree as it is now, not as some past run +# recorded it. +# +# So the choice is made ONCE, here, by one named predicate -- not by an inline +# test in each driver, which is the shape of every bug this project has had. +_record_strategy() { + ownership_established && echo owner || echo timestamp +} + +# Write ~/pkg.lst for one package by scanning for files it owns. +# Returns 1 (and writes nothing) when the scan finds nothing: a package that +# owns no files has not installed, and an empty record must never overwrite a +# real one. Empty-as-pass has cost this project three separate bugs. +_write_pkg_lst() { + local name="$1" owner home tmp n + owner="$(pkg_owner_name "$name")" + user_exists "$owner" || return 1 + home="$(pkgusr_home_for "$name")" + [ -d "$home" ] || return 1 + # SCAN_PRUNE is empty until this runs, and an unpruned scan sweeps + # /build, /sources and the unpacked source tree into the record -- the + # exact bug that once put 3348 scratch files into a manifest. + scan_prune_set + tmp="$home/pkg.lst.new" + # /var/mail/<owner> IS NOT AN INSTALLED FILE. + # + # useradd creates a mail spool for every account, so it is owned by the + # package user and the scan picks it up -- and then a package that + # installed NOTHING has a one-line manifest whose one path exists, which + # is the evidence `state` accepts for "installed". That is how Pathspec, + # whose install step had been commented out by a bad filter, was skipped + # by the planner and Hatchling died on `No module named 'pathspec'`. + # An empty record is honest; a record of the account's own furniture is + # not. + find "${SNAP_ROOT%/}/" -xdev -user "$owner" "${SCAN_PRUNE[@]}" \ + -not -path "$home" -not -path "$home/*" \ + -not -path "${SNAP_ROOT%/}/var/mail/*" 2>/dev/null \ + | LC_ALL=C sort -u > "$tmp" || { rm -f "$tmp"; return 1; } + n="$(count_stdin_lines < "$tmp")" + if [ "${n:-0}" -eq 0 ]; then + rm -f "$tmp" + return 1 + fi + chown "$owner" "$tmp" 2>/dev/null || true + mv "$tmp" "$home/pkg.lst" || return 1 + return 0 +} + +# THE recorder. Every driver calls this after a successful install. +# record_install <name> +# The manifest is always kept: it holds what the build TOUCHED, which an owner +# scan cannot know, and cleanup needs it. pkg.lst is written whenever +# ownership makes it meaningful, so both files exist and agree. +cmd_record_install() { + local name="${1:-}" + [ -n "$name" ] || die "usage: lfs-helper record-install <package>" + local strategy; strategy="$(_record_strategy)" + case "$strategy" in + owner) + if _write_pkg_lst "$name"; then + say "# recorded $name by owner scan (pkg.lst)" + else + # not fatal, and not silent: the manifest still stands + warn "# $name: owner scan found nothing -- pkg.lst not written" + warn "# (the build-stage manifest is still the record)" + return 1 + fi ;; + *) + say "# $name: recorded by the build-stage manifest (ownership is" + say "# not established yet; pkg.lst follows at that step)" ;; + esac + return 0 +} + +# Give every package a pkg.lst once ownership exists. Run at the ownership +# step, so a package built during the build stage gets its owner-scanned +# record without being rebuilt. +cmd_reconcile_records() { + need_root + ownership_established || { + warn "# ownership is not established yet -- nothing to reconcile." + return 0 + } + local run=0; [ "${1:-}" = "--run" ] && run=1 + local n=0 miss=0 name + while IFS= read -r name; do + [ -n "$name" ] || continue + local home; home="$(pkgusr_home_for "$name")" + [ -d "$home" ] || continue + [ -s "$home/pkg.lst" ] && continue + if [ "$run" = 0 ]; then + n=$((n+1)); continue + fi + if _write_pkg_lst "$name"; then n=$((n+1)); else miss=$((miss+1)); fi + done < <(steps 2>/dev/null || true) + if [ "$run" = 0 ]; then + say "# $n package(s) have no pkg.lst yet" + warn "(dry run -- re-run with --run to write them)" + else + ok "# wrote pkg.lst for $n package(s)" + [ "$miss" -gt 0 ] && \ + warn "# $miss package(s) own no files -- left for verify to explain" + fi + return 0 +} + + # Can ownership be established yet? Prints why not, when asked to. ownership_possible() { local explain="${1:-}" why="" @@ -6012,17 +9498,20 @@ cmd_build_all() { say "" _vfy_user_order 1 || true + _strip_if_asked_for _warn_if_strip_was_asked_for _warn_if_no_login _say_what_is_yours_to_finish } -# The two things this tool deliberately does not do. +# The two things build-all deliberately does not do. # # The kernel and the bootloader are decisions about the whole MACHINE, not # about a package, and getting either wrong costs you the system you are # building on -- so they are yours, on purpose, and the build ends by saying -# so rather than leaving you to discover it after the reboot. +# so rather than leaving you to discover it after the reboot. The kernel +# has its own tool (lfs-kernel) precisely so its config can be edited before +# anything is built; the message points there rather than at the book. # # rEFInd is opt-in for the same reason and stays that way: the machine booted # in order to build LFS at all, so it already has a working bootloader, and @@ -6032,11 +9521,13 @@ _say_what_is_yours_to_finish() { say "" say "# Two things are yours to finish, on purpose:" say "#" - say "# 1. THE KERNEL. Nothing here configures, builds or installs one." - say "# Book 10.3, from inside the chroot:" - say "# cd /sources && tar -xf linux-*.tar.xz && cd linux-*" - say "# make menuconfig && make && make modules_install" - say "# cp -iv arch/x86/boot/bzImage /boot/vmlinuz-lfs" + say "# 1. THE KERNEL. build-all does not build one. lfs-kernel does," + say "# as a package (p_linux), with its firmware and headers -- edit" + say "# /etc/pkgusr/kernel.conf first (VERSION, ESP, KERNEL_NAME," + say "# FIRMWARE_ONLY), then:" + say "# lfs-kernel # build + install the configured version" + say "# lfs-kernel verify # is everything where the tools expect" + say "# Or by hand, book 10.3, from inside the chroot." say "#" say "# 2. THE BOOTLOADER. Nothing was written to any ESP, boot sector" say "# or partition table, and your current bootloader is untouched." @@ -6049,28 +9540,41 @@ _say_what_is_yours_to_finish() { say "# writes a raw device, and never replaces what boots you today." } -# You asked for stripping. Nothing did it, and nothing will say so. +# You asked for stripping, so it is done here -- at the END, when everything +# is built, so nothing that would still be rebuilt gets stripped twice. # -# `lfs build-system session` stores the answer and the prompt says NOT YET -# IMPLEMENTED -- but that was six hours ago, at the top of an interview, and -# the build otherwise finishes as though every answer had been acted on. A -# setting that is collected, carried into the chroot environment, and then -# silently ignored is worse than one that was never offered. +# `cmd_strip` is the ONE implementation; this only decides whether to call it. +# It cannot fail the build: a system with its debug symbols still in it is +# complete and correct, only larger -- so a strip failure is reported by +# `_warn_if_strip_was_asked_for` below, never by an exit code here. +_strip_if_asked_for() { + [ "${LFS_STRIP:-0}" = 1 ] || return 0 + say "" + say "# stripping debug symbols (you answered yes in the interview)" + if cmd_strip --run; then + STRIP_DONE=1 + fi + return 0 +} + +# You asked for stripping and it did not finish. +# +# The answer used to be collected, carried into the chroot, and then ignored +# with a NOT YET IMPLEMENTED notice at the end of a six-hour build. Now the +# work is done above; what remains of the notice is for the one case left -- +# the run failed partway -- and it says what to run, not what is missing. # # Said at the END, next to the login warning, because that is where the things # you still have to do yourself are collected. _warn_if_strip_was_asked_for() { [ "${LFS_STRIP:-0}" = 1 ] || return 0 + [ "${STRIP_DONE:-0}" = 1 ] && return 0 say "" - warn "# you asked for debug symbols to be stripped, and NOTHING DID IT." - warn "#" - warn "# Stripping rewrites every binary, and in this system a file's" - warn "# owner is the record of which package installed it -- so it has" - warn "# to run as that owner, package by package. That is not written" - warn "# yet, and doing it as root would hand the whole tree to root." + warn "# you asked for debug symbols to be stripped, and it did not finish." warn "#" warn "# The system is complete and correct; it is only larger than you" - warn "# asked for. Book 8.85 also removes libtool .la files." + warn "# asked for. Run it again yourself:" + warn "# lfs-helper strip --run" return 1 } @@ -6123,18 +9627,335 @@ _warn_if_no_login() { return 1 } +# Wait (bounded) for the background writers this toolchain knows about. +# Named rather than "any activity": guessing at every process on the system +# would hang on anything long-running the user is doing on purpose. +_SETTLE_PROCS="mandb make-ca update-mime-database gtk-update-icon-cache +gdk-pixbuf-query-loaders glib-compile-schemas fc-cache p11-kit trust +list_package" + +_settle_tree_before_stamp() { + local waited=0 max="${LFS_SETTLE_SECONDS:-60}" busy + command -v pgrep >/dev/null 2>&1 || return 0 + while [ "$waited" -lt "$max" ]; do + busy="" + local pr + for pr in $_SETTLE_PROCS; do + pgrep -x "$pr" >/dev/null 2>&1 && busy="$busy $pr" + done + [ -z "$busy" ] && { [ "$waited" -gt 0 ] && say "# tree is quiet again"; return 0; } + [ "$waited" = 0 ] \ + && say "# waiting for background work to finish:$busy" + sleep 2 + waited=$((waited + 2)) + done + warn "still running after ${max}s:$busy" + warn " this build's file list may claim files THEY write -- check it" + warn " with: lfs-helper verify (raise LFS_SETTLE_SECONDS to wait longer)" + return 0 +} + +# The shell startup files BLFS packages assume. +# +# BLFS pages drop snippets into /etc/profile.d and call pathprepend / +# pathappend / pathremove, which the book defines in "The Bash Shell Startup +# Files" (postlfs/profile). LFS alone does not install them, so rust's +# /etc/profile.d/rustc.sh said +# pathprepend: command not found +# on every package-user login -- noise at best, and a failed configure step +# at worst. +# +# We do not copy the book's files: /etc/profile is the administrator's, and +# an existing one is never touched. We add what the SNIPPETS need -- the +# three functions, in a profile.d file of our own that sorts first -- and, +# only if nothing sources profile.d yet, a small /etc/profile that does. +install_profile_helpers() { + local d="${SNAP_ROOT%/}/etc/profile.d" f="${SNAP_ROOT%/}/etc/profile" + mkdir -p "$d" || { warn "could not create $d"; return 1; } + set_install_dir_owner "$d" 2>/dev/null || true + soft "$d is not group-writable -- packages cannot add profile snippets" \ + -- real_chmod g+w "$d" + + cat > "$d/00-path-functions.sh" <<'PROFEOF' +# Path helpers that BLFS profile.d snippets call. +# Installed by lfs-helper: LFS does not provide them, and every snippet that +# says `pathprepend /opt/foo/bin` needs them to exist before it runs. +pathremove() { + local IFS=':' NEWPATH DIR PATHVARIABLE=${2:-PATH} + for DIR in ${!PATHVARIABLE}; do + [ "$DIR" != "$1" ] && NEWPATH=${NEWPATH:+$NEWPATH:}$DIR + done + export $PATHVARIABLE="$NEWPATH" +} +pathprepend() { + pathremove "$1" "${2:-PATH}" + local PATHVARIABLE=${2:-PATH} + export $PATHVARIABLE="$1${!PATHVARIABLE:+:${!PATHVARIABLE}}" +} +pathappend() { + pathremove "$1" "${2:-PATH}" + local PATHVARIABLE=${2:-PATH} + export $PATHVARIABLE="${!PATHVARIABLE:+${!PATHVARIABLE}:}$1" +} +export -f pathremove pathprepend pathappend +PROFEOF + soft "the path helpers are not world-readable -- logins will not see them" \ + -- real_chmod 644 "$d/00-path-functions.sh" + + # SAY WHAT WAS WRITTEN. This reported only what it declined to do, so a + # user who had just been given the helpers was told "it did not install + # the files". + ok "# wrote $d/00-path-functions.sh (pathremove, pathprepend, pathappend)" + + if [ -f "$f" ] && grep -q 'profile\.d' "$f" 2>/dev/null; then + ok "# your /etc/profile already sources /etc/profile.d -- nothing else to do" + return 0 + fi + if [ -f "$f" ]; then + # Offer, do not impose: /etc/profile is the administrator's file, and + # appending to it is a change to the system like any other. + if confirm_change "append a loop to /etc/profile so it reads /etc/profile.d/*.sh (a backup is kept as /etc/profile.before-lfs-helper)"; then + cp -p "$f" "$f.before-lfs-helper" 2>/dev/null || true + cat >> "$f" <<'PROFEOF' + +# added by lfs-helper: BLFS packages install snippets into /etc/profile.d +for _s in /etc/profile.d/*.sh; do + [ -r "$_s" ] && . "$_s" +done +unset _s +PROFEOF + ok "# /etc/profile now reads /etc/profile.d (backup: $f.before-lfs-helper)" + else + warn "left /etc/profile alone. Add this to it, or the snippets" + warn "packages install will never run:" + warn " for s in /etc/profile.d/*.sh; do [ -r \"\$s\" ] && . \"\$s\"; done" + fi + return 0 + fi + cat > "$f" <<'PROFEOF' +# /etc/profile -- written by lfs-helper because none existed. +# It exists so that the snippets BLFS packages install into /etc/profile.d +# are actually read. Edit freely; it will not be overwritten. +for _s in /etc/profile.d/*.sh; do + [ -r "$_s" ] && . "$_s" +done +unset _s +PROFEOF + soft "/etc/profile is not world-readable" -- real_chmod 644 "$f" + ok "# wrote /etc/profile (it sources /etc/profile.d) and the path helpers" +} + +# The rest of what "The Bash Shell Startup Files" sets up. +# +# The book's chapter creates nine files: /etc/profile, /etc/bashrc, four +# profile.d snippets (extrapaths, readline, umask, i18n) and four skeleton +# dotfiles. We had only the path helpers, so a package user's shell had no +# umask policy, no INPUTRC, no locale and no ~/.bashrc. +# +# These are OUR versions, not copies of the book's, and NOTHING that already +# exists is touched -- these files belong to the administrator. +_write_if_absent() { + local path="$1" + if [ -e "$path" ]; then + detail "# kept existing $path" + return 1 + fi + cat > "$path" || { warn "could not write $path"; return 1; } + soft "$path is not readable by everyone" -- real_chmod 644 "$path" + ok "# wrote $path" + return 0 +} + +install_shell_startup_files() { + local d="${SNAP_ROOT%/}/etc/profile.d" skel="${SNAP_ROOT%/}/etc/skel" + mkdir -p "$d" "$skel" 2>/dev/null || true + + _write_if_absent "$d/umask.sh" <<'EOF' +# A group-writable umask for members of a group of their own (the LFS +# convention: uid == gid), 022 for everyone else. +if [ "$(id -gn)" = "$(id -un)" ] && [ "$EUID" -gt 99 ]; then + umask 002 +else + umask 022 +fi +EOF + + _write_if_absent "$d/readline.sh" <<'EOF' +# Where readline looks for its configuration. +[ -z "${INPUTRC:-}" ] && [ -f /etc/inputrc ] && export INPUTRC=/etc/inputrc +EOF + + _write_if_absent "$d/i18n.sh" <<'EOF' +# Locale. Set LANG here (e.g. en_US.UTF-8); an unset LANG means the C +# locale, which is a choice, not an accident. +if [ -z "${LANG:-}" ] && command -v locale >/dev/null 2>&1; then + : # export LANG=en_US.UTF-8 +fi +export LESSCHARSET="${LESSCHARSET:-utf-8}" +EOF + + _write_if_absent "$d/extrapaths.sh" <<'EOF' +# Directories that exist on a BLFS system but are not in the base PATH. +# Packages installed under /opt add themselves through their own snippets; +# this covers the common local ones. +[ -d /usr/local/bin ] && pathprepend /usr/local/bin +[ -d /usr/local/sbin ] && [ "$EUID" -eq 0 ] && pathprepend /usr/local/sbin +[ -d /usr/local/lib/pkgconfig ] && pathappend /usr/local/lib/pkgconfig PKG_CONFIG_PATH +EOF + + _write_if_absent "${SNAP_ROOT%/}/etc/bashrc" <<'EOF' +# /etc/bashrc -- for INTERACTIVE shells (a login shell reads /etc/profile). +# Aliases and functions belong here; environment variables belong in +# /etc/profile.d. +[ -z "${PS1:-}" ] && return +[ -r /etc/profile.d/readline.sh ] && . /etc/profile.d/readline.sh +[ -r /etc/dircolors ] && eval "$(dircolors -b /etc/dircolors)" 2>/dev/null +alias ls='ls --color=auto' +alias grep='grep --color=auto' +EOF + + _write_if_absent "$skel/.bash_profile" <<'EOF' +# A login shell reads this. Keep the login environment in /etc/profile and +# the interactive settings in ~/.bashrc, so both kinds of shell agree. +[ -r /etc/profile ] && . /etc/profile +[ -r ~/.bashrc ] && . ~/.bashrc +EOF + + _write_if_absent "$skel/.bashrc" <<'EOF' +[ -r /etc/bashrc ] && . /etc/bashrc +EOF + + _write_if_absent "$skel/.bash_logout" <<'EOF' +# Runs when a login shell exits. +EOF + + _write_if_absent "$skel/.profile" <<'EOF' +# Read by shells that are not bash. +[ -r /etc/profile ] && . /etc/profile +EOF + return 0 +} + +# Files every package must be able to rewrite. +# +# /usr/share/info/dir is the info INDEX: every package that installs an info +# page runs install-info against it, and as a package user that is +# install-info: Operation not permitted for /usr/share/info/dir +# which kills nettle after it has already installed its libraries. The file +# is never_claim_list'd -- nobody OWNS it -- but nothing made it writable by +# the people who must update it. A shared directory is root:install and +# group-writable; a shared file is the same rule, one level down. +shared_files_list() { + printf '%s\n' \ + "${SNAP_ROOT%/}/usr/share/info/dir" \ + "${SNAP_ROOT%/}/etc/xml/catalog" \ + "${SNAP_ROOT%/}/etc/xml/docbook" \ + "${SNAP_ROOT%/}/usr/share/applications/mimeinfo.cache" \ + "${SNAP_ROOT%/}/usr/share/mime/mime.cache" \ + "${SNAP_ROOT%/}/usr/share/glib-2.0/schemas/gschemas.compiled" \ + "${SNAP_ROOT%/}/usr/share/icons/hicolor/icon-theme.cache" +} + +# The rest of that list is the same answer to the same question, reached one +# failure at a time -- so here is the whole family at once, before each one +# costs its own session: +# update-desktop-database -> applications/mimeinfo.cache +# "The databases in [/usr/share/applications] could not be updated." +# update-mime-database -> mime/mime.cache +# glib-compile-schemas -> glib-2.0/schemas/gschemas.compiled +# gtk-update-icon-cache -> icons/hicolor/icon-theme.cache +# Each is REGENERATED from what other packages installed, by whichever +# package happens to run the tool last. None is installed by a single +# package, so none may be owned by one. + +# The test for membership, since this list will keep growing: +# is the file REWRITTEN by many packages rather than installed by one? +# /usr/share/info/dir (install-info), /etc/xml/catalog and /etc/xml/docbook +# (xmlcatalog) are indexes: docbook-xml creates the catalog as its own user +# and docbook-xsl then cannot write it -- +# could not open /etc/xml/catalog for saving +# -- which is the info-index failure again, one directory over. A file that +# only one package ever writes does NOT belong here; it belongs to that +# package. + +# The directories those files live in. They may NOT be sticky -- see below. +shared_file_dirs() { + local f + while IFS= read -r f; do + [ -n "$f" ] && dirname "$f" + done < <(shared_files_list) | sort -u +} + +# Apply the rule to whichever of them exist. Idempotent; safe to repeat. +# +# THE FILE'S MODE IS ONLY HALF OF IT. install-info does not write +# /usr/share/info/dir in place: it writes dir.new and renames it over the +# old one. In a STICKY directory only the owner of a file may replace it, +# so a root-owned index in a sealed /usr/share/info gives +# install-info: Operation not permitted for /usr/share/info/dir +# -- EPERM, not EACCES, and no amount of group-write on the file changes it. +# Sealing (sticky, so package users cannot delete each other's files) and +# sharing (everyone must replace this file) are contradictory rules, so a +# directory that holds a shared index is excluded from sealing. +share_shared_files() { + local f d n=0 + while IFS= read -r f; do + [ -f "$f" ] || continue + if ! { [ "$(stat -c %U:%G "$f" 2>/dev/null)" = "root:install" ] && [ -w "$f" ]; }; then + real_chown root:install "$f" 2>/dev/null || continue + real_chmod 664 "$f" 2>/dev/null || continue + n=$((n + 1)) + fi + done < <(shared_files_list) + while IFS= read -r d; do + [ -d "$d" ] || continue + # GROUP-WRITABLE FOR WHICH GROUP? Only the mode was being fixed, so + # /etc/xml came out root:root 775 -- group-write for group root, + # which helps no package user at all. The group is half of the rule; + # a shared directory is root:INSTALL and group-writable, like every + # other install directory. (/etc/xml is not in install_dirs_list, + # so nothing else was going to give it the group.) + if [ "$(stat -c %G "$d" 2>/dev/null)" != install ] \ + || [ -k "$d" ] || [ ! -w "$d" ]; then + real_chown root:install "$d" 2>/dev/null + real_chmod g+w,o-t "$d" 2>/dev/null && n=$((n + 1)) + fi + done < <(shared_file_dirs) + [ "$n" -gt 0 ] && detail "# $n shared path(s) fixed (root:install, group-writable, not sticky)" + return 0 +} + # Are the install directories already sticky? Checking a handful of the ones # that certainly exist is enough to tell whether sealing has been done. _install_dirs_are_sealed() { + # ANY sticky install directory means sealing has happened. + # + # This demanded that all five probed directories be sticky, which made a + # MIXED tree read as "not sealed at all" -- and verify then proposed + # stripping the sticky bit off ~50 directories that were correctly + # sealed: + # installdir /usr/lib root:install 1775 -> root:install 775 + # A tree goes mixed for ordinary reasons: sealing interrupted, or an + # install directory created after the seal (verify itself creates none, + # but packages do). Sealing is a deliberate, one-way, end-of-build act, + # so the conservative reading is that a partially sealed tree is sealed: + # verify then COMPLETES the seal on the stragglers instead of undoing it. + # A never-sealed mid-build tree has no sticky install dirs at all and + # still reads as unsealed, which is what the build needs. local d probe=0 sealed=0 while IFS= read -r d; do d="${SNAP_ROOT%/}$d" [ -d "$d" ] || continue probe=$((probe + 1)) [ -k "$d" ] && sealed=$((sealed + 1)) - [ "$probe" -ge 5 ] && break + [ "$probe" -ge 12 ] && break done < <(install_dirs_list) - [ "$probe" -gt 0 ] && [ "$sealed" = "$probe" ] + if [ "$probe" -gt 0 ] && [ "$sealed" -gt 0 ] && [ "$sealed" != "$probe" ]; then + warn "install directories are PARTLY sealed ($sealed of $probe probed are sticky)" + warn " treating the tree as sealed and completing it -- a sticky bit" + warn " that seal-install-dirs set is never taken back off here" + fi + [ "$probe" -gt 0 ] && [ "$sealed" -gt 0 ] } # --------------------------------------------------------------------------- # @@ -6528,6 +10349,34 @@ _manifests_in_build_order() { # does, not print the whole usage page again. cmd_help_for() { case "$1" in + pm-install) + cat <<'EOF' +lfs-helper pm-install <account> <script> + +What /usr/bin/packagemanager_install used to be (item 6, step 7): resolve the +account name, confirm, create the account if needed (with the home repair), +stage the tarball from /sources, offer collector-group membership, keep the +install_last/diff records, run the nimgnu_/u_ special installers -- then hand +the script to the ONE build loop (cmd_build --script). + +Driven by packagemanager; honours PM_YES, PM_MODE, PM_NO_AUTO_FIX and +LFS_HOME_USER (the desktop user u_ application accounts serve). +EOF + ;; + fix-home) + cat <<'EOF' +lfs-helper fix-home <name> [--run] + +Puts one account's home where every tool looks (pkgusr-home). The hint's +add_package_user creates /usr/src/<account>; the tools use +/usr/src/pkgusr/<account>. This repoints the passwd entry and merges the +stray directory into the real home -- moving what is missing, deleting what +is identical, and never touching a file that differs. + +Without --run it only reports. `lfs-helper verify` runs the same check for +every package account. +EOF + ;; export-groups) cat <<'EOF' lfs-helper export-groups [<file>] @@ -6561,6 +10410,20 @@ Builds one package as its own package user, tracking every file it installs. expect some failures) --jobs N override MAKEFLAGS for this build --no-auto-fix do not grant collector-group access on a permission failure +EOF + ;; + strip) + cat <<'EOF' +lfs-helper strip [--run] [--keep-la] + +Strips debug symbols from the installed system -- the book's 8.85, run so the +package-user record survives it. strip replaces each file with a new one +owned by whoever ran it, so the files are grouped by owner and each group is +stripped as that owner. Libtool .la files are removed too (the book's +cleanup); --keep-la keeps them. + +Saves several GB; makes a later crash harder to debug. build-all runs this +automatically when the interview answer `strip` is yes. EOF ;; *) return 1 ;; @@ -6609,6 +10472,7 @@ Ownership and permissions (the package-user model): lfs-helper verify --fix and repair what it can lfs-helper fix-perms <name> --run grant one package the access it needs lfs-helper grant-dir <dir> <user> --run + lfs-helper rename-group <old> <new> --run # fix a mistyped group name give one user access to one directory Collector groups (shared directories): @@ -6624,11 +10488,24 @@ Look around / tidy up: lfs-helper wrapper-dir print where the build wrappers live lfs-helper make-wrappers --run (re)write the build wrappers lfs-helper owner-name <package> print the account name for a package + lfs-helper fix-home <name> [--run] put an account's home where the tools look + lfs-helper pm-install <account> <script> prepare and install one package + (what packagemanager_install used to be) lfs-helper manifests [<name>] what each package installed lfs-helper which-package <path> which package installed this file lfs-helper clean-sources --run tidy /sources lfs-helper clean-state --run remove build scratch (keeps the records) lfs-helper prune-locales --run drop unwanted translations + lfs-helper strip --run strip debug symbols, each file as its + owner, so the ownership record holds + lfs-helper record-install <pkg> write the install record (one door: owner + scan once ownership exists, else the + build-stage manifest) + lfs-helper reconcile-records --run + give every package a pkg.lst, once + lfs-helper run-script --user U --script S [--phase P] [--log F] + run one phase as the package user (the + one runner both drivers use) lfs-helper find-user-site --run find pip installs hidden in ~/.local Any command: -h explains it. Dry run is the default; --run applies. @@ -6667,14 +10544,21 @@ case "${1:-}" in pkgusr-info) shift; cmd_pkgusr_info "$@" ;; pkgusr-home) shift; cmd_pkgusr_home "$@" ;; owner-name) shift; cmd_owner_name "$@" ;; + fix-home) shift; cmd_fix_home "$@" ;; + pm-install) shift; cmd_pm_install "$@" ;; clean-state) shift; cmd_clean_state "$@" ;; which-package) shift; cmd_which_package "$@" ;; grant-dir) shift; cmd_grant_dir "$@" ;; + rename-group) shift; cmd_rename_group "$@" ;; make-wrappers) shift; cmd_make_wrappers "$@" ;; wrapper-dir) shift; cmd_wrapper_dir "$@" ;; export-groups) shift; cmd_export_groups "$@" ;; import-groups) shift; cmd_import_groups "$@" ;; prune-locales) shift; cmd_prune_locales "$@" ;; + strip) shift; cmd_strip "$@" ;; + record-install) shift; cmd_record_install "$@" ;; + run-script) shift; cmd_run_script "$@" ;; + reconcile-records) shift; cmd_reconcile_records "$@" ;; repair-adoption) shift; cmd_repair_adoption "$@" ;; adopt-existing|adopt-dirs) shift; cmd_adopt_existing "$@" ;; @@ -6695,8 +10579,24 @@ case "${1:-}" in esac cmd_verify "$@" ;; seal-install-dirs) shift; cmd_seal_install_dirs "$@" ;; + # the shell startup helpers BLFS profile.d snippets call. Wired into + # cmd_init_pkgusr too, but that is not the command people run -- the + # bootstrap (`packagemanager setup`) is, and it calls this. + install-profile) + shift + # `--yes` here, not only LFS_ASSUME_YES in the environment: this + # command's one question (append the profile.d loop to /etc/profile) + # is the difference between cargo being on PATH and not, and asking + # people to export a variable to answer it is a poor trade. + for _a in "$@"; do + case "$_a" in -y|--yes) LFS_ASSUME_YES=1 ;; esac + done + install_profile_helpers + install_shell_startup_files ;; done) shift; mark_done "${1:?usage: lfs-helper done <name>}"; ok "marked '$1' built" ;; undone) shift; unmark "${1:?usage: lfs-helper undone <name>}"; ok "cleared '$1'" ;; -h|--help|help|"") usage ;; *) die "unknown command '$1' (try: lfs-helper --help)" ;; esac + +} # end of the read-ourselves-once group diff --git a/lfs-kernel b/lfs-kernel new file mode 100755 index 0000000..5a4c91b --- /dev/null +++ b/lfs-kernel @@ -0,0 +1,789 @@ +#!/bin/bash +# lfs-kernel -- build and install a kernel. +# +# lfs-kernel build and install the configured version +# lfs-kernel list what kernel.org offers right now +# lfs-kernel 6.12.40 that version, once (config untouched) +# lfs-kernel latest the newest stable, once +# lfs-kernel detect-esp print the EFI directory this would use +# lfs-kernel verify [ver] check that everything is installed +# +# Settings: /etc/pkgusr/kernel.conf, INSTALLED by `make install` so it can be +# edited before the first run. The build commands are in one block below, +# after the line THE COMMANDS; everything above it picks a version and checks +# paths. +set -u + +CONF=/etc/pkgusr/kernel.conf +MIRROR_JSON=https://www.kernel.org/releases.json +MIRROR_TAR=https://cdn.kernel.org/pub/linux/kernel + +# -------------------------------------------------------------------------- +# where does the kernel go? $ESP is the directory on the EFI System +# Partition that your boot manager reads -- for rEFInd, the one holding your +# vmlinuz. Find it with: findmnt -t vfat / ls /boot/efi/EFI +# -------------------------------------------------------------------------- +_detect_esp() { + local m + for m in $(findmnt -rn -t vfat,msdos -o TARGET 2>/dev/null); do + # a directory that already holds a kernel is the right one + for d in "$m"/EFI/*/ "$m"/EFI/BOOT/ "$m"/; do + [ -d "$d" ] || continue + case "$(echo "$d"*vmlinuz* "$d"*bzImage* 2>/dev/null)" in + *vmlinuz*|*bzImage*) printf '%s' "${d%/}"; return 0 ;; + esac + done + done + # nothing with a kernel in it: the first vfat mount, if any + findmnt -rn -t vfat,msdos -o TARGET 2>/dev/null | head -n1 +} + +case "${1:-}" in + list) + echo "# kernel.org offers:" + _releases | awk '{printf " %-12s %s\n", $1, $2}' + echo "# build one: lfs-kernel <version>" + exit 0 ;; + detect-esp) + printf '%s\n' "$(_detect_esp)"; exit 0 ;; + verify) + shift; _VERIFY=1; VERSION="${1:-}" ;; + -h|--help) + sed -n '2,14p' "$0" | sed 's/^# \?//'; exit 0 ;; + "") ;; + -*) echo "!! unknown option: $1 (try: lfs-kernel --help)"; exit 1 ;; + *) VERSION="$1" ;; +esac + +# The config is INSTALLED, not written on first run: it is a file to edit +# before building, not a side effect of trying to. `make install` places it +# and never overwrites an edited one. +if [ ! -f "$CONF" ]; then + echo "!! $CONF is missing. It ships with the tools:" + echo " make install # places $CONF" + echo " or write one; lfs-kernel detect-esp prints the EFI directory." + exit 1 +fi +. "$CONF" + +# A CONFIG FROM AN OLDER RELEASE. +# +# kernel.conf is never overwritten once edited, so an installed copy can +# predate a change to what the settings MEAN. SRC in particular was dropped +# in 1.14.123 -- the sources live in the package user's home now -- and a +# config still setting it is describing a directory nothing reads. +if [ -n "${SRC:-}" ]; then + echo "# note: SRC is set in $CONF and is no longer used -- the sources" + echo "# live in the package user's home (see PKG). A newer config" + echo "# without it is at $CONF.new if make install left one." + unset SRC +fi +[ -n "${PKG:-}" ] || { + echo "!! $CONF has no PKG setting, so it predates 1.14.123." + echo " Take the new one: cp $CONF{.new,} (yours is $CONF.new's basis)" + echo " or add: PKG=linux" + exit 1 +} + +: "${PKG:=linux}"; : "${JOBS:=-j$(nproc)}" +: "${CONFIG_FROM:=running}"; : "${V4L2LOOPBACK:=no}" +: "${KERNEL_NAME:=vmlinuz}" + +# -------------------------------------------------------------------------- +# which version? +# -------------------------------------------------------------------------- +_releases() { # "<moniker> <version>" per line, in kernel.org's order + # Pair the keys by NAME, not by position: releases.json lists "version" + # before "moniker", and a paste of alternate lines silently swapped the + # two columns -- so `awk $1=="stable"` matched nothing and "latest" + # resolved to the empty string. Key order in someone else's JSON is not + # a thing to depend on. + wget -qO- "$MIRROR_JSON" 2>/dev/null \ + | tr ',{}' '\n\n\n' \ + | awk -F'"' ' + /"moniker"/ { m = $4 } + /"version"/ { v = $4 } + m != "" && v != "" { print m, v; m = ""; v = "" }' +} +_latest_stable() { _releases | awk '$1=="stable"{print $2; exit}'; } + +# `verify` inspects what is on disk, so it must not fetch: resolving +# "latest" would ask kernel.org for a version it is never going to look at, +# and fail on a machine with no network while reporting on one that is +# perfectly installed. +if [ "${_VERIFY:-0}" = 1 ]; then + : +elif [ "$VERSION" = latest ]; then + VERSION="$(_latest_stable)" + [ -n "$VERSION" ] || { echo "!! could not reach $MIRROR_JSON -- set VERSION in $CONF"; exit 1; } + echo "# newest stable on kernel.org: $VERSION" +fi + +# THE KERNEL IS A PACKAGE. Its sources belong in its own account's home, +# next to every other package's, so `packagemanager verify`, pkg.lst and +# `remove` all work on it. SRC used to be a free-standing directory in the +# config -- a location invented outside the model, which is exactly what the +# model exists to avoid. +acct="p_$PKG" +# `verify` must not CREATE anything -- a report that makes an account is not +# a report. It only reads $acct's home below. +if [ "${_VERIFY:-0}" != 1 ] && ! id "$acct" >/dev/null 2>&1; then + packagemanager add-user "$PKG" || { echo "!! could not create $acct"; exit 1; } +fi +[ "${_VERIFY:-0}" = 1 ] || lfs-helper fix-home "$acct" --run >/dev/null 2>&1 || true +SRC="$(getent passwd "$acct" 2>/dev/null | cut -d: -f6)/src" +[ "${_VERIFY:-0}" = 1 ] || install -d -o "$acct" -g "$acct" -m 755 "$SRC" + +# Everything that touches the tree runs AS THE PACKAGE USER: a root-owned +# file left in its home would break the next build and every rule about who +# owns what. Only the tarball fetch and the ESP copies are root's. +_as_pkg() { su - "$acct" -c "cd '$tree' 2>/dev/null || cd '$SRC'; $*"; } + +tree="$SRC/linux-$VERSION" +tarball="$SRC/linux-$VERSION.tar.xz" +major="${VERSION%%.*}" + +# -------------------------------------------------------------------------- +# the ESP has to be the EFI partition, not a directory of the same name. +# Every copy below would SUCCEED on the root filesystem and the firmware +# would find nothing -- a failure that only appears at the next boot. +# -------------------------------------------------------------------------- +[ -n "${ESP:-}" ] || { echo "!! ESP is not set in $CONF. Find it with: findmnt -t vfat"; exit 1; } +if [ ! -d "$ESP" ]; then + echo "!! $ESP does not exist. Mount the EFI partition first, or fix ESP" + echo " in $CONF -- creating it here would put vmlinuz where the" + echo " firmware cannot read it." + exit 1 +fi +case "$(stat -f -c %T "$ESP" 2>/dev/null)" in + msdos|vfat|fat*|exfat) ;; + "") echo "# could not identify the filesystem under $ESP -- check it before rebooting" ;; + *) echo "!! $ESP is not on a vfat filesystem, so it is not the EFI partition." + echo " findmnt -t vfat # and set ESP in $CONF"; exit 1 ;; +esac + +# BUILDING IN A CHROOT IS FINE -- IF THE PATHS ARE THIS TREE'S. +# The chroot usually IS the system that will boot. What can go astray is +# each path, so name them and carry on. Judge only on positive evidence: +# where /proc/1/root cannot be read, the comparison says nothing. +_realroot="$(stat -c %d:%i /proc/1/root/. 2>/dev/null || true)" +_thisroot="$(stat -c %d:%i / 2>/dev/null || true)" +if [ "${_VERIFY:-0}" != 1 ] \ + && [ -n "$_realroot" ] && [ -n "$_thisroot" ] && [ "$_realroot" != "$_thisroot" ]; then + echo "# a chroot: fine, as long as these are the tree you boot --" + echo "# $ESP (EFI partition, mounted here)" + echo "# /lib/modules (modules_install and depmod write here)" + echo "# /proc/config.gz (shared with the host: the RUNNING kernel)" + echo "# set CONFIG_FROM=file:<path> in $CONF to start from another config" +fi + +# ---- verify: is everything where it should be? --------------------------- +# +# "is the kernel part done? do we have a validation command so we know +# everything is installed as expected?" Every piece this tool installs is +# checked here, by looking at the system rather than at what the build +# printed: the image and its companions on the ESP, the modules and their +# dependency index, the headers, the embedded firmware, and the two package +# manifests. +if [ "${_VERIFY:-0}" = 1 ]; then + _bad=0 + _ck() { # _ck <description> <test...> + local what="$1"; shift + if "$@" >/dev/null 2>&1; then + echo " ok $what" + else + echo " FAIL $what" + _bad=$((_bad + 1)) + fi + } + if [ -z "$VERSION" ] || [ "$VERSION" = latest ]; then + # whatever was installed most recently, rather than whatever + # kernel.org offers today + VERSION="$(ls -1t "$ESP/$KERNEL_NAME"-* 2>/dev/null | head -1)" + VERSION="${VERSION##*/$KERNEL_NAME-}" + fi + [ -n "$VERSION" ] || { echo "!! nothing installed as $ESP/$KERNEL_NAME-* yet"; exit 1; } + echo "### verifying kernel $VERSION" + _ck "$ESP/$KERNEL_NAME (what the firmware loads)" test -s "$ESP/$KERNEL_NAME" + _ck "$ESP/$KERNEL_NAME-$VERSION" test -s "$ESP/$KERNEL_NAME-$VERSION" + _ck "the two are the same image" cmp -s "$ESP/$KERNEL_NAME" "$ESP/$KERNEL_NAME-$VERSION" + _ck "$ESP/System.map-$VERSION" test -s "$ESP/System.map-$VERSION" + _ck "$ESP/config-$VERSION" test -s "$ESP/config-$VERSION" + _ck "/lib/modules/$VERSION" test -d "/lib/modules/$VERSION" + _ck "modules.dep (depmod has run)" test -s "/lib/modules/$VERSION/modules.dep" + _ck "headers in /usr/include/linux" test -s /usr/include/linux/version.h + # the firmware this config embeds must be on disk, or the image is + # missing pieces it thinks it has + _cfg="$ESP/config-$VERSION" + _fwlist="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' "$_cfg" 2>/dev/null)" + if [ -n "$_fwlist" ]; then + for _f in $_fwlist; do + _ck "firmware /lib/firmware/$_f" test -e "/lib/firmware/$_f" + done + fi + # every directory FIRMWARE_ONLY asked for should be on disk. The + # embedded list above only covers blobs the IMAGE carries; the wireless + # firmware is loaded at run time and was checked by nothing at all -- + # which is why "what about the iwl stuff?" had to be asked. + for _d in ${FIRMWARE_ONLY:-}; do + case "$_d" in + *[*?[]*) _ck "firmware matching $_d" \ + sh -c "ls -d /lib/firmware/$_d >/dev/null 2>&1" ;; + *) _ck "firmware /lib/firmware/$_d" test -e "/lib/firmware/$_d" ;; + esac + done + # named for what it actually is, since intel/ is where the wireless lives + [ -d /lib/firmware/intel/iwlwifi ] \ + && echo " note $(ls /lib/firmware/intel/iwlwifi 2>/dev/null | wc -l) Intel wireless blob(s) in intel/iwlwifi" + + # ...and the out-of-tree module, when the config asks for one. verify + # checked everything the tool installs EXCEPT this -- the one piece whose + # build failed three times, which is precisely the thing a report should + # not be silent about. + case "${V4L2LOOPBACK:-no}" in + yes|1|true) + _ck "v4l2loopback.ko in /lib/modules/$VERSION" \ + sh -c "find /lib/modules/'$VERSION' -name 'v4l2loopback.ko*' \ + -print -quit 2>/dev/null | grep -q ." + _ck "modules.dep knows v4l2loopback" \ + sh -c "grep -q v4l2loopback /lib/modules/'$VERSION'/modules.dep 2>/dev/null" ;; + esac + + # and the package side: both accounts should have a manifest + for _a in "p_$PKG" p_linux-firmware; do + _h="$(getent passwd "$_a" 2>/dev/null | cut -d: -f6)" + if [ -n "$_h" ]; then + _ck "$_a records its files" test -s "$_h/pkg.lst" + fi + done + # A manifest with a source tree in it is a manifest that would delete a + # source tree. 266k entries was the symptom; anything above a few + # thousand for the kernel means the home slipped back in. + _h="$(getent passwd "p_$PKG" 2>/dev/null | cut -d: -f6)" + if [ -n "$_h" ] && [ -s "$_h/pkg.lst" ]; then + _n="$(grep -c . "$_h/pkg.lst")" + echo " note p_$PKG records $_n file(s)" + # ...and if that is implausible, SHOW the lines rather than guessing + # at which path let them in. Two releases were spent reasoning about + # a filter without ever looking at what it failed to match. + if [ "$_n" -gt 50000 ]; then + echo " FAIL that is far too many for a kernel -- a source tree is in it." + echo " the first few entries are:" + head -3 "$_h/pkg.lst" | sed 's/^/ /' + echo " and around the middle:" + sed -n "$((_n / 2))p" "$_h/pkg.lst" | sed 's/^/ /' + _bad=$((_bad + 1)) + fi + # Two ways a source tree gets in: directly, or through + # /lib/modules/<ver>/build, which is a SYMLINK to it. The second is + # the one that actually happened -- and the check that looked only + # for the home said nothing about 266k entries. + if grep -q "^$_h/" "$_h/pkg.lst" 2>/dev/null; then + echo " FAIL the manifest includes the account's own home" + echo " (sources, not installed files)" + _bad=$((_bad + 1)) + fi + if grep -qE "^/lib/modules/[^/]+/(build|source)/." "$_h/pkg.lst" 2>/dev/null; then + echo " FAIL the manifest was scanned THROUGH" + echo " /lib/modules/$VERSION/build, a symlink to the source" + echo " tree -- so it lists the whole tree as installed files." + _bad=$((_bad + 1)) + fi + [ "$_bad" = 0 ] || echo " fix both with: packagemanager reload-pkg-list $PKG" + fi + + _lg=/etc/pkgusr/kernel.last-good + if [ -s "$_lg" ]; then + _lgv="$(cat "$_lg")" + if [ "$_lgv" = "$(uname -r | sed 's/-.*//')" ]; then + echo " note last kernel known to boot: $_lgv (it is running now)" + else + echo " note last kernel known to boot: $_lgv" + fi + else + echo " note no kernel confirmed to boot yet -- $VERSION is untested," + echo " and the running $(uname -r) was not built by this tool" + fi + echo + if [ "$_bad" = 0 ]; then + echo "everything the kernel step installs is present." + echo " running kernel: $(uname -r) installed: $VERSION" + exit 0 + fi + echo "$_bad check(s) failed -- run lfs-kernel to (re)install." + exit 1 +fi + +echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" + +# -------------------------------------------------------------------------- +# THE COMMANDS. Everything above chooses the version and checks the paths; +# everything below is the build, and is meant to be read. +# -------------------------------------------------------------------------- +install -d "$SRC" +if [ ! -d "$tree" ]; then + [ -f "$tarball" ] || { + echo "# fetching $MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" + wget -q --show-progress -O "$tarball.part" \ + "$MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" \ + && mv "$tarball.part" "$tarball" \ + || { rm -f "$tarball.part" + echo "!! download failed -- KERNEL_LIST=1 shows what exists"; exit 1; } + } + tar -xf "$tarball" -C "$SRC" || exit 1 + chown -R "$acct:$acct" "$tree" "$tarball" 2>/dev/null || true +fi +[ -d "$tree" ] || { echo "!! $tree is missing"; exit 1; } + +# boot logo (optional) +if [ -n "${BOOTLOGO:-}" ]; then + logo="$SRC/$(basename "$BOOTLOGO")" + [ -f "$logo" ] || { wget -qO "$logo" "$BOOTLOGO" && chown "$acct:$acct" "$logo"; } || logo="" + [ -n "$logo" ] && _as_pkg "cp -v '$logo' drivers/video/logo/logo_linux_clut224.ppm" +fi + +# .config +case "$CONFIG_FROM" in + running) _as_pkg "gunzip < /proc/config.gz > .config" || { + echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; + file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; + keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; + *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; +esac +# ---- refreshing the config for a NEW kernel ------------------------------ +# +# A config from one kernel is never complete for another: every version adds +# symbols the old file has no answer for. Something must decide them, and +# which something matters: +# +# olddefconfig take the DEFAULT for each new symbol, no questions. The +# right choice for an unattended run, and what this tool has +# always done -- but the defaults are upstream's, not yours. +# oldconfig ask about each new symbol, one at a time. Slow, and needs +# someone at the terminal, but nothing is decided silently. +# menuconfig open the editor and decide everything at once. +# none use the .config exactly as it is; `make` will then ask, or +# fail, depending on what is missing. +# +# It also says how many symbols the refresh had to answer, because "47 new +# symbols were set to their defaults" is worth knowing before a boot. +: "${CONFIG_REFRESH:=olddefconfig}" +_cfg_before="$(mktemp)"; cp "$tree/.config" "$_cfg_before" 2>/dev/null || true +case "$CONFIG_REFRESH" in + none) echo "# not refreshing the config (CONFIG_REFRESH=none)" ;; + olddefconfig|oldconfig|menuconfig|*config) + echo "# refreshing the config for $VERSION (make $CONFIG_REFRESH) ..." + _as_pkg "make $CONFIG_REFRESH" || exit 1 ;; + *) echo "!! CONFIG_REFRESH=$CONFIG_REFRESH is not understood"; exit 1 ;; +esac +if [ -s "$_cfg_before" ] && [ "$CONFIG_REFRESH" != none ]; then + _newsym="$(diff <(sort "$_cfg_before") <(sort "$tree/.config") 2>/dev/null \ + | grep -c '^>' || true)" + [ "${_newsym:-0}" -gt 0 ] \ + && echo "# $_newsym config line(s) differ from the config you started from" +fi +rm -f "$_cfg_before" + +# what a Wayland compositor needs; a warning, not a refusal +for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do + grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" +done +# Drivers that need firmware are no use without the driver. These are the +# ones a laptop misses most often; the firmware for each is in FIRMWARE_ONLY. +for opt in CONFIG_IWLWIFI:"Intel wireless (AX200 and friends)" \ + CONFIG_IWLMVM:"the AX200's own MVM firmware layer" \ + CONFIG_BT_HCIBTUSB:"USB Bluetooth (the AX200's other half)" \ + CONFIG_DRM_AMDGPU:"AMD graphics"; do + _o="${opt%%:*}"; _why="${opt#*:}" + grep -qE "^$_o=(y|m)" "$tree/.config" \ + || echo "!! $_o is not set -- no $_why" +done + +# ---- firmware, as its own package ---------------------------------------- +# +# The drivers need it at RUN time and CONFIG_EXTRA_FIRMWARE needs it at BUILD +# time, so it is installed before the check below rather than skipped. It +# gets its own package user: linux-firmware is a separate upstream project +# with its own release pace, and `packagemanager verify` should see it as +# one thing. +: "${FIRMWARE_FROM:=none}" +# A pattern is anything with a wildcard; cone mode cannot express one. +_sparse_mode() { + case "$*" in *[*?[]*) printf '%s' "--no-cone" ;; *) printf '' ;; esac +} +# In --no-cone mode each entry is a gitignore-style pattern, and a bare +# directory name has to become /dir/ to mean "that directory". +_sparse_args() { + local mode e out="" + mode="$(_sparse_mode "$@")" + for e in $*; do + if [ -n "$mode" ]; then + case "$e" in + */|*[*?[]*) out="$out $e" ;; + *) out="$out /$e/" ;; + esac + else + out="$out $e" + fi + done + printf '%s' "${out# }" +} + +# WHAT IS THERE, NOT JUST WHAT IS NOT. linux-firmware moves blobs between +# the top level and subdirectories between releases, so a name or pattern +# from last year can simply be gone. "iwlwifi does not exist" sends the +# reader to go and look; the search below is the answer. +_firmware_not_found() { + local src="$1" want="$2" stem + stem="$(printf '%s' "$want" | sed 's/[*?].*//; s/[-_]*$//')" + echo "!! FIRMWARE_ONLY asks for '$want', which is not in $src" + [ -n "$stem" ] || return 0 + # ASK THE INDEX, NOT THE WORKING TREE. + # + # A sparse checkout REMOVES everything it did not select, so searching + # the directory finds only what is already installed -- amdgpu and intel + # -- and reports "nothing" for the very file being looked for. git's + # index lists every path the repository has, selected or not, which is + # the question actually being asked. + local _hits="" + if [ -d "$src/.git" ]; then + _hits="$(git -C "$src" ls-files 2>/dev/null | grep -i -- "$stem" | head -12)" + fi + [ -n "$_hits" ] || _hits="$(find "$src" -maxdepth 2 -iname "*$stem*" \ + -printf '%P\n' 2>/dev/null | head -12)" + if [ -n "$_hits" ]; then + echo " the repository has these with '$stem' in the name:" + printf ' %s\n' $_hits + echo " Put a directory or a pattern covering them in FIRMWARE_ONLY" + echo " in $CONF -- a directory name is the simplest thing that works." + else + echo " nothing in the repository matches '$stem' at all." + if [ -d "$src/.git" ]; then + echo " its top level holds:" + git -C "$src" ls-tree --name-only HEAD 2>/dev/null | head -12 \ + | sed 's/^/ /' + fi + fi +} + +_install_firmware() { + local acct="p_linux-firmware" src="" only="${FIRMWARE_ONLY:-}" + id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 + lfs-helper fix-home "$acct" --run >/dev/null 2>&1 || true + local home; home="$(getent passwd "$acct" | cut -d: -f6)" + + case "$FIRMWARE_FROM" in + none) return 0 ;; + copy:*) + src="${FIRMWARE_FROM#copy:}" + [ -d "$src" ] || { echo "!! $src does not exist (FIRMWARE_FROM)"; return 1; } ;; + git) + src="$home/src/linux-firmware" + install -d -o "$acct" -g "$acct" -m 755 "$home/src" + # linux-firmware is over ten gigabytes. With FIRMWARE_ONLY set + # a SPARSE checkout fetches those directories and nothing else -- + # amdgpu alone is a few hundred megabytes. Without it, the whole + # thing, which is what "empty means everything" costs. + if [ -d "$src/.git" ]; then + echo "# updating linux-firmware" + su - "$acct" -c "cd '$src' && git pull --ff-only" || return 1 + if [ -n "$only" ]; then + # not `|| true`: if this fails the checkout keeps whatever + # the last run selected, and the missing firmware is then + # blamed on the pattern + su - "$acct" -c "cd '$src' && git sparse-checkout set $(_sparse_mode "$only") $(_sparse_args "$only")" \ + || { echo "!! could not narrow the firmware checkout to: $only"; return 1; } + fi + elif [ -n "$only" ]; then + echo "# cloning linux-firmware, only: $only" + su - "$acct" -c "git clone --depth 1 --filter=blob:none --sparse '$FIRMWARE_GIT' '$src'" || return 1 + # --no-cone when any entry is a PATTERN rather than a + # directory: Intel wireless firmware is iwlwifi-cc-a0-*.ucode + # at the top level, and cone mode (the default) matches + # directories only, so those files would never arrive. + su - "$acct" -c "cd '$src' && git sparse-checkout set $(_sparse_mode "$only") $(_sparse_args "$only")" || return 1 + else + echo "# cloning ALL of linux-firmware -- this is over 10 GB." + echo "# set FIRMWARE_ONLY in $CONF to fetch only what you need." + su - "$acct" -c "git clone --depth 1 '$FIRMWARE_GIT' '$src'" || return 1 + fi ;; + *) echo "!! FIRMWARE_FROM=$FIRMWARE_FROM not understood"; return 1 ;; + esac + + # LIB/FIRMWARE IS THIS PACKAGE'S DIRECTORY. + # + # It already existed, owned by root or by whoever made it first, so every + # copy failed with "Permission denied" -- two hundred lines of it. A + # package user cannot chown, and firmware has exactly one owner, so root + # hands the directory over here. (Where another package genuinely shares + # a directory, the collector groups do this; that is not the case for + # /lib/firmware.) + local d + install -d -m 755 /lib/firmware || return 1 + if [ "$(stat -c %U /lib/firmware 2>/dev/null)" != "$acct" ]; then + echo "# handing /lib/firmware to $acct (it owns the firmware)" + chown "$acct:$acct" /lib/firmware || return 1 + fi + if [ -z "$only" ]; then + # An EMPTY value is nearly always a quoting accident, not a request + # for ten gigabytes: `FIRMWARE_ONLY=amdgpu intel iwlwifi-cc-*` + # unquoted runs `intel` as a command and assigns nothing. Ask. + echo "!! FIRMWARE_ONLY is empty, which means ALL of linux-firmware" + echo " (over 10 GB). If that is really what you want, set" + echo " FIRMWARE_ALL=yes" + echo " in $CONF. If not, the value probably needs quoting:" + echo ' FIRMWARE_ONLY="amdgpu intel iwlwifi-cc-*"' + [ "${FIRMWARE_ALL:-no}" = yes ] || return 1 + echo "# installing ALL of linux-firmware (FIRMWARE_ALL=yes)" + su - "$acct" -c "cp -a '$src'/* /lib/firmware/" || return 1 + else + for d in $only; do + # a pattern matches files at the top level; a name is a directory + case "$d" in + *[*?[]*) + if ! su - "$acct" -c "ls -d $src/$d >/dev/null 2>&1"; then + _firmware_not_found "$src" "$d"; return 1 + fi + echo "# installing firmware: $d" + su - "$acct" -c "cp -a $src/$d /lib/firmware/" || return 1 ;; + *) + # ...the same help for a NAME. Only the pattern case + # searched, so `iwlwifi` (a directory that does not + # exist in this release) said "does not exist" and left + # the reader exactly where the pattern had. + [ -e "$src/$d" ] || { _firmware_not_found "$src" "$d"; return 1; } + echo "# installing firmware: $d" + su - "$acct" -c "cp -a '$src/$d' /lib/firmware/" || return 1 ;; + esac + done + fi + packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true + return 0 +} + +# FIRMWARE TROUBLE SHOULD NOT COST YOU THE KERNEL. +# +# This ran before the build and exited on failure, so a wrong FIRMWARE_ONLY +# meant no kernel was built at all -- and the message did not say so. A +# missing blob only BLOCKS the build when the config embeds it +# (CONFIG_EXTRA_FIRMWARE); otherwise it is a run-time concern, and the check +# further down catches the case that matters. +_fw_ok=1 +if [ "$FIRMWARE_FROM" != none ]; then + _install_firmware || _fw_ok=0 +fi +if [ "$_fw_ok" = 0 ]; then + if grep -q '^CONFIG_EXTRA_FIRMWARE="..*"$' "$tree/.config"; then + echo "!! this config builds firmware INTO the kernel, so the build" + echo " cannot proceed without it. Fix FIRMWARE_ONLY above." + exit 1 + fi + echo "!! carrying on WITHOUT the firmware above: this config does not" + echo " embed any (no CONFIG_EXTRA_FIRMWARE), so the kernel will build." + echo " The drivers that wanted it will fail at run time until it is" + echo " installed -- re-run lfs-kernel once FIRMWARE_ONLY is right." +fi + +# FIRMWARE BUILT INTO THE KERNEL MUST BE ON DISK. +# +# CONFIG_EXTRA_FIRMWARE embeds blobs in the image, so the build reads them +# from CONFIG_EXTRA_FIRMWARE_DIR (/lib/firmware by default). A config taken +# from a distribution kernel usually lists a dozen, and an LFS tree has no +# linux-firmware -- so the build dies deep in drivers/base, ten minutes in, +# with a message that names one file and no reason: +# No rule to make target '/lib/firmware/amdgpu/renoir_asd.bin' +# Check them all before compiling anything, and name every one that is +# missing. +_fw="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' "$tree/.config")" +if [ -n "$_fw" ]; then + _fwdir="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE_DIR="\(.*\)"$/\1/p' "$tree/.config")" + : "${_fwdir:=/lib/firmware}" + _missing_fw="" + for _f in $_fw; do + [ -f "$_fwdir/$_f" ] || _missing_fw="$_missing_fw $_f" + done + if [ -n "$_missing_fw" ]; then + echo "!! this config builds firmware INTO the kernel" + echo " (CONFIG_EXTRA_FIRMWARE), and these are not in $_fwdir:" + for _f in $_missing_fw; do echo " $_f"; done + echo " The build would fail in drivers/base after compiling for" + echo " several minutes. Two ways out:" + echo " * add them to FIRMWARE_ONLY in $CONF -- this tool installs" + echo " firmware itself (FIRMWARE_FROM=$FIRMWARE_FROM). The" + echo " subdirectory of each file above is what to list." + echo " * build without it:" + echo " cd $tree && ./scripts/config --set-str EXTRA_FIRMWARE ''" + echo " then run lfs-kernel again" + echo " the drivers then load it at RUN time from $_fwdir instead." + exit 1 + fi +fi + +# /lib/modules, /usr/include and the doc directory end up owned and recorded +# like any other package's files. +echo "# compiling ($JOBS) -- this is the long one; expect many minutes ..." +_as_pkg "make $JOBS" || exit 1 +_as_pkg "make modules_prepare" || exit 1 +echo "# installing modules into /lib/modules/$VERSION ..." +_as_pkg "make modules_install" || exit 1 + +# KERNEL_NAME is what the boot manager loads (rEFInd reads the file name); +# with ESP it is the pair that decides whether this is a test kernel or the +# real one. +[ -f "$ESP/$KERNEL_NAME" ] && cp -v "$ESP/$KERNEL_NAME" "$ESP/${KERNEL_NAME}_old" +cp -v "$tree/arch/x86/boot/bzImage" "$ESP/$KERNEL_NAME" +cp -v "$tree/arch/x86/boot/bzImage" "$ESP/$KERNEL_NAME-$VERSION" +cp -v "$tree/System.map" "$ESP/System.map-$VERSION" +cp -v "$tree/.config" "$ESP/config-$VERSION" +# SAY IT BEFORE IT TAKES A WHILE, NOT AFTER. +# +# Documentation is about ten thousand small files and the copy runs for a +# minute or two in complete silence; so does the manifest scan below, which +# walks the filesystem. A tool that goes quiet for two minutes is +# indistinguishable from one that has hung. +echo "# copying Documentation to /usr/share/doc/linux-$VERSION (~10k files, a minute or two) ..." +_as_pkg "install -d /usr/share/doc/linux-$VERSION" || exit 1 +_as_pkg "cp -r Documentation -T /usr/share/doc/linux-$VERSION" || exit 1 +# DEPMOD THE KERNEL WE BUILT, NOT THE ONE THAT IS RUNNING. +# +# `depmod -a` with no argument means `uname -r` -- 7.2.0, the kernel this +# chroot booted, whose /lib/modules does not exist here: +# depmod: ERROR: could not open directory /lib/modules/7.2.0 +# The modules that were just installed are 7.2.4's, and that is what needs +# an index. +depmod -a "$VERSION" || exit 1 + +# RECORD THE FILES NOW, WHILE THEY ARE INSTALLED. +# +# This used to run at the very end, after v4l2loopback and the headers -- so +# when the module build failed, the manifest was never written and `verify` +# reported "p_linux records its files: FAIL" about a kernel that was +# perfectly installed. The kernel is on disk at this point; record it. +# (Run again after the headers, which add /usr/include.) +_record() { + echo "# recording what $acct owns (scans the filesystem; this is slow) ..." + packagemanager reload-pkg-list "$PKG" >/dev/null 2>&1 || return 0 + local _h; _h="$(getent passwd "$acct" | cut -d: -f6)" + echo "# recorded $(grep -c . "$_h/pkg.lst" 2>/dev/null) file(s) for $acct" +} +_record + +# v4l2loopback (optional, out of tree) +case "$V4L2LOOPBACK" in yes|1|true) + d="$SRC/v4l2loopback" + # THE REPOSITORY IS THE PACKAGE USER'S, FROM THE FIRST COMMAND. + # + # git pull/clone ran as ROOT and the chown came afterwards, so the next + # command -- run as the package user -- met a repository owned by + # somebody else: + # fatal: detected dubious ownership in repository at '.../v4l2loopback' + # Owning it first and cloning as the account avoids the whole class, + # rather than papering over it with safe.directory. + install -d -o "$acct" -g "$acct" -m 755 "$d" || exit 1 + [ -d "$d/.git" ] && chown -R "$acct:$acct" "$d" 2>/dev/null + if [ -d "$d/.git" ]; then + su - "$acct" -c "cd '$d' && git pull --ff-only" || exit 1 + else + su - "$acct" -c "git clone --depth 1 '$V4L2LOOPBACK_GIT' '$d'" || exit 1 + fi + # BUILD IT AGAINST THE TREE, NOT AGAINST /lib/modules/$(uname -r). + # + # An out-of-tree module's own Makefile defaults to + # /lib/modules/`uname -r`/build, which is the RUNNING kernel: + # make[1]: *** /lib/modules/7.2.0/build: No such file or directory + # Driving kbuild from the tree directly sidesteps every such default and + # works for any module, not just this one. + su - "$acct" -c "make -C '$tree' M='$d' clean" || exit 1 + su - "$acct" -c "make -C '$tree' M='$d' modules" || exit 1 + su - "$acct" -c "make -C '$tree' M='$d' modules_install" || exit 1 + depmod -a "$VERSION" || exit 1 ;; +esac + +# THE HEADERS BELONG TO THE HEADERS PACKAGE. +# +# /usr/include/asm/*.h is owned by p_linux-headers -- the LFS step that +# installed them -- and /usr/include is sticky, so p_linux cannot replace a +# single one: +# cp: cannot create regular file '/usr/include/asm/types.h': Permission denied +# Installing a newer kernel's headers is an UPDATE OF THAT PACKAGE, not a +# second package writing over it. So the copy runs as that account, and its +# manifest is refreshed afterwards. (HEADERS=no in kernel.conf leaves +# /usr/include alone entirely, which is right when the running system's +# headers are deliberately older than the kernel being built.) +: "${HEADERS:=yes}" +: "${HEADERS_PKG:=linux-headers}" +case "$HEADERS" in + yes|1|true) + _hacct="p_$HEADERS_PKG" + if ! id "$_hacct" >/dev/null 2>&1; then + packagemanager add-user "$HEADERS_PKG" \ + || { echo "!! could not create $_hacct"; exit 1; } + lfs-helper fix-home "$_hacct" --run >/dev/null 2>&1 || true + fi + echo "# generating headers ..." + _as_pkg "make headers" || exit 1 + _as_pkg "find usr/include -type f ! -name '*.h' -delete" + # the tree is p_linux's; the destination is p_linux-headers'. Copy + # via a place both can read. + # ROOT COPIES, THEN HANDS THE FILES OVER. + # + # Running the copy AS p_linux-headers cannot work: the source is + # inside p_linux's home, which is 0750, so it cannot even stat it -- + # cp: cannot stat '.../linux-7.2.4/usr/include': Permission denied + # and opening one account's home to another would undo the point of + # having accounts. Root can read both sides; ownership afterwards is + # what makes the files the headers package's. Same shape as the ESP + # copies and the setuid bit. + echo "# installing headers into /usr/include (as root, owned by $_hacct) ..." + cp -r "$tree/usr/include" /usr || { + echo "!! could not install the headers into /usr/include" + echo " if some of them belong to a third package:" + echo " packagemanager reclaim $HEADERS_PKG --run" + exit 1; } + chown -R "$_hacct:$_hacct" /usr/include 2>/dev/null || true + packagemanager reload-pkg-list "$HEADERS_PKG" >/dev/null 2>&1 || true + echo "# headers installed and recorded for $_hacct" ;; + *) echo "# headers: skipped (HEADERS=$HEADERS)" ;; +esac + +# THE LAST WORKING SOURCE IS NEVER DELETED. +# +# A kernel that boots is the only way back from one that does not, and its +# TREE is what you need to rebuild or bisect -- not just the bzImage. The +# version that is running right now is recorded as last-good before this one +# is; nothing here ever removes that tree, and `lfs-kernel prune` (below) +# refuses to. +# ONLY A KERNEL THAT HAS BOOTED IS KNOWN TO BOOT. +# +# The old `elif` wrote the version just built into last-good whenever the +# file was empty -- so `verify` reported "last kernel known to boot: 7.2.4" +# about a kernel that has never run, while 7.2.0 was the one carrying the +# machine. A fallback that records a guess as a fact is worse than no +# record: the whole point of this file is to name something you can go back +# to. +_lastgood=/etc/pkgusr/kernel.last-good +_running="$(uname -r | sed 's/-.*//')" +if [ "$_running" = "$VERSION" ]; then + # the kernel just built is the one running: it demonstrably boots + printf '%s\n' "$VERSION" > "$_lastgood" + echo "# $VERSION is running, so it is now the kernel known to boot" +elif [ -d "$SRC/linux-$_running" ]; then + printf '%s\n' "$_running" > "$_lastgood" + echo "# keeping $SRC/linux-$_running as the last kernel known to boot" +else + echo "# the running kernel is $_running, and there is no tree for it here," + echo "# so nothing is recorded as known-good. $VERSION will be recorded" + echo "# the first time lfs-kernel runs while $VERSION is booted." + [ -s "$_lastgood" ] && echo "# (unchanged: $(cat "$_lastgood"))" +fi + + +_record + +echo +echo "# $VERSION installed as $ESP/$KERNEL_NAME" +echo "# previous kernel kept as $ESP/${KERNEL_NAME}_old" +echo "# its source kept at $SRC/linux-$(cat "$_lastgood" 2>/dev/null)" +echo "# owned by $acct (packagemanager verify | grep $PKG)" +echo "# running kernel is $(uname -r) -- reboot to use $VERSION" +[ "$_fw_ok" = 1 ] || echo "# NOTE: firmware was NOT installed (see the message above)" diff --git a/lfs-phases b/lfs-phases new file mode 100755 index 0000000..ced43ca --- /dev/null +++ b/lfs-phases @@ -0,0 +1,568 @@ +#!/bin/bash +# lfs-phases -- the shared phase runner sourced by every generated install +# script. The script holds what is specific to the package: its variables at +# the top and its commands in build_pkg / install_pkg / configure_pkg / +# test_pkg. Everything the same for every package lives here. +# +# A script ends with: +# . "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@" +# +# Variables a script may set (all optional unless noted): +# name_version binutils-2.45 (required) +# pkg_glob how to find the tarball in the sources store (LFS scripts) +# pkg the tarball's exact filename (BLFS scripts) +# link where to download it when it is not on disk +# md5_sum checksum of $pkg +# additional_links=(...) further files to fetch beside it +# is_git=1 git_ref=... clone $link instead of fetching a tarball +# pkg_unpack=no fetch but do not extract: the page's own commands do that +# pkgusr_stage cross | chroot | (unset = the booted system) +# +# Functions a script may define: +# prepare_pkg runs in unpack, before the tarball is fetched +# build_pkg configure + compile (required) +# install_pkg the step that writes into system directories (required) +# configure_pkg post-install configuration +# test_pkg the test suite; runs only when PKGUSR_TESTS=1 +# unpack_pkg replaces the built-in unpack entirely (rare) +# +# Environment the runner passes (all optional): +# PKGUSR_LIB, LFS_BUILD_ROOT, LFS_SOURCES_DIR, PKGUSR_TESTS (=1), +# PM_SKIP_MD5 (=1), PKGUSR_PHASE (internal: this process runs one phase) +# +# Every function and variable here carries a pkgusr_ / _pp_ prefix, because +# scripts are also SOURCED by lfs-helper to read their metadata, and this file +# comes along -- it must not shadow anything in the sourcing shell. + +PKGUSR_LIB_VERSION="1.14.146" + +_pp_build_id() { + local f="${BASH_SOURCE[0]}" + [ -r "$f" ] && md5sum "$f" 2>/dev/null | cut -c1-7 +} + +# ----------------------------------------------------------------- output ---- +_pp_say() { echo "$*"; } +_pp_note() { echo "note: $*" >&2; } +_pp_fail() { echo "!! $*" >&2; } + +# ---------------------------------------------------------------- layout ---- +# Where the sources are and where this package builds. Decided per process, +# from the environment the runner passed; nothing here is exported, because +# BUILD_ROOT is a name packages use (libvpx's Makefile has `BUILD_ROOT?=.`). +_pp_layout() { + # THE PACKAGE'S NAME, NOT THE FILE'S. + # + # The same script is run under several filenames by design -- the book + # copy (install_util-macros-1.20.2), the staged copy (install_util-macros) + # and the account's copy (install_p_util-macros) -- so a marker named + # after $0 was written by unpack under one name and looked for by install + # under another: + # !! no unpacked source in /usr/src/pkgusr/p_util-macros/src + # while the source sat right there. name_version is the package, and it + # is the same in every copy of the script. + _pp_step="${name_version:-${0##*/}}"; _pp_step="${_pp_step%.sh}" + if [ "${pkgusr_stage:-}" = "cross" ]; then + # chapters 5-6, as the lfs user on the host: the tree is $LFS + if [ -z "${LFS:-}" ]; then + _pp_fail '$LFS is not set -- refusing to guess. export LFS=/mnt/lfs (see book 2.6) and re-run.' + exit 2 + fi + export LFS + if [ -z "${LFS_TGT:-}" ]; then + LFS_TGT="$(uname -m)-lfs-linux-gnu"; export LFS_TGT + _pp_note "\$LFS_TGT was unset -- defaulting to $LFS_TGT (book 4.4)" + fi + case ":$PATH:" in + *":$LFS/tools/bin:"*) ;; + *) PATH="$LFS/tools/bin:$PATH"; export PATH + _pp_note "added $LFS/tools/bin to \$PATH (book 4.4)" ;; + esac + elif [ "${pkgusr_stage:-}" = "chroot" ]; then + # chapters 7-10, inside the chroot: the tree is / + export LFS="${LFS:-/}" + fi + if [ -z "${MAKEFLAGS:-}" ]; then + export MAKEFLAGS="-j$(nproc)" + fi + if [ -n "${LFS_SOURCES_DIR:-}" ]; then SOURCES_DIR="$LFS_SOURCES_DIR" + elif [ -n "${SOURCES_DIR:-}" ]; then : + elif [ "${pkgusr_stage:-}" = "cross" ]; then SOURCES_DIR="$LFS/sources" + else SOURCES_DIR="/sources"; fi + if [ -n "${LFS_BUILD_ROOT:-}" ]; then BUILD_ROOT="$LFS_BUILD_ROOT" + elif [ -n "${BUILD_ROOT:-}" ]; then : + elif [ "${pkgusr_stage:-}" = "cross" ]; then BUILD_ROOT="$LFS/build" + elif [ -d "$HOME/src" ]; then BUILD_ROOT="$HOME/src" + elif [ "${pkgusr_stage:-}" = "chroot" ]; then BUILD_ROOT="/build" + else BUILD_ROOT="$PWD"; fi + _pp_dir_marker="$BUILD_ROOT/.pkgusr-$_pp_step.dir" + _pp_cwd_marker="$BUILD_ROOT/.pkgusr-$_pp_step.cwd" + _pp_built_marker="$BUILD_ROOT/.pkgusr-$_pp_step.built" +} + +# A systemd book on a SysV system: the book's commands call systemctl and its +# family, and a missing command aborts the phase. Shim each one that is +# absent; on a systemd system nothing changes. +_pp_systemd_shims() { + local c + for c in systemctl systemd-run systemd-tmpfiles systemd-sysusers \ + systemd-machine-id-setup systemd-hwdb loginctl busctl journalctl \ + timedatectl hostnamectl localectl udevadm; do + command -v "$c" >/dev/null 2>&1 && continue + eval "$c() { echo \"# skipped (no $c on this system): $c \$*\"; }" + done +} + +# --------------------------------------------------------------- sources ---- +# Make BUILD_ROOT look like the sources directory to anything that walks up +# out of an unpacked tree: the LFS book writes +# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 +# from INSIDE the unpacked directory. Every downloaded file is linked, not a +# known list -- a list would be wrong the first time a new book adds one. +_pp_link_sources() { + [ -d "$SOURCES_DIR" ] || return 0 + local f b + for b in "$BUILD_ROOT"/*; do + [ -L "$b" ] && [ ! -e "$b" ] && rm -f "$b" + done + for f in "$SOURCES_DIR"/*; do + [ -f "$f" ] || continue + b="$BUILD_ROOT/${f##*/}" + # never shadow real content: an unpacked tree wins over a link + [ -e "$b" ] && [ ! -L "$b" ] && continue + ln -sfn "$f" "$b" 2>/dev/null || true + done +} + +# Put one file into BUILD_ROOT from wherever it already is on disk. Prints +# nothing; returns 1 when it is nowhere. +_pp_stage_file() { + local f="$1" d + [ -f "$BUILD_ROOT/$f" ] && return 0 + for d in "$HOME" "$SOURCES_DIR" /sources; do + [ -f "$d/$f" ] || continue + ln -sfn "$d/$f" "$BUILD_ROOT/$f" 2>/dev/null \ + || cp -p "$d/$f" "$BUILD_ROOT/$f" 2>/dev/null \ + || { cat "$d/$f" > "$BUILD_ROOT/$f" || rm -f "$BUILD_ROOT/$f"; } + [ -f "$BUILD_ROOT/$f" ] && return 0 + done + return 1 +} + +# Pick the tarball a glob names in the sources store. Two traps: patterns +# that match nothing must not swallow the one that did, and several packages +# ship docs beside the source (tcl8.6.16-html.tar.gz sorts before +# tcl8.6.16-src.tar.gz) -- prefer an explicit -src archive, never a doc one. +_pp_pick_glob() { + local _c + cd "$SOURCES_DIR" 2>/dev/null || return 1 + for _c in $pkg_glob; do + [ -e "$_c" ] || continue + case "$_c" in *-src.*|*-source.*) echo "$_c"; return 0 ;; esac + done + for _c in $pkg_glob; do + [ -e "$_c" ] || continue + case "$_c" in + *-html.*|*-doc.*|*-docs.*|*-man.*|*-manual.*|*-tests.*) continue ;; + *-doc-*|*-docs-*|*-html-*|*-man-pages-*|*-tests-*) continue ;; + esac + echo "$_c"; return 0 + done + return 1 +} + +# The GNU ftpmirror answers 502 more often than not, and a dead mirror asked +# three times is not a retry -- so the fallbacks are tried before giving up. +_pp_mirrors_for() { + local u="$1" f="${1##*/}" rest + printf '%s\n' "$u" + case "$u" in + *ftpmirror.gnu.org/*|*ftp.gnu.org/*) + rest="${u#*gnu.org/}"; rest="${rest#gnu/}" + printf '%s\n' "https://ftp.gnu.org/gnu/$rest" \ + "https://mirrors.kernel.org/gnu/$rest" \ + "https://mirror.csclub.uwaterloo.ca/gnu/$rest" ;; + esac + # the LFS mirror of last resort keeps most BLFS sources by filename + [ -n "${pkgusr_book_mirror:-}" ] && printf '%s\n' "${pkgusr_book_mirror%/}/$f" + return 0 +} + +_pp_fetch() { + local url="$1" try u + for try in 1 2 3; do + while IFS= read -r u; do + [ -n "$u" ] || continue + [ "$u" = "$url" ] || _pp_say "trying mirror: $u" + wget -4 -c "$u" && return 0 + done <<< "$(_pp_mirrors_for "$url" | awk '!seen[$0]++')" + [ "$try" = 3 ] && break + _pp_say "every mirror failed (try $try/3) -- retrying in $((try * 15))s" + sleep $((try * 15)) + done + return 1 +} + +_pp_check_md5() { + [ -n "${md5_sum:-}" ] || return 0 + grep -wq "$md5_sum" <<< "$(md5sum "$pkg")" && return 0 + if [ "${PM_SKIP_MD5:-0}" = 1 ]; then + _pp_fail "md5sum does NOT match, and PM_SKIP_MD5=1 -- continuing anyway" + return 0 + fi + _pp_fail "md5sum check failed for $pkg" + echo " expected $md5_sum" >&2 + echo " got $(set -- $(md5sum "$pkg" 2>/dev/null); echo "${1:-<no file>}")" >&2 + echo " To accept a mismatch on purpose: PM_SKIP_MD5=1 $0 unpack" >&2 + return 1 +} + +# ---------------------------------------------------------------- unpack ---- +_pp_unpack() { + mkdir -p "$BUILD_ROOT" 2>/dev/null + cd "$BUILD_ROOT" || { _pp_fail "no build scratch at $BUILD_ROOT and cannot create it"; return 1; } + rm -f "$_pp_dir_marker" "$_pp_cwd_marker" + case "${pkgusr_stage:-}" in cross|chroot) _pp_link_sources ;; esac + if declare -F prepare_pkg >/dev/null; then + prepare_pkg + # WHERE THE PAGE'S OWN COMMANDS LEFT OFF. + # + # Xorg Libraries writes lib-7.md5 in the build root, then `mkdir lib + # && cd lib` and downloads into it -- and the build loop reads + # `../lib-7.md5`, which only resolves from INSIDE lib. Returning to + # the build root here threw that away, so the build started one level + # too high and stopped on + # the list this build loops over is missing or empty: ../lib-7.md5 + # after downloading all thirty-two tarballs successfully. + _pp_prepared_in="$PWD" + cd "$BUILD_ROOT" || return 1 + fi + + if [ "${is_git:-}" = 1 ]; then + pkg_dir="$name_version" + if [ -d "$pkg_dir/.git" ]; then + git -C "$pkg_dir" fetch --all --tags --prune + else + rm -rf "$pkg_dir" && git clone --recurse-submodules "$link" "$pkg_dir" + fi + # A REF THAT DOES NOT EXIST IS A TYPO IN THE STACK, NOT A BUILD ERROR. + # + # git says only + # error: pathspec '1.0.0' did not match any file(s) known to git + # which reads like a build failure and sends people to the log. The + # repository is right here and knows what it does have; show it, and + # say where the wrong value came from. + if [ -n "${git_ref:-}" ] && ! git -C "$pkg_dir" checkout "$git_ref"; then + _pp_fail "'$git_ref' is not a tag, branch or commit in $link" + echo " The stack entry (or this script) asks for a ref the" >&2 + echo " repository does not have. Recent tags:" >&2 + git -C "$pkg_dir" tag --sort=-v:refname 2>/dev/null | head -8 \ + | sed 's/^/ /' >&2 + echo " Branches:" >&2 + git -C "$pkg_dir" branch -r --format='%(refname:short)' 2>/dev/null \ + | head -5 | sed 's/^/ /' >&2 + echo " Fix ref= in the stack entry for '${name:-$name_version}'," >&2 + echo " or set git_ref= in this script, then run it again." >&2 + return 1 + fi + git -C "$pkg_dir" submodule update --init --recursive + echo "$pkg_dir" > "$_pp_dir_marker" + _pp_say "checked out into $BUILD_ROOT/$pkg_dir" + return 0 + fi + + # the tarball: by exact name, by glob in the store, or downloaded + if [ -z "${pkg:-}" ] && [ -n "${pkg_glob:-}" ]; then + pkg="$(_pp_pick_glob)" || pkg="" + cd "$BUILD_ROOT" || return 1 + [ -n "$pkg" ] || { _pp_fail "no $pkg_glob in $SOURCES_DIR -- run: lfs build-system get-sources --run"; return 1; } + fi + if [ -z "${pkg:-}" ]; then + # a meta page: it builds a list of packages and its own commands fetch + # them. Remember where they left off -- Xorg Libraries does + # `mkdir lib && cd lib` and its build loop reads ../lib-7.md5 from there. + _pp_say "no single tarball -- this page builds a list of packages; its own commands fetch them" + printf '%s\n' "${_pp_prepared_in:-$PWD}" > "$_pp_cwd_marker" 2>/dev/null || true + [ -n "${_pp_prepared_in:-}" ] && [ "$_pp_prepared_in" != "$PWD" ] \ + && _pp_say "# the build resumes in $_pp_prepared_in" + return 0 + fi + _pp_stage_file "$pkg" || { [ -n "${link:-}" ] && _pp_fetch "$link"; } + [ -f "$pkg" ] || { _pp_fail "no $pkg in $BUILD_ROOT and nowhere to fetch it from"; return 1; } + _pp_check_md5 || return 1 + local li f + for li in "${additional_links[@]}"; do + f="${li##*/}" + _pp_stage_file "$f" || _pp_fetch "$li" || return 1 + done + if [ "${pkg_unpack:-yes}" = "no" ]; then + _pp_say "fetched $pkg (the page's own commands unpack it)" + printf '%s\n' "${_pp_prepared_in:-$PWD}" > "$_pp_cwd_marker" 2>/dev/null || true + return 0 + fi + + # Extract into a scratch dir, then move the ONE top-level directory to + # its name -- never `mv tmp/* .`, which nests it inside an existing + # same-named dir. --no-same-owner: as root, tar would restore the uids + # recorded in the archive. + rm -rf .pkgusr-tmp && mkdir .pkgusr-tmp && cd .pkgusr-tmp || return 1 + case "$pkg" in + *.zip) unzip -q ../"$pkg" ;; + *) tar --no-same-owner -xf ../"$pkg" ;; + esac || { cd ..; rm -rf .pkgusr-tmp; return 1; } + local n; n="$(ls -A | wc -l)" + if [ "$n" = 1 ] && [ -d "$(ls -A)" ]; then + pkg_dir="$(ls -A)" + cd .. && rm -rf "$pkg_dir" && mv ".pkgusr-tmp/$pkg_dir" "$pkg_dir" + else + pkg_dir="$name_version" + cd .. && rm -rf "$pkg_dir" && mv .pkgusr-tmp "$pkg_dir" + fi + rm -rf .pkgusr-tmp + echo "$pkg_dir" > "$_pp_dir_marker" + _pp_say "unpacked into $BUILD_ROOT/$pkg_dir" +} + +# ---------------------------------------------------------------- enter ---- +# The unpacked source directory, for build/install/configure/test. +_pp_enter_source() { + cd "$BUILD_ROOT" || return 1 + local d="" pkgdir="" + [ -f "$_pp_dir_marker" ] && pkgdir="$(cat "$_pp_dir_marker" 2>/dev/null)" + if [ -n "$pkgdir" ] && [ -d "$pkgdir" ]; then + cd "$pkgdir" || return 1 + return 0 + fi + # No marker, but the tree may still be here: one written by an older + # version of this runner, or by a copy of the script under another name. + # Look before giving up -- "run unpack" is the wrong answer when the + # unpacked source is sitting in front of us, and for a big package it + # throws away a compile that succeeded. + if [ -n "${name_version:-}" ] && [ -d "$name_version" ]; then + _pp_say "# found $BUILD_ROOT/$name_version (no marker) -- using it" + echo "$name_version" > "$_pp_dir_marker" 2>/dev/null || true + cd "$name_version" || return 1 + return 0 + fi + if [ -n "${pkg:-}${pkg_glob:-}" ]; then + local _only="" _c _n=0 + for _c in */; do + [ -d "$_c" ] || continue + _n=$((_n + 1)); _only="${_c%/}" + done + if [ "$_n" = 1 ]; then + _pp_say "# found $BUILD_ROOT/$_only (no marker) -- using it" + echo "$_only" > "$_pp_dir_marker" 2>/dev/null || true + cd "$_only" || return 1 + return 0 + fi + fi + if [ "${pkg_unpack:-yes}" = "no" ] \ + || { [ -z "${pkg:-}" ] && [ -z "${pkg_glob:-}" ] && [ "${is_git:-}" != 1 ]; }; then + # a meta page has nothing to enter; resume where its commands left off + [ -f "$_pp_cwd_marker" ] && d="$(cat "$_pp_cwd_marker" 2>/dev/null)" + [ -n "$d" ] && [ -d "$d" ] && cd "$d" + return 0 + fi + _pp_fail "no unpacked source in $BUILD_ROOT -- run: $0 unpack" + return 1 +} + +# Resume in the SAME subdir the build finished in (pkg_dir/build for meson, +# an out-of-tree build/ for glibc) -- the book's install commands assume it. +# A build that died part-way may have left no record: fall back to the usual +# out-of-tree layouts. +_pp_enter_build() { + _pp_enter_source || return 1 + local d + if [ -f "$_pp_cwd_marker" ]; then + d="$(cat "$_pp_cwd_marker" 2>/dev/null)" + [ -n "$d" ] && [ -d "$d" ] && { cd "$d"; return 0; } + fi + [ -e build.ninja ] || [ -e Makefile ] && return 0 + for d in build builddir _build obj; do + if [ -d "$d" ] && { [ -f "$d/Makefile" ] || [ -f "$d/build.ninja" ] \ + || [ -f "$d/config.status" ]; }; then + _pp_say "# resuming in ./$d (no build marker was recorded)" + cd "$d"; return 0 + fi + done + return 0 +} + +# ----------------------------------------------------------------- phases ---- +# Each phase runs as its OWN process: bash disables `set -e` inside any +# function reached from a tested context (an && chain, an || handler), and +# that suppression is inherited by subshells -- so a failing make would be +# silently ignored. A separate process has its own -e state that nothing +# upstream can suppress. The process is this script again, with +# PKGUSR_PHASE naming the one phase to run, and stdin closed so a book command +# that reads stdin (Expect's PTY check) cannot swallow anything. +_pp_child() { + PKGUSR_PHASE="$1" bash "$_pp_self" </dev/null +} + +_pp_phase_unpack() { + if declare -F unpack_pkg >/dev/null; then unpack_pkg; else _pp_unpack; fi +} + +_pp_phase_build() { + _pp_enter_source || return 1 + # remove a stale CONFIGURED build dir so re-running "build" works, but + # never a source dir that merely happens to be named build/: only the one + # the last build attempt recorded, or one holding build-system artifacts + local d here; here="$PWD" + if [ -f "$_pp_cwd_marker" ]; then + d="$(cat "$_pp_cwd_marker" 2>/dev/null)" + case "$d" in "$here"/?*) [ -d "$d" ] && rm -rf "$d" ;; esac + fi + for d in build builddir _build; do + if [ -d "$d" ] && { [ -f "$d/build.ninja" ] || [ -f "$d/CMakeCache.txt" ] \ + || [ -d "$d/meson-private" ] || [ -f "$d/config.status" ]; }; then + rm -rf "$d" + fi + done + # Record where the build ends -- even when it fails part-way, so install + # and configure can still find it. A failure that never left the source + # dir keeps the previous record: that is where the build subdir is. + # + # And record SEPARATELY that the build actually finished. Without that, + # running a later phase on a source tree whose build never completed dies + # inside the package instead of here: + # make: *** No rule to make target 'install'. Stop. + # -- which says nothing about the real problem, that there is no Makefile + # because configure never ran. + rm -f "$_pp_built_marker" 2>/dev/null + _pp_here="$PWD" + trap '_pp_build_exit' EXIT + build_pkg +} + +_pp_build_exit() { + local rc=$? + if [ "$rc" = 0 ] || [ "$PWD" != "$_pp_here" ]; then + pwd > "$_pp_cwd_marker" 2>/dev/null || true + fi + [ "$rc" = 0 ] && { : > "$_pp_built_marker" 2>/dev/null || true; } + return 0 +} + +# Say it BEFORE the package's own tools fail in their own words. A warning, +# not a refusal: a tree built by an older runner has no record either, and +# refusing there would be wrong. +_pp_warn_unbuilt() { + [ -f "$_pp_built_marker" ] && return 0 + [ -f "$_pp_cwd_marker" ] && return 0 + _pp_fail "nothing here recorded a finished build phase for $name_version." + echo " If the next lines are 'No rule to make target' or a missing" >&2 + echo " Makefile, the build never completed and no later phase can" >&2 + echo " stand in for it. Run it first:" >&2 + echo " $0 build" >&2 + return 0 +} + +_pp_phase_install() { + _pp_enter_build || return 1 + _pp_warn_unbuilt + install_pkg +} + +_pp_phase_configure() { + declare -F configure_pkg >/dev/null || return 0 + _pp_enter_build || return 1 + configure_pkg +} + +_pp_phase_test() { + _pp_enter_build || return 1 + if declare -F test_pkg >/dev/null; then + test_pkg + elif [ -f build.ninja ]; then meson test || ninja test + elif [ -f Makefile ]; then make check || make test + else _pp_say "no recognized test target -- define test_pkg() for this package" + fi +} + +_pp_run_tests() { + case "${PKGUSR_TESTS:-${LFS_RUN_TESTS:-${PM_TEST:-0}}}" in 1) return 0 ;; esac + return 1 +} + +# The books expect some tests to fail, so the result is reported, not fatal: +# read the summary and judge for yourself. +_pp_test_step() { + local rc=0 + _pp_child test || rc=$? + if [ $rc -ne 0 ]; then + _pp_fail "test suite exited $rc -- check the summary above." + echo " Some failures are normal (the book lists the usual suspects);" >&2 + echo " this does NOT stop the install." >&2 + fi + return 0 +} + +_pp_phase_fail() { + echo "!! phase '$1' FAILED (exit $2)" >&2 + exit "$2" +} + +_pp_usage() { + cat <<EOF +usage: $0 {all|unpack|build|test|install|configure|update} + all full run (default); set PKGUSR_TESTS=1 to also run tests + unpack find or fetch the tarball and extract it + build configure + compile (needs unpack) + test run the test suite (needs build) + install install step only -- re-run after fixing a failure, without recompiling + configure post-install configuration + update unpack + build + install, no reconfigure (keeps your config) +EOF +} + +# The entry point a script calls last. Does nothing when the script was only +# SOURCED (lfs-helper reads the variables that way). +pkgusr_run() { + if [ "${#BASH_SOURCE[@]}" -lt 2 ]; then + # piped on stdin or bash -c: each phase re-runs the script as a file, + # so it must BE a file + _pp_fail "run an install script by path (bash install_x all), not on stdin" + exit 2 + fi + [ "${BASH_SOURCE[1]}" = "$0" ] || return 0 # sourced: metadata only + [ -n "${name_version:-}" ] || { _pp_fail "$0 sets no name_version"; exit 2; } + _pp_self="$(readlink -f "$0")" + _pp_layout + _pp_systemd_shims + if [ -n "${PKGUSR_PHASE:-}" ]; then + # the child process: one phase, at top level, under set -e + set -e + "_pp_phase_$PKGUSR_PHASE" + exit $? + fi + local phase="${1:-all}" p rc + case "$phase" in + all|update) + for p in unpack build; do + _pp_child "$p" || _pp_phase_fail "$p" $? + done + _pp_run_tests && _pp_test_step + _pp_child install || _pp_phase_fail install $? + [ "$phase" = all ] && { _pp_child configure || _pp_phase_fail configure $?; } + ;; + unpack|build|install|configure) + _pp_child "$phase" || _pp_phase_fail "$phase" $? ;; + test) + _pp_test_step ;; + *) _pp_usage; exit 2 ;; + esac + return 0 +} + +# Run directly, this file only identifies itself. +if [ "${BASH_SOURCE[0]}" = "$0" ]; then + case "${1:-}" in + --version) echo "lfs-phases $PKGUSR_LIB_VERSION (build $(_pp_build_id))" ;; + *) sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' ;; + esac + exit 0 +fi diff --git a/lfs-pkgusr.zip b/lfs-pkgusr.zip new file mode 100644 index 0000000..72bdaf5 Binary files /dev/null and b/lfs-pkgusr.zip differ diff --git a/lfs-sanity.sh b/lfs-sanity.sh index 5fe120e..85f4cf3 100644 --- a/lfs-sanity.sh +++ b/lfs-sanity.sh @@ -25,8 +25,17 @@ PFX="${PFX:-p}"; PFX="${PFX%_}_" sec() { echo; echo "=============================================================="; \ echo "== $*"; echo "=============================================================="; } +# One colour rule, same as every other tool here: red is a problem, green is a +# clean result, plain is information. The report is often read in a hurry, +# scrolled fast, or grepped -- a problem that is not red is a problem missed. +# Colour only on a terminal, so the saved report file stays clean text. +if [ -t 1 ]; then + _C_ERR=$'\033[0;31m'; _C_OK=$'\033[0;32m'; _C_OFF=$'\033[0m' +else + _C_ERR=""; _C_OK=""; _C_OFF="" +fi note(){ echo " $*"; } -hit() { echo "!! $*"; hits=$((hits+1)); } +hit() { echo "${_C_ERR}!! $*${_C_OFF}"; hits=$((hits+1)); } echo "lfs-pkgusr sanity report -- $(date '+%Y-%m-%d %H:%M:%S')" echo "host: $(uname -srm)" @@ -35,7 +44,7 @@ echo "roots: PKGROOT=$PKGROOT CFGROOT=$CFGROOT STATE=$STATE" sec "0. tool versions (host and chroot copies must MATCH)" # A stale chroot copy running old code has cost more debugging time in this # project than anything else, which is why each tool prints a build id. -for t in lfs-helper lfs packagemanager blfs packagemanager_install; do +for t in lfs-helper lfs packagemanager blfs; do if command -v "$t" >/dev/null 2>&1; then printf ' %-22s %s\n' "$t" \ "$("$t" --version 2>/dev/null | head -1 || echo '(no --version)')" @@ -177,9 +186,17 @@ done sec "5. files owned by a uid with NO name" # The sharp version of the /sources problem: a uid that resolves to nothing is # unambiguous, whereas a uid that resolves to the WRONG name looks fine. -# /tmp and /build are scratch: nothing there has to have an owner, and both -# are full of half-unpacked trees mid-build. -o="$(find / -xdev \( -path /tmp -o -path /build -o -path /sources \) -prune \ +# /tmp, /build, /sources AND every package's unpacked source tree are scratch: +# nothing there has to have an owner. Build trees moved from /build into each +# package user's home, and this scan did not follow -- so a finished tree filled +# the report with tcl's own documentation: +# !! files with no owner (first 40): +# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm +# A tarball can carry any uid it likes. Unpacked sources are not installed, +# nothing owns them, and asking who does has no answer. +o="$(find / -xdev \( -path /tmp -o -path /build -o -path /sources \ + -o -path "$STATE" \ + -o -path "$PKGROOT/*/src" -o -path "$CFGROOT/*/src" \) -prune \ -o -xdev \( -nouser -o -nogroup \) -print 2>/dev/null | head -40)" if [ -n "$o" ]; then hit "files with no owner (first 40):"; printf ' %s\n' $o @@ -193,9 +210,19 @@ getent group install 2>/dev/null | sed 's/^/ /' \ getent group install 2>/dev/null | awk -F: '$3!=9999{print "!! install is gid "$3", want 9999"}' c="$(getent group 2>/dev/null | awk -F: '$3>=90000' | wc -l)" note "$c collector group(s) in the 90000+ range" -# same cap as the accounts: nogroup is 65534 and belongs to the system -getent group 2>/dev/null | awk -F: '$3>=10000 && $3<65000 && $1 !~ /^p_|^u_|^cfg_/ \ - {print "!! group "$1" (gid "$3") is outside every convention"}' +# same cap as the accounts: nogroup is 65534 and belongs to the system. +# +# COUNTED, not just printed. This awk wrote its own "!!" line and never +# touched $hits, so a report that flagged +# !! group nimgnu_p_openssl (gid 10093) is outside every convention +# ended with "no problems found by these checks". A summary that disagrees +# with the body teaches you to skip the summary. +_badgrp="$(getent group 2>/dev/null \ + | awk -F: '$3>=10000 && $3<65000 && $1 !~ /^p_|^u_|^cfg_/ {print $1" (gid "$3")"}')" +if [ -n "$_badgrp" ]; then + hit "group(s) outside every convention -- collector groups belong at 90000+:" + printf ' %s\n' $_badgrp +fi sec "7. how much of the tree still belongs to root" # Root-owned files under the install dirs are normally the temporary system's, @@ -214,7 +241,19 @@ sec "8. build state" # will never close)` on a tree where adoption had completed perfectly. PROG="$STATE/progress" [ -f "$PROG/steporder" ] && note "steps in order: $(grep -c . "$PROG/steporder")" -[ -f "$PROG/steps-built" ] && note "steps built: $(grep -c . "$PROG/steps-built")" +if [ -f "$PROG/steps-built" ]; then + _nb="$(grep -c . "$PROG/steps-built")" + _no="$(grep -c . "$PROG/steporder" 2>/dev/null || echo 0)" + note "steps built: $_nb" + # More built than exist means the progress file remembers steps the current + # step order no longer has -- usually one removed from the tools since this + # tree was generated (last-step, refind). Harmless, but it makes "104 of + # 105" arithmetic wrong for the rest of the build. + if [ "$_nb" -gt "$_no" ] 2>/dev/null; then + note " ($((_nb - _no)) built step(s) are not in the current order --" + note " left over from a step removed since; clear with: lfs-helper undone <name>)" + fi +fi [ -d "$STATE/manifests" ] && note "manifests: $(ls -1 "$STATE/manifests"/*.files 2>/dev/null | wc -l)" [ -f "$PROG/adopted.list" ] \ && note "adopted.list: $(grep -c . "$PROG/adopted.list") entries" \ @@ -248,7 +287,15 @@ for m in "$STATE"/manifests/*.files; do command -v lfs-helper >/dev/null 2>&1 && [ -e "$_f" ] \ && _own="$(stat -c %U "$_f" 2>/dev/null)" case "$_own" in - p_*|cfg_*) note "manifest '$b' -> no account '$a', but its files belong to '$_own'" ;; + p_*|cfg_*) + note "manifest '$b' -> no account '$a', but its files belong to '$_own'" ;; + root) + # A configuration step that writes ROOT-owned files. cfg_clock + # writes /etc/adjtime, cfg_hosts writes /etc/hosts: root's files, + # by design, and the step has no account because it installs no + # software. There is nothing here that adoption could ever want, + # so reporting it listed eleven findings on a perfect tree. + : ;; *) echo " ? manifest '$b' -> no account '$a'" ;; esac done @@ -257,9 +304,9 @@ command -v lfs-helper >/dev/null 2>&1 && { echo; echo "--- lfs-helper list ---"; sec "SUMMARY" if [ "$hits" -eq 0 ]; then - echo " no problems found by these checks" + echo " ${_C_OK}no problems found by these checks${_C_OFF}" else - echo " $hits problem area(s) flagged above -- search this file for '!!'" + echo " ${_C_ERR}$hits problem area(s) flagged above -- search this file for '!!'${_C_OFF}" fi echo echo "Nothing was changed. This report is read-only." diff --git a/lfs-sysvbook b/lfs-sysvbook new file mode 100755 index 0000000..b6ff448 --- /dev/null +++ b/lfs-sysvbook @@ -0,0 +1,641 @@ +#!/usr/bin/env python3 +# lfs-sysvbook -- graft the SysV init flavour onto a systemd-only LFS book. +# +# LFS 13.0 dropped the SysV edition. The difference between the two flavours +# was never the 80+ packages -- it is five SysV-only sections, two +# systemd-only sections, chapter 9, the fstab, and a handful of configure +# flags. This tool takes the last SysV book (the DONOR, e.g. 12.4) and a new +# systemd book (the TARGET, e.g. 13.0-systemd) and produces a hybrid book +# plus its wget-list and md5sums, stored under a new version label so the +# `lfs` tool builds from it like any other book: +# +# lfs fetch 12.4 +# lfs fetch 13.0-systemd +# lfs-sysvbook make # -> 13.0-sysv in the lfs store +# lfs-sysvbook check --book 13.0-sysv +# lfs --book 13.0-sysv run +# +# What `make` does to the target book: +# ch8: Systemd section -> donor "Udev from Systemd" (pinned donor version) +# D-Bus section -> removed (BLFS provides it on SysV) +# Sysklogd,SysVinit-> inserted after E2fsprogs, from the donor +# ch9: replaced wholesale with the donor's (bootscripts, sysv network, ...) +# 10.2 fstab -> donor's (SysV mounts /proc, /sys, ... via fstab) +# 7.6 passwd/group -> systemd-* account lines filtered out +# flags: Man-DB, Procps-ng, Util-linux get their SysV configure flags +# wget-list/md5sums: target's, minus removed sections' files, plus the +# donor files the transplanted sections reference (URLs and MD5s are +# read from each book's own chapter 3, so no network is needed) +# +# Transplanted sections keep the DONOR's package versions (e.g. udev from +# systemd-257.8) -- those are the instructions that are known to work. +# Everything else builds at the target's versions. + +import argparse +import hashlib +import os +import re +import sys + +try: + from bs4 import BeautifulSoup +except ImportError: + sys.stderr.write( + "lfs-sysvbook needs beautifulsoup4 (same as the lfs tool):\n" + " pip install beautifulsoup4\n" + " or on the built system:\n" + " packagemanager pip install beautifulsoup4\n") + sys.exit(2) + +VERSION = "1.0.0" + + +def _build_id(): + """Fingerprint of this file's own contents -- cannot go stale.""" + try: + with open(os.path.abspath(__file__), "rb") as fh: + return hashlib.md5(fh.read()).hexdigest()[:7] + except OSError: + return "unknown" + + +# --------------------------------------------------------------------------- # +# the store -- same layout as the lfs tool +# --------------------------------------------------------------------------- # +def store_dir(): + return os.environ.get("LFS_STORE") or "/usr/share/lfs" + + +def books_dir(): + return os.path.join(store_dir(), "books") + + +def book_path(ver): + return os.path.join(books_dir(), f"LFS-BOOK-{ver}-NOCHUNKS.html") + + +def wget_list_path(ver): + return os.path.join(books_dir(), f"wget-list-{ver}.txt") + + +def md5sums_path(ver): + return os.path.join(books_dir(), f"md5sums-{ver}.txt") + + +def load_soup(path): + with open(path, "rb") as f: + html = f.read() + for parser in ("lxml", "html.parser"): + try: + return BeautifulSoup(html, parser) + except Exception: + continue + return BeautifulSoup(html, "html.parser") + + +# --------------------------------------------------------------------------- # +# what makes a SysV book a SysV book -- all flavour knowledge lives here +# --------------------------------------------------------------------------- # +# Sections removed from the target (systemd-only), by anchor id. +REMOVE = ["ch-system-dbus"] + +# Target section replaced by a donor section: (target id, donor id). +REPLACE = [ + ("ch-system-systemd", "ch-system-udev"), # udev built from systemd src + ("chapter-config", "chapter-config"), # whole chapter 9 + ("ch-bootable-fstab", "ch-bootable-fstab"), # SysV mounts via fstab +] + +# Donor sections inserted after a target section: (after-target-id, [donor ids]). +INSERT_AFTER = [ + ("ch-system-e2fsprogs", ["ch-system-sysklogd", "ch-system-sysvinit"]), +] + +# Configure-flag edits inside target sections that exist in both flavours: +# (anchor id, description, pattern, replacement, done-marker). +# The marker makes each rule idempotent and lets `check` verify it took. +FLAG_PATCHES = [ + ("ch-system-procps", + "Procps-ng: drop --with-systemd", + r"[ \t]*\\\n[ \t]*--with-systemd\b", + "", + None), # done when the pattern no longer matches + ("ch-system-man-db", + "Man-DB: do not install systemd units/tmpfiles", + r"(--with-grap=\S+)", + r"\1 \\\n" + r" --with-systemdtmpfilesdir= \\\n" + r" --with-systemdsystemunitdir=", + "--with-systemdtmpfilesdir"), + ("ch-system-util-linux", + "Util-linux: build without systemd support", + r"(?m)^([ \t]*)(ADJTIME_PATH=)", + r"\1--without-systemd \\\n" + r"\1--without-systemdsystemunitdir \\\n" + r"\1\2", + "--without-systemd"), +] + +# 7.6: account lines that only a systemd system needs. +CREATEFILES_ID = "ch-tools-createfiles" +SYSTEMD_ACCOUNT_LINE = re.compile(r"(?m)^systemd-[^\n]*\n?") + +FILE_TOKEN = re.compile(r"([A-Za-z0-9_.+-]+\.(?:tar\.[a-z0-9]+|tgz|patch))") + + +# --------------------------------------------------------------------------- # +# DOM helpers -- a "unit" is a section's titlepage div + its content div +# --------------------------------------------------------------------------- # +def unit(soup, anchor_id, book_label): + """(titlepage div, content div) of the section whose heading carries + <a id=anchor_id>. In NOCHUNKS books these are two sibling divs.""" + a = soup.find("a", id=anchor_id) + if a is None: + raise SystemExit(f"! section '{anchor_id}' not found in {book_label}.\n" + f" The book layout changed -- this tool needs an " + f"update for it.") + tp = None + for p in a.parents: + if p.name == "div" and "titlepage" in (p.get("class") or []): + tp = p + break + if tp is None: + raise SystemExit(f"! '{anchor_id}' in {book_label} has no titlepage " + f"div -- unexpected book layout.") + content = tp.find_next_sibling("div") + if content is None or "titlepage" in (content.get("class") or []): + raise SystemExit(f"! '{anchor_id}' in {book_label} has no content " + f"div -- unexpected book layout.") + return tp, content + + +def unit_text(tp, content): + return tp.get_text(" ", strip=True) + "\n" + content.get_text("\n") + + +def title_of(tp): + return re.sub(r"\s+", " ", + tp.get_text(" ", strip=True)).replace("\u00a0", " ") + + +def _title_token(tp): + """Name-Version token from the section heading, same rule as `lfs`: + first whitespace token that ends in -<digits...>.""" + t = tp.get_text(" ", strip=True).replace("\u00a0", " ") + t = re.sub(r"^\d+(\.\d+)*\.\s*", "", t.strip()) + tok_re = re.compile(r"^[A-Za-z][A-Za-z0-9+_.]*" + r"(?:(?:::|[-_])[A-Za-z][A-Za-z0-9+_.]*)*" + r"[-_][0-9][0-9.]*(?:-[0-9]+)*$") + for tok in t.split(): + if tok_re.match(tok): + return tok + return None + + +def files_of(tp, content): + """Tarballs/patches a section needs: its title token, the title tokens of + any sections inside it (a transplanted chapter 9 carries LFS-Bootscripts + only in a heading), and every file name its text mentions.""" + toks = set() + t = _title_token(tp) + if t: + toks.add(("token", t)) + for h in content.find_all(re.compile(r"^h\d$")): + t = _title_token(h) + if t: + toks.add(("token", t)) + for f in FILE_TOKEN.findall(unit_text(tp, content)): + toks.add(("file", os.path.basename(f))) + return toks + + +def _norm(s): + return re.sub(r"[^a-z0-9]", "", s.lower()) + + +def match_urls(url_map, wanted, label): + """URLs from {basename: (url, md5)} that a section's tokens refer to. + A 'file' token matches its exact basename; a 'token' (Name-Version) + matches a basename whose stem normalises to the same string.""" + def stem(name): + s = re.sub(r"\.(tar\.[a-z0-9]+|tgz|patch)$", "", name) + return _norm(re.sub(r"-src$", "", s)) + + out, missing = {}, [] + for kind, val in sorted(wanted): + hit = None + if kind == "file" and val in url_map: + hit = val + else: + # a title token, or prose naming the tarball with the wrong + # compression suffix (12.4 prose says .tar.xz, github ships .gz) + for base in url_map: + if stem(base) == stem(val): + hit = base + break + if hit: + out[hit] = url_map[hit] + elif kind == "file": + missing.append((val, label)) + return out, missing + + +# --------------------------------------------------------------------------- # +# chapter 3 of each book carries every download URL and MD5 -- no network +# --------------------------------------------------------------------------- # +def url_map_from_book(path): + """{basename: (url, md5)} from the book's own package/patch lists.""" + with open(path, encoding="utf-8", errors="replace") as f: + html = f.read() + urls = [(m.start(), m.group(1)) for m in + re.finditer(r'href=\s*"(https?://[^"]+?' + r'\.(?:tar\.[a-z0-9]+|tgz|patch))"', html)] + md5s = [(m.start(), m.group(1)) for m in + re.finditer(r'MD5 sum:\s*(?:<[^>]+>\s*)*([0-9a-f]{32})', html)] + out, mi = {}, 0 + for i, (upos, url) in enumerate(urls): + nxt = urls[i + 1][0] if i + 1 < len(urls) else len(html) + while mi < len(md5s) and md5s[mi][0] < upos: + mi += 1 + md5 = md5s[mi][1] if mi < len(md5s) and md5s[mi][0] < nxt else None + out.setdefault(os.path.basename(url), (url, md5)) + return out + + +# --------------------------------------------------------------------------- # +# mini command parser -- same rules as `lfs` (for check + patching) +# --------------------------------------------------------------------------- # +def _is_admonition(tag): + cls = set(getattr(tag, "get", lambda _k: None)("class") or []) + return bool({"admon", "note", "tip", "caution", "warning", + "important"} & cls) + + +def runnable_commands(scope): + """Command blocks the lfs tool would execute from this tag's subtree.""" + out = [] + for pre in scope.find_all("pre"): + classes = pre.get("class") or [] + if not any(c in classes for c in ("userinput", "root", "install")): + continue + if any(_is_admonition(p) for p in pre.parents): + continue + out.append(pre.get_text("").replace("\u00a0", " ").rstrip()) + return out + + +def patch_pre_text(scope, pattern, repl): + """Apply a regex to the text nodes of runnable command blocks.""" + n = 0 + for pre in scope.find_all("pre"): + classes = pre.get("class") or [] + if not any(c in classes for c in ("userinput", "root", "install")): + continue + for node in list(pre.find_all(string=True)): + new = re.sub(pattern, repl, str(node)) + if new != str(node): + node.replace_with(new) + n += 1 + return n + + +# --------------------------------------------------------------------------- # +# make +# --------------------------------------------------------------------------- # +def resolve_input(ver, file_arg, what): + if file_arg: + if not os.path.isfile(file_arg): + raise SystemExit(f"! {what} file not found: {file_arg}") + return os.path.basename(file_arg), file_arg + p = book_path(ver) + if not os.path.isfile(p): + raise SystemExit(f"! {what} book {ver} is not in the store.\n" + f" lfs fetch {ver}\n" + f" or pass a local file with --{what}-file.") + return ver, p + + +def cmd_make(args): + d_label, d_path = resolve_input(args.donor, args.donor_file, "donor") + t_label, t_path = resolve_input(args.target, args.target_file, "target") + out_ver = args.out or (t_label.replace("-systemd", "") + "-sysv" + if "-systemd" in t_label else t_label + "-sysv") + print(f"donor : {d_label} ({d_path})") + print(f"target: {t_label} ({t_path})") + print(f"out : {out_ver}") + + donor = load_soup(d_path) + target = load_soup(t_path) + warnings = [] + + # -- collect the file needs BEFORE surgery moves nodes around ----------- # + donor_urls = url_map_from_book(d_path) + target_urls = url_map_from_book(t_path) + + drop_files = set() + for tid in REMOVE + [t for t, _d in REPLACE if t != "chapter-config"]: + # chapter-config (ch9) references no downloads in the systemd book + drop_files |= files_of(*unit(target, tid, t_label)) + dropped, _ = match_urls(target_urls, drop_files, t_label) + + add_files = set() + for _tid, did in REPLACE: + add_files |= files_of(*unit(donor, did, d_label)) + for _after, dids in INSERT_AFTER: + for did in dids: + add_files |= files_of(*unit(donor, did, d_label)) + added, missing = match_urls(donor_urls, add_files, d_label) + added = {b: v for b, v in added.items() + if b not in target_urls or b in dropped} + + # -- surgery ------------------------------------------------------------ # + print("\nsections:") + for tid in REMOVE: + tp, content = unit(target, tid, t_label) + title = title_of(tp) + tp.decompose() + content.decompose() + print(f" removed {title} [{tid}]") + + for tid, did in REPLACE: + t_tp, t_content = unit(target, tid, t_label) + d_tp, d_content = unit(donor, did, d_label) + title_old = title_of(t_tp) + title_new = title_of(d_tp) + t_tp.replace_with(d_tp.extract()) + t_content.replace_with(d_content.extract()) + print(f" replaced {title_old}\n" + f" -> {title_new} [{tid}]") + + for after_id, dids in INSERT_AFTER: + _tp, anchor_content = unit(target, after_id, t_label) + at = anchor_content + for did in dids: + d_tp, d_content = unit(donor, did, d_label) + title = title_of(d_tp) + at.insert_after(d_content.extract()) + at.insert_after(d_tp.extract()) + at = d_content + print(f" inserted {title} [after {after_id}]") + + # -- 7.6: filter systemd account lines out of /etc/passwd, /etc/group -- # + _tp, cf = unit(target, CREATEFILES_ID, t_label) + n = patch_pre_text(cf, SYSTEMD_ACCOUNT_LINE, "") + print(f"\n7.6 passwd/group: systemd-* account lines removed " + f"({n} heredoc(s) touched)" if n else + "\n7.6 passwd/group: nothing to remove") + if not n: + warnings.append("7.6: no systemd-* account lines found -- " + "book layout may have changed") + + # -- configure-flag patches --------------------------------------------- # + print("flags:") + for aid, desc, pat, repl, marker in FLAG_PATCHES: + _tp, content = unit(target, aid, t_label) + text = "\n".join(runnable_commands(content)) + if marker and marker in text: + print(f" ok {desc} (already applied)") + continue + n = patch_pre_text(content, pat, repl) + if n: + print(f" patched {desc}") + else: + print(f" !! NOT applied: {desc}") + warnings.append(f"flag patch did nothing: {desc} -- check the " + f"section by hand") + + # -- stale TOC entries pointing at removed sections --------------------- # + ids = {t.get("id") for t in target.find_all(id=True)} + pruned = 0 + for a in target.find_all("a", href=True): + href = a["href"] + if href.startswith("#") and href[1:] not in ids: + dt = a.find_parent("dt") + if dt is not None: + dt.decompose() + pruned += 1 + if pruned: + print(f"toc: {pruned} entries for removed sections pruned") + + # -- serialize + relabel ------------------------------------------------ # + # The target's release label is read from the book itself (the + # `echo <label> > /etc/lfs-release` command), so a --target-file with any + # filename still gets /etc/lfs-release, os-release and the titles renamed. + html = str(target) + m = re.search(r"echo\s+([0-9][\w.+-]*)\s*(?:>|>)\s*/etc/lfs-release", + html) + if m and m.group(1) != out_ver: + html = html.replace(m.group(1), out_ver) + print(f"relabelled: {m.group(1)} -> {out_ver}") + elif not m: + warnings.append("could not find the release label in the target " + "book -- titles and /etc/lfs-release keep its name") + + # -- wget-list + md5sums ------------------------------------------------ # + # Chapter 3 sometimes offers an alternative download (linux-6.18.tar.xz + + # patch instead of linux-6.18.10.tar.xz); alternatives carry no MD5. Keep + # an MD5-less entry only when no checksummed entry for the same package + # exists. + def name_stem(base): + return re.match(r"[A-Za-z_+-]+", base).group(0).lower() + + entries = [(b, u, m) for b, (u, m) in target_urls.items() + if b not in dropped] + entries += [(b, u, m) for b, (u, m) in added.items()] + with_md5 = {name_stem(b) for b, _u, m in entries if m} + wget, md5 = [], [] + for base, url, m in entries: + if not m and name_stem(base) in with_md5: + print(f" (skipped alternative download: {base})") + continue + wget.append(url) + if m: + md5.append(f"{m} {base}") + + print("\ndownloads:") + for b in sorted(dropped): + print(f" - {b}") + for b in sorted(added): + print(f" + {b}") + for f, where in missing: + warnings.append(f"'{f}' is referenced but has no download URL in " + f"{where} -- add it to the wget-list by hand if the " + f"build asks for it") + + # -- write -------------------------------------------------------------- # + outdir = args.outdir or books_dir() + try: + os.makedirs(outdir, exist_ok=True) + except PermissionError: + raise SystemExit(f"! cannot create {outdir} -- run as root or pass " + f"--outdir somewhere writable.") + paths = { + "book": os.path.join(outdir, f"LFS-BOOK-{out_ver}-NOCHUNKS.html"), + "wget-list": os.path.join(outdir, f"wget-list-{out_ver}.txt"), + "md5sums": os.path.join(outdir, f"md5sums-{out_ver}.txt"), + } + try: + with open(paths["book"], "w", encoding="utf-8") as f: + f.write(html) + with open(paths["wget-list"], "w") as f: + f.write("\n".join(wget) + "\n") + with open(paths["md5sums"], "w") as f: + f.write("\n".join(md5) + "\n") + except PermissionError: + raise SystemExit(f"! cannot write into {outdir} -- run as root or " + f"pass --outdir somewhere writable.") + for k, p in paths.items(): + os.chmod(p, 0o644) + print(f" {k:9s} -> {p}") + + if warnings: + print("\nwarnings:") + for w in warnings: + print(f" ! {w}") + print(f"\nverify it: lfs-sysvbook check --book {out_ver}" + + ("" if not args.outdir else f" --file {paths['book']}")) + if not args.outdir: + print(f"build it: lfs --book {out_ver} run") + return 0 + + +# --------------------------------------------------------------------------- # +# check +# --------------------------------------------------------------------------- # +def cmd_check(args): + if args.file: + path, label = args.file, os.path.basename(args.file) + wl = args.wget_list + else: + ver = args.book + if not ver: + raise SystemExit("! pass --book <version> or --file <html>") + path, label, wl = book_path(ver), ver, wget_list_path(ver) + if not os.path.isfile(path): + raise SystemExit(f"! no book at {path}") + soup = load_soup(path) + ok = True + + def result(good, msg): + nonlocal ok + print((" ok " if good else " FAIL ") + msg) + ok = ok and good + + print(f"checking {label}:") + + ids = {t.get("id") for t in soup.find_all(id=True)} + for aid in ("ch-system-udev", "ch-system-sysklogd", "ch-system-sysvinit", + "ch-config-bootscripts", "ch-config-usage"): + result(aid in ids, f"SysV section present: {aid}") + for aid in ("ch-system-dbus", "ch-config-systemd-custom"): + result(aid not in ids, f"systemd section absent: {aid}") + # ch-system-systemd must now be udev's heading, not a Systemd build + a = soup.find("a", id="ch-system-systemd") + if a is not None: + h = a.find_parent(re.compile(r"^h\d$")) + result(h is not None and "Udev" in h.get_text(), + "no Systemd package section") + + # no runnable command may use systemd machinery + bad = [] + for pre in soup.find_all("pre"): + classes = pre.get("class") or [] + if not any(c in classes for c in ("userinput", "root", "install")): + continue + if any(_is_admonition(p) for p in pre.parents): + continue + text = pre.get_text("") + for pat in (r"\bsystemctl\b", r"--with-systemd(?![a-z-])", + r"\bsystemd-analyze\b"): + if re.search(pat, text): + bad.append((pat, text.strip().splitlines()[0][:60])) + result(not bad, "no systemd commands in runnable blocks" + + ("" if not bad else " " + "; ".join(f"{t} in '{l}...'" + for t, l in bad[:3]))) + + # flag patches took + for aid, desc, _pat, _repl, marker in FLAG_PATCHES: + if not marker: + continue + a = soup.find("a", id=aid) + if a is None: + result(False, f"section {aid} missing") + continue + tp = next(p for p in a.parents + if p.name == "div" and "titlepage" in (p.get("class") or [])) + content = tp.find_next_sibling("div") + result(marker in "\n".join(runnable_commands(content)), desc) + + # 7.6 heredocs carry no systemd accounts + a = soup.find("a", id=CREATEFILES_ID) + if a is not None: + tp = next(p for p in a.parents + if p.name == "div" and "titlepage" in (p.get("class") or [])) + content = tp.find_next_sibling("div") + result(not SYSTEMD_ACCOUNT_LINE.search( + "\n".join(runnable_commands(content))), + "7.6: no systemd accounts in /etc/passwd and /etc/group") + + # every file a runnable command references is downloadable + if wl and os.path.isfile(wl): + with open(wl) as f: + have = {os.path.basename(u.strip()) for u in f if u.strip()} + need = set() + for pre in soup.find_all("pre"): + classes = pre.get("class") or [] + if not any(c in classes for c in ("userinput", "root", "install")): + continue + if any(_is_admonition(p) for p in pre.parents): + continue + for tok in FILE_TOKEN.findall(pre.get_text("")): + need.add(os.path.basename(tok)) + # names a build creates itself are not downloads + miss = {n for n in need if n not in have + and not n.startswith("lfs-temp-tools-") # 7.13 creates it + and not any(h.startswith(n.split(".tar")[0]) for h in have)} + result(not miss, "wget-list covers every file the commands reference" + + ("" if not miss else ": missing " + ", ".join(sorted(miss)))) + else: + print(f" (no wget-list at {wl} -- file coverage not checked)") + + print("PASS" if ok else "FAIL") + return 0 if ok else 1 + + +# --------------------------------------------------------------------------- # +def main(): + ap = argparse.ArgumentParser( + prog="lfs-sysvbook", + description="Graft the SysV flavour of an old LFS book onto a new " + "systemd-only release.") + ap.add_argument("--version", action="version", + version=f"lfs-sysvbook {VERSION} (build {_build_id()})") + sub = ap.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("make", help="produce the hybrid book + wget-list + " + "md5sums") + p.add_argument("--donor", default="12.4", + help="cached SysV book version (default: 12.4, the last " + "SysV release)") + p.add_argument("--target", default="13.0-systemd", + help="cached systemd book version (default: 13.0-systemd)") + p.add_argument("--donor-file", help="local donor html instead of the store") + p.add_argument("--target-file", help="local target html instead of the store") + p.add_argument("--out", help="version label for the result " + "(default: target with -systemd -> -sysv)") + p.add_argument("--outdir", help="write here instead of the lfs store") + p.set_defaults(func=cmd_make) + + p = sub.add_parser("check", help="verify a hybrid book is SysV-clean") + p.add_argument("--book", help="version label in the lfs store") + p.add_argument("--file", help="local html file instead") + p.add_argument("--wget-list", help="wget-list to check coverage against " + "(with --file)") + p.set_defaults(func=cmd_check) + + args = ap.parse_args() + sys.exit(args.func(args)) + + +if __name__ == "__main__": + main() diff --git a/packagemanager b/packagemanager old mode 100644 new mode 100755 index 1a0e769..2fc602f --- a/packagemanager +++ b/packagemanager @@ -33,20 +33,21 @@ Roadmap (built incrementally, one command at a time) check [TODO] report install state of a package search [TODO] find a package in the book -> blfs search dependencies [TODO] recursive dependency order -> blfs deps/order - install [TODO] resolve deps, build missing, run packagemanager_install + install [TODO] resolve deps, build missing, run lfs-helper pm-install remove [TODO] remove a package-user + its files (via pkg.lst) reload-pkg-list [TODO] regenerate /usr/src/<name>/pkg.lst list [TODO] list all package-users files_with_broken_id [TODO] find files whose uid/gid don't resolve Design decisions still open (we'll settle these as we implement the commands): - * install: auto-run packagemanager_install per package, or just print/queue? - * keep packagemanager_install (bash) as the privileged engine, or fold it in? + * install: auto-run lfs-helper pm-install per package, or just print/queue? + * packagemanager_install: retired in 1.12.0 -- folded into lfs-helper pm-install * fix `check`, implement `remove`; parameterize the hardcoded site values. """ import argparse import hashlib +import glob import json import os import re @@ -79,15 +80,33 @@ except ImportError: # non-unix: degrade gracefully # system built with a different layout keeps it; the environment still wins # over both, for tests and one-off runs. def _base_dir_default(): + """The BASE the account roots hang under -- /usr/src, not /usr/src/pkgusr. + + ONE KEY, TWO MEANINGS. `lfs` writes `pkgusr_home` as the FULL package-user + root (its LAYOUT_DEFAULTS say /usr/src/pkgusr); this read the same key as + the base and then appended PKGUSR_SUBDIR itself, so a tree configured by + lfs got both: + + logs: /usr/src/pkgusr/pkgusr/p_wget/log + + Taking the parent when the value already ends in the subdirectory makes the + two readings agree without either tool having to change what it writes -- + and a base that does NOT end in it is still taken literally, which is what + a hand-written config means. + """ # CONFIG_PATH is defined further down, so read the path the same way it # does rather than depending on definition order. path = os.environ.get("PKGUSR_CONFIG") or "/etc/pkgusr/packagemanager.conf" + sub = os.environ.get("PKGUSR_SUBDIR", "pkgusr") try: with open(path) as f: for line in f: k, _, v = line.partition("=") if k.strip() == "pkgusr_home" and v.strip(): - return v.strip() + v = v.strip().rstrip("/") + if sub and os.path.basename(v) == sub: + return os.path.dirname(v) or "/" + return v except OSError: pass return "/usr/src" @@ -109,7 +128,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.11.6" +TOOL_VERSION = "1.14.146" def _build_id(): @@ -139,6 +158,15 @@ def load_config(): if _config is not None: return _config _config = dict(_DEFAULT_CONFIG) + # WHICH KEYS CAME FROM THE FILE, as opposed to the defaults above. The + # conflict warning below compared the lfs config against whatever was in + # _config, and a default counted -- so a system with no packagemanager.conf + # at all was told its settings disagreed: + # note: collector_prefix is 'sysgroup' here but 'nimgnu' in the lfs + # config -- using 'nimgnu' + # Nothing said sysgroup. That is the built-in default being reported as an + # opinion someone held. + from_file = set() try: with open(CONFIG_PATH) as f: for line in f: @@ -146,6 +174,7 @@ def load_config(): if line and not line.startswith("#") and "=" in line: k, v = line.split("=", 1) _config[k.strip()] = v.strip() + from_file.add(k.strip()) except OSError: pass @@ -162,7 +191,7 @@ def load_config(): val = shared.get(key) if not val: continue - if _config.get(key) and _config[key] != val: + if key in from_file and _config.get(key) and _config[key] != val: printWarning( f"note: {key} is '{_config[key]}' here but '{val}' in the lfs " f"config -- using '{val}', which is what the system was built " @@ -172,7 +201,19 @@ def load_config(): # Keys owned by the lfs config, not by packagemanager's own file. -_SHARED_WITH_LFS = ("collector_prefix",) +# EVERY setting both tools have. `lfs build-system session` asks for these +# once, on the host, and the answers are what the system was actually built +# with -- so they are authoritative here, not a default to fall back to. +# +# This was `collector_prefix` alone. The other three were collected by the +# same interview, written into the same config.json, copied into the same +# chroot, and then asked for again by first_run_setup as though nothing knew +# them. +# The layout keys travel with the tree too, so a build system with a +# different directory layout keeps it without the environment having to be +# right on every invocation (1.13.69 did this for the prefixes). +_SHARED_WITH_LFS = ("collector_prefix", "pkgusr_prefix", "cfguser_prefix", + "main_user", "pkgusr_subdir", "cfguser_subdir") def _shared_lfs_config(): @@ -190,6 +231,38 @@ def _shared_lfs_config(): return {} +def _save_shared_key(key, value): + """Write a key where the READER will look for it. + + `config --pkgusr-prefix ""` wrote packagemanager.conf and reported + success -- while the value is read from /usr/share/lfs/config.json, which + is authoritative for the shared keys. `--show` then still said "p_, the + lfs config", and nothing had changed. + + A setting that is saved somewhere the tool does not read is not saved. + """ + for cand in (os.environ.get("LFS_STORE"), "/usr/share/lfs", + os.path.expanduser("~/.local/share/lfs")): + if not cand: + continue + path = os.path.join(cand, "config.json") + if not os.path.isfile(path): + continue + try: + with open(path) as f: + data = json.load(f) + data[key] = value + with open(path, "w") as f: + json.dump(data, f, indent=2, sort_keys=True) + f.write("\n") + printInfo(" wrote %s to %s (it wins over packagemanager.conf)" + % (key, path)) + return True + except (OSError, ValueError) as e: + printWarning(" could not write %s: %s" % (path, e)) + return False + + def save_config(cfg): try: os.makedirs(os.path.dirname(CONFIG_PATH) or ".", exist_ok=True) @@ -283,6 +356,11 @@ def pkgusr_name(name): """ if not name: return name + # Lower-case FIRST, exactly like lfs-helper's pkg_owner_name: the book's + # anchors are not consistent (DocBook, Wget, Which), and p_DocBook next + # to the helper's p_docbook is two accounts for one package -- a failure + # box this session printed both names in one breath. + name = name.lower() kind = pkgusr_kind(name) # an application user or a collector group is not a package and is never # renamed here @@ -313,8 +391,51 @@ def unprefix_pkgusr(name): # /usr/src/pkgusr/p_gcc a package # /usr/src/cfg/cfg_bootscripts a config step that installs files # /usr/src/u_firefox an application user -PKGUSR_SUBDIR = os.environ.get("PKGUSR_SUBDIR", "pkgusr") -CFGUSR_SUBDIR = os.environ.get("CFGUSR_SUBDIR", "cfg") +def _early_conf(key, default): + """A value from packagemanager.conf, read before load_config() exists. + + THE DIRECTORIES HAD NO CONFIG KEYS. BASE_DIR has read `pkgusr_home` from + this file for a long time; the two subdirectories were ENVIRONMENT ONLY, + so a tree with a different layout depended on the environment being right + every time -- and could not be made tree-specific the way the prefixes + now are (1.13.69). + + Precedence, as everywhere else: environment, then the file, then the + built-in default. + """ + env = {"pkgusr_subdir": "PKGUSR_SUBDIR", + "cfguser_subdir": "CFGUSR_SUBDIR"}.get(key) + if env and os.environ.get(env) is not None: + return os.environ[env] + # The LFS config is authoritative for the layout as it is for the + # prefixes -- that is what makes it travel with the tree rather than + # depending on the environment of whoever runs the command. + for cand in (os.environ.get("LFS_STORE"), "/usr/share/lfs", + os.path.expanduser("~/.local/share/lfs")): + if not cand: + continue + try: + import json as _json + with open(os.path.join(cand, "config.json")) as f: + v = _json.load(f).get(key) + if v is not None: + return str(v) + except (OSError, ValueError): + continue + path = os.environ.get("PKGUSR_CONFIG") or "/etc/pkgusr/packagemanager.conf" + try: + with open(path) as f: + for line in f: + k, _, v = line.partition("=") + if k.strip() == key: + return v.strip() # an empty value is an answer + except OSError: + pass + return default + + +PKGUSR_SUBDIR = _early_conf("pkgusr_subdir", "pkgusr") +CFGUSR_SUBDIR = _early_conf("cfguser_subdir", "cfg") def pkgusr_root(name): @@ -392,7 +513,11 @@ def printLine(): def printSuccess(msg): print(f"{colors.green}{msg}{colors.normal}") def printInfo(msg): print(f"{colors.blue}{msg}{colors.normal}") -def printWarning(msg): print(f"{colors.yellow}{msg}{colors.normal}") +# Warnings go to STDERR, like lfs-helper's warn(). A note printed to stdout +# lands inside whatever a caller is capturing -- the config-conflict note was +# read back as part of `print(pm.BASE_DIR)` by anything scripting this module, +# and the value it polluted was correct. +def printWarning(msg): print(f"{colors.yellow}{msg}{colors.normal}", file=sys.stderr) def printError(msg): print(f"{colors.red}{msg}{colors.normal}") @@ -410,6 +535,26 @@ def ask_yes_no(question): # shared helpers # --------------------------------------------------------------------------- # +# BASE_DIR is /usr/src, which holds more than package users -- lfs-pkgusr, +# a stray root/, whatever else lives there. Listing every directory reported +# them as packages ("root broken has a home dir but no user"). An account +# either exists in the user database or carries the skeleton every package +# user is created with. +_ACCOUNT_MARKERS = ("build.conf", ".project", "install_last", "pkg.lst", + ".bash_profile") + + +def _looks_like_an_account(path, name): + # the prefix IS the evidence: p_gcc is a package user by construction + for pfx in (pkgusr_prefix(), user_prefix(), cfguser_prefix()): + if pfx and name.startswith(pfx): + return True + # NOT "a user of this name exists" -- /usr/src/root passed that test + # because root exists, and came out of verify as a broken package. A + # system user is not evidence that a DIRECTORY is a package home. + return any(os.path.lexists(os.path.join(path, m)) for m in _ACCOUNT_MARKERS) + + def list_all_users(): """Every package-user, i.e. every directory directly under BASE_DIR.""" # Directory names ARE user names, so they carry the prefix. Hand back @@ -427,8 +572,37 @@ def list_all_users(): except OSError: continue found = True - out += [unprefix_pkgusr(d) for d in names - if os.path.isdir(os.path.join(root, d))] + # A COLLECTOR GROUP IS NOT A PACKAGE. + # + # `nimgnu_perl-modules`, `nimgnu_python-modules` and friends are + # GROUPS that own shared install directories. They were listed as + # package-users and reported "broken -- has a user but no home dir", + # so a verify of a healthy system ended with four scary lines about + # things that are working exactly as designed. A false alarm in a + # health check is worse than no check: it is what teaches people to + # skip the output. + # BOTH SPELLINGS. lfs-helper fell back to "nimgnu" while this tool + # and `lfs` fell back to "sysgroup" (fixed in 1.14.35), so a system + # whose config never set the key has groups under the OTHER name. + # Recognise them however they were created; the config still decides + # what NEW ones are called. + _cps = {collector_prefix(), "sysgroup_", "nimgnu_"} + # ONE UNDERSCORE. collector_prefix() already ends in "_", so this + # tested startswith("sysgroup__") and never matched anything -- the + # filter above has never worked, and every collector group was still + # listed as a "broken" package. + # ...and check the name BOTH ways. A directory may be `p_nimgnu_x` + # (the package prefix stacked on a group name) or plain `nimgnu_x`; + # the raw name misses the first, the unprefixed one misses nothing. + for d in names: + if not os.path.isdir(os.path.join(root, d)): + continue + bare = unprefix_pkgusr(d) + if any(c and (d.startswith(c) or bare.startswith(c)) for c in _cps): + continue + if not _looks_like_an_account(os.path.join(root, d), d): + continue + out.append(bare) if not found: printError(f"{BASE_DIR} does not exist (looked in " f"{', '.join(pkgusr_roots())}).") @@ -481,6 +655,10 @@ _BLFS_ACTIVITY = { } +# distinct blfs notes already shown this invocation (see run_blfs) +_FORWARDED_NOTES = set() + + def run_blfs(args): """Run blfs; return (returncode, stdout, stderr). rc 127 => blfs missing. @@ -498,18 +676,78 @@ def run_blfs(args): pass try: r = subprocess.run(cmd, capture_output=True, text=True) + # blfs says important things on stderr -- "note: no book matched + # '13.0-systemd'; using the cached ..." is a decision being + # overridden, and capturing it silently is how a wrong book got used + # without a word. Forward each DISTINCT note once: a plan touching + # ninety packages makes ninety blfs calls, and the same note ninety + # times is noise wearing a warning's colour. + _notes = [ln for ln in (r.stderr or "").splitlines() + if ln.startswith("note:") and ln not in _FORWARDED_NOTES] + if _notes: + try: + clear_status() + except Exception: + pass + for ln in _notes: + _FORWARDED_NOTES.add(ln) + printWarning(ln) return r.returncode, r.stdout, r.stderr except FileNotFoundError: return 127, "", "blfs executable not found" + finally: + # the status line this call set is ITS to clear: left standing, the + # caller's next print lands on the same line + # ("reading versions from the bookChecking 97 package(s)...") + if verb in _BLFS_ACTIVITY: + try: + clear_status() + except Exception: + pass + + +_VERSION_MAP = None # anchor -> name_version, the whole book, once def book_version(anchor): - """Current name-version of a package in the book, or None if unavailable.""" + """Current name-version of a package in the book, or None if unavailable. + + ONE PROCESS FOR THE WHOLE BOOK, not one per question. This ran `blfs + debug <anchor>` every time it was called -- a subprocess that opens and + reads the book cache to answer one lookup. A stack preview asks it once + per entry (56 for sway) and the planner asks it for every package it + considers, so a command whose real work is a dictionary lookup spent + minutes launching processes: + + looking up pipewire in the book <- one process + looking up ... in the book <- another + + `book_versions_all` was already here, used only by `update`. The map is + loaded on first use and answers everything after that; a name the map does + not have (an alias, a URL, a fragment) still falls back to the old call. + """ + global _VERSION_MAP + if _VERSION_MAP is None: + try: + _VERSION_MAP = book_versions_all() or {} + except Exception: + _VERSION_MAP = {} + hit = _VERSION_MAP.get(anchor) + if hit: + return hit + if _VERSION_MAP: + low = anchor.lower() + for a, nv in _VERSION_MAP.items(): + if a.lower() == low: + return nv rc, out, _ = run_blfs(["debug", anchor]) if rc != 0: return None m = re.search(r"^title\s*:\s*(.+)$", out, re.M) - return m.group(1).strip() if m else None + nv = m.group(1).strip() if m else None + if nv and _VERSION_MAP is not None: + _VERSION_MAP[anchor] = nv # ask once, even for an alias + return nv def book_rdeps(target, no_recommended=False, optional=False): @@ -560,16 +798,19 @@ def cmd_blfs(args): def find_pm_install(): - """The privileged install engine (bash). Overridable with PM_INSTALL_BIN - so tests can substitute a stub.""" + """The install front door. packagemanager_install is retired (item 6, + step 7); the front-matter and the loop both live in lfs-helper now, behind + `lfs-helper pm-install <account> <script>`. Overridable with + PM_INSTALL_BIN so tests can substitute a stub.""" override = os.environ.get("PM_INSTALL_BIN") if override: return override.split() here = os.path.dirname(os.path.abspath(__file__)) - local = os.path.join(here, "packagemanager_install") + local = os.path.join(here, "lfs-helper") if os.path.isfile(local): - return [local] if os.access(local, os.X_OK) else ["bash", local] - return ["packagemanager_install"] + base = [local] if os.access(local, os.X_OK) else ["bash", local] + return base + ["pm-install"] + return ["lfs-helper", "pm-install"] # =========================================================================== # @@ -619,35 +860,96 @@ def _parse_bash_vars(text, scalars=(), arrays=()): """Pull simple key="value" and single-line key=(a b c) assignments out of a generated install script. Good enough for our own install_last files (all header vars are single lines at the top).""" + # GREEDY, AND THE LINE HAS A COMMENT ON IT. + # + # The stack's generated scripts document their own metadata: + # installed_programs=() # e.g. ('gst-launch-1.0' 'gst-inspect-1.0') + # installed_program="sway" # e.g. "ffmpeg" (checked on PATH) + # `\((.*)\)` ran from the first "(" to the LAST ")" -- into the comment -- + # and `"(.*)"` likewise from the first quote to the last. Every git and + # tar package in `verify` therefore reported + # missing: program ), program #, program e.g., program ('gst-launch-1.0 + # for twenty packages, in every run, all session. Non-greedy, and a + # trailing comment is allowed and ignored. out = {} for key in scalars: - m = re.search(rf'^{key}="(.*)"[ \t]*$', text, re.M) + m = re.search(rf'^{key}="(.*?)"[ \t]*(#.*)?$', text, re.M) if m: out[key] = m.group(1) for key in arrays: - m = re.search(rf'^{key}=\((.*)\)[ \t]*$', text, re.M) + m = re.search(rf'^{key}=\((.*?)\)[ \t]*(#.*)?$', text, re.M) if m: items = re.findall(r"'[^']*'|\"[^\"]*\"|\S+", m.group(1)) out[key] = [it.strip("'\"") for it in items] return out +def _same_version(a, b): + """Do these two name_version strings describe the same release? + + The books capitalize differently -- BLFS anchors are lowercase + ('which-2.23'), the LFS book titles its packages ('Which-2.23') -- and + the VERSION file records the LFS spelling. Comparing exactly would + invent an update for every package both books carry, which is precisely + the risk of reading VERSION at all.""" + if not a or not b: + return False + return a.strip().lower() == b.strip().lower() + + def read_install_last(home): - """Parsed vars from /usr/src/<name>/install_last (or install_<name>).""" + """Parsed vars from /usr/src/<name>/install_last (or install_<name>). + + Falls back to the VERSION file when there is no install script record. + Both build paths write down what they installed -- packagemanager into + install_last, lfs-helper into VERSION -- and reading only the first meant + every package the LFS build produced came back with no version at all: + "97 checked, 0 judged" on a system where `lfs update` compares all 97 + happily. Same fact, two files; this reads whichever exists. + + (The real fix is one build loop -- open item 6 -- but a tool must not be + blind to the other loop's records in the meantime.)""" if not home: return {} - for candidate in ("install_last", - "install_" + os.path.basename(home)): - path = os.path.join(home, candidate) - if os.path.isfile(path): - try: - text = open(path, encoding="utf-8", errors="replace").read() - except OSError: - return {} - return _parse_bash_vars(text, _LAST_SCALARS, _LAST_ARRAYS) + # install_last first: it is the record of the script AS IT LAST RAN, the + # only file that states what is actually installed. + path = os.path.join(home, "install_last") + if os.path.isfile(path): + try: + text = open(path, encoding="utf-8", errors="replace").read() + except OSError: + return {} + return _parse_bash_vars(text, _LAST_SCALARS, _LAST_ARRAYS) + # Then lfs-helper's record. It must come BEFORE the install script: that + # script is regenerated whenever the book moves on, so its name_version is + # what the BOOK has, and reading it would compare the book with itself and + # call everything current forever. (`lfs update` avoids the same trap.) + vpath = os.path.join(home, "VERSION") + if os.path.isfile(vpath): + try: + for line in open(vpath, encoding="utf-8", errors="replace"): + line = line.strip() + if line and not line.startswith("#"): + return {"name_version": line, "_source": "VERSION"} + except OSError: + pass + # Last resort, and only for a home with no record of its own. + path = os.path.join(home, "install_" + os.path.basename(home)) + if os.path.isfile(path): + try: + text = open(path, encoding="utf-8", errors="replace").read() + except OSError: + return {} + return _parse_bash_vars(text, _LAST_SCALARS, _LAST_ARRAYS) return {} +def _is_account_furniture(path): + """Paths that exist because the ACCOUNT exists, not because anything was + installed: useradd's mail spool, and the home directory itself.""" + return path.startswith("/var/mail/") or path.startswith("/var/spool/mail/") + + class User: __slots__ = ("search", "name", "uid", "gid", "home", "pkg_list", "meta") @@ -673,23 +975,91 @@ class User: if not self.home and not self.has_user: return "not installed" if self.home and self.has_user: - if self.meta.get("name_version") or self._manifest_has_real_paths(): + # install_last says a version was installed -- but if the manifest + # beside it is mostly gone, the files are not there and the claim + # is stale. The manifest wins when there is one. + if self.pkg_list: + if self._manifest_has_real_paths(): + return "installed" + return "not installed" + if self._declares_content_that_is_missing(): + # the script says where it installs, and it is not there + return "not installed" + if self.meta.get("name_version"): return "installed" return "not installed" return "broken" - def _manifest_has_real_paths(self): + def _declares_content_that_is_missing(self): + """The book says what this package installs -- is any of it there? + + A SECOND OPINION, for when there is no manifest to consult. Every + generated script carries the page's own Installed-programs / + -libraries / -directories, e.g. for Pathspec + + installed_directories='/usr/lib/python3.14/site-packages/pathspec ...' + + so "installed" can be checked against the system instead of inferred + from the existence of an install_last record. Only decides NOT + installed when the script names paths and none of them exist; a + package whose page names nothing is left to the older rule. + """ + paths = [] + for key in ("installed_directories", "installed_directory", + "installed_libraries", "installed_library"): + for p in (self.meta.get(key) or "").split(): + if p.startswith("/"): + paths.append(p) + if not paths: + return False + return not any(os.path.exists(p) for p in paths) + + def _manifest_has_real_paths(self): # noqa: D401 (see _is_account_furniture) + """Does this manifest still DESCRIBE the system? + + One surviving path used to be enough, and that is how a record kept + lying for three sessions: binutils claimed 330 paths with 190 gone, + 140 survivors said "installed", the planner skipped it, and packages + that needed it failed somewhere else entirely (xmllint, glib, + -lfreetype). `lfs-helper verify` had the right rule all along -- a + record with most of its files missing is not describing an installed + package -- but only the manual sweep applied it. The planner uses the + same rule now, so an ordinary `packagemanager install` / `stack --run` + rebuilds such a package instead of skipping it. + + Counted over the WHOLE manifest (capped at 5000 paths, which no + package reaches by much). Sampling the first N is wrong: pkg.lst is + path-sorted, so a package whose /usr/bin survived and whose + /usr/share died would look healthy from the top.""" if not self.pkg_list: return False + seen = alive = 0 try: with open(self.pkg_list, encoding="utf-8", errors="replace") as f: for line in f: p = line.strip() - if p.startswith("/") and os.path.lexists(p): - return True + if not p.startswith("/"): + continue + # NOT AN INSTALLED FILE, WHOEVER WROTE THE RECORD. + # useradd's mail spool is owned by the package user, so + # older manifests contain it -- and for a package that + # installed nothing it is the ONLY line, so the record + # counted 1 of 1 alive and said "installed". Excluding it + # from the scan (1.14.16) only fixed records written after; + # every pkg.lst already on disk still claimed it, which is + # why pathspec was skipped again on the next run. + if _is_account_furniture(p): + continue + seen += 1 + if os.path.lexists(p): + alive += 1 + if seen >= 5000: + break except OSError: - pass - return False + return False + if not seen: + return False + return alive * 2 > seen # more than half still there @property def broken(self): @@ -706,11 +1076,19 @@ class User: def gather_user(name): """Populate a User from the system user db, the on-disk package dir, and - its install_last script.""" + its install_last script. + + The lookup goes through pkgusr_name(): callers speak in PACKAGE names + ('glibc') and list_all_users hands them back unprefixed, while the account + is 'p_glibc'. Looking the raw name up got a KeyError, so has_user stayed + false, so state came out 'broken' -- and every package the LFS build made + landed in the "cannot judge" bucket of `packagemanager update` with its + home, VERSION and manifest sitting right there. pkgusr_name is + idempotent, so this is safe for a name that already carries the prefix.""" u = User(name) if pwd is not None: try: - pw = pwd.getpwnam(name) + pw = pwd.getpwnam(pkgusr_name(name)) u.name, u.uid, u.gid = pw.pw_name, str(pw.pw_uid), str(pw.pw_gid) except KeyError: pass @@ -823,7 +1201,12 @@ def _script_name_version(path): try: with open(path) as f: for line in f: - m = re.match(r'\s*name_version="([^"]+)"', line) + # EITHER QUOTE. Since 1.12.18 the generator emits book strings + # SINGLE-quoted (a dollar sign in prose is not a variable), and this + # matched only double quotes -- so _script_name_version returned None, + # the local script never matched, and every edit a user made in + # install_<name>-<version> was quietly ignored for the generated one. + m = re.match(r"""\s*name_version=["']([^"']+)["']""", line) if m: return m.group(1) except OSError: @@ -831,6 +1214,35 @@ def _script_name_version(path): return None +def _pending_edit_conflict(home, iv, bv): + """Would updating iv -> bv silently orphan local edits? + + `install_last` is the read-only record of the script AS IT LAST RAN; + the canonical `install_<iv>` beside it is where edits live. If they + differ, the user has changes that never ran -- and the update will + generate a fresh script for `bv` that carries none of them. The make-ca + systemctl edit was exactly this, and --regenerate discarding it is what + the handoff warned about. + + Self-clearing: once a local `install_<bv>` exists in the home (the user + merged their edits into the new version's script -- resolve_script will + prefer it), the conflict is resolved and this answers False. Missing + files answer False too: with no record there is nothing to judge.""" + if not iv or not bv or iv == bv or iv == "(reinstall)": + return False + if os.path.isfile(os.path.join(home, f"install_{bv}")): + return False + canon = os.path.join(home, f"install_{iv}") + last = os.path.join(home, "install_last") + if not (os.path.isfile(canon) and os.path.isfile(last)): + return False + try: + return open(canon, errors="replace").read() \ + != open(last, errors="replace").read() + except OSError: + return False + + def _warn_if_wrong_file_edited(user, chosen): """Point out an edit made to the wrong file. @@ -857,6 +1269,239 @@ def _warn_if_wrong_file_edited(user, chosen): printWarning(f" Edit this instead: {chosen}") + +_GENVER_RE = re.compile(r"^### generated by blfs (\S+)", re.M) + +def _blfs_gen_version(): + """The generator's own version string, asked once.""" + if not hasattr(_blfs_gen_version, "_v"): + rc, out, _ = run_blfs(["--version"]) + m = re.search(r"blfs (\S+)", out or "") + _blfs_gen_version._v = m.group(1) if (rc == 0 and m) else None + return _blfs_gen_version._v + + +def _script_is_user_edited(path): + """Did somebody change this script after it was generated? + + The generator stamps every script and never writes it again, so an mtime + newer than the stamp line's own file creation is the only signal there + is. We use a marker instead: a script the user has edited gets a + ".edited" sibling the first time it is regenerated, and we treat the + presence of a `# EDITED` line, or of that sibling, as "hands off". + """ + if os.path.exists(path + ".edited"): + return True + try: + with open(path) as f: + head = f.read(4000) + except OSError: + return False + return "# EDITED" in head or "# do not regenerate" in head.lower() + + +def _script_gen_version(path): + try: + head = open(path).read(400) + except OSError: + return None + m = _GENVER_RE.search(head) + return m.group(1) if m else None + + +def _prewarm_scripts(plan, script_dir, force_regen=False): + """Generate every script the plan needs, in ONE blfs call. + + resolve_script generates per package, and a plan of 151 packages therefore + ran 151 subprocesses that each loaded and indexed the whole book. With a + warm cache that is a fifth of a second each; on a fresh system, where the + first call INDEXES the book, it is the difference between a wait and a + hang -- and it looked like a hang, because the only output was one + "-- regenerating" line per package with minutes of silence between them. + + Nothing else changes: this writes the same files into the same directory, + and resolve_script then finds them cached. A local script in a package + user's home still wins; those are not touched here. + """ + wanted = [] + for anchor, nv, _reason in plan: + local = os.path.join(pkgusr_home(pkgusr_name(anchor)), f"install_{nv}") + if os.path.isfile(local) and not force_regen: + continue # the user's own copy wins + cached = os.path.join(script_dir, f"install_{nv}") + if not force_regen and os.path.isfile(cached) \ + and _script_name_version(cached) == nv: + _cv, _gv = _script_gen_version(cached), _blfs_gen_version() + if not (_cv and _gv and _cv != _gv): + continue # current, keep it + wanted.append(anchor) + if len(wanted) < 2: + return # nothing to gain + printInfo(" generating %d install script(s) from the book " + "(one pass) ..." % len(wanted)) + os.makedirs(script_dir, exist_ok=True) + rc, out, err = run_blfs(["script"] + wanted + ["-o", script_dir]) + made = len(re.findall(r"^Created ", out, re.M)) + if rc != 0 and made == 0: + # not fatal: resolve_script will generate them one at a time, as before + printWarning(" could not generate them in one pass (%s); " + "falling back to one at a time" + % (err or out).strip().splitlines()[-1:] or "no output") + return + printInfo(" %d written" % made) + + +def _stop_if_prereqs_missing(args): + """A plan that cannot finish should not start. + + Warning and building anyway is how a missing cmake got found twenty-two + packages deep, after the warning had already scrolled off. The check is + exact -- these are the scripts' own guards and their own command words -- + so stopping is safe, and --ignore-prereqs is there for the case where it + is wrong. + """ + if not (_MISSING_TOOLS or _MISSING_ENV): + return + if not getattr(args, "run", False): + return # a dry run is allowed to show the plan + if getattr(args, "ignore_prereqs", False): + printWarning(" --ignore-prereqs: starting anyway; the packages above" + " will fail where they need what is missing.") + return + printError("Not starting: this plan is missing what %d of its packages" + " need." % len({p for v in list(_MISSING_TOOLS.values()) + + list(_MISSING_ENV.values()) for p in v})) + # SAY WHEN THE PACKAGE IS THERE AND ONLY THE TOOL IS NOT FOUND. + for _t in sorted(_MISSING_TOOLS): + _prov = _provider_of(_t) + if _prov and gather_user(_prov).state == "installed": + printInfo(" NOTE: %s is installed, so `%s` exists but is not on" + " PATH here." % (_prov, _t)) + printInfo(" BLFS puts it outside /usr (e.g. /opt/%s/bin)" + " and adds it in" % _prov) + printInfo(" /etc/profile.d -- which a login shell reads and" + " this one did not.") + printInfo(" Check: bash -lc 'command -v %s'" % _t) + printInfo(" Fix the lines above, then run the same command again --" + " finished packages are skipped.") + printInfo(" To start regardless: --ignore-prereqs") + sys.exit(1) + + +def _report_ignored_required(steps): + """A package in the plan whose REQUIRED dependency is being ignored. + + webkitgtk requires Ruby; the stack's avoid list contains ruby; so the plan + left out a hard requirement of a package the user asked for, and cmake + said "Ruby 2.5 or higher is required" after an hour of building. Skipping + an avoided branch is right (1.14.24) -- doing it without a word, when the + thing skipped is REQUIRED by something being built, is not. + """ + if not _IGNORED_REQUIRED: + return + _in = {a.lower() for a, _n, _r, _p in steps} + pairs = [(w, d) for w, d in _IGNORED_REQUIRED if w.lower() in _in] + if not pairs: + return + print() + printWarning("%d package(s) in this plan REQUIRE something the ignore list" + " excludes:" % len(pairs)) + for w, d in pairs: + printWarning(" %-20s requires %s, which is ignored" % (w, d)) + printWarning(" They will fail where they need it. Either drop it from the") + printWarning(" ignore list (in the stack file, or --ignore), or install it:") + printWarning(" packagemanager install %s --run --recursive" + % " ".join(sorted({d for _w, d in pairs}))) + print() + + +def _report_cycles(steps): + """Name the packages a dependency cycle forces out of order. + + vala recommends graphviz; graphviz -> gegl -> babl -> librsvg -> vala + closes the loop, so one recommended edge has to give and vala is built + first -- without graphviz, and (for vala) with configure stopping on + "Package 'libgvc' not found". There is no order that satisfies a cycle; + what there is, is a rebuild afterwards, and the person can only ask for it + if they are told. + """ + if not _CYCLE_PAIRS: + return + # Only the direction that ACTUALLY happened. The breaker drops both + # edges of a two-package cycle, so freetype2/harfbuzz was reported in both + # directions and one of the two lines was always false. The final order + # decides which one is true. + _pos = {a.lower(): i for i, (a, _n, _r, _p) in enumerate(steps)} + _pairs = [(a, d) for a, d in _CYCLE_PAIRS + if a.lower() in _pos and d.lower() in _pos + and _pos[a.lower()] < _pos[d.lower()]] + if not _pairs: + return + print() + printWarning("%d package(s) are built before something they recommend" + " (a dependency cycle):" % len(_pairs)) + for a, d in _pairs: + printWarning(" %-16s is built before %s" % (a, d)) + printWarning(" That is unavoidable -- the recommendation runs both ways." + " They build") + printWarning(" without the optional feature; rebuild afterwards if you" + " want it:") + printWarning(" packagemanager install %s --run --reinstall" + % " ".join(sorted({a for a, _d in _pairs}))) + print() + + +def _report_missing_env(): + """One block naming every variable the plan's scripts will refuse without. + + Reported per package this was one to three lines each: in a 151-package + plan the same XORG_PREFIX notice appeared dozens of times, scrolled past, + and the run stopped on it seventeen packages in. + """ + if not _MISSING_ENV: + return + print() + printWarning("This plan cannot finish: %d build variable(s) are not set." + % len(_MISSING_ENV)) + for name in sorted(_MISSING_ENV): + users = _MISSING_ENV[name] + printWarning(" %-14s needed by %d package(s), first: %s" + % (name, len(users), users[0])) + printWarning(" Set them once -- they apply to every build:") + for name in sorted(_MISSING_ENV): + printWarning(" packagemanager env --set %s=%s" + % (name, _ENV_SUGGESTIONS.get(name, "..."))) + printWarning(" (exporting them in your shell does not reach a package" + " user; %s does)" % BUILD_ENV_FILE) + print() + + +def _report_missing_tools(): + """One block, under the plan, naming every build-system driver the plan's + own commands call and this system does not have.""" + _report_missing_env() + if not _MISSING_TOOLS: + return + print() + printWarning("This plan cannot finish: %d build tool(s) are missing." + % len(_MISSING_TOOLS)) + for tool in sorted(_MISSING_TOOLS): + users = _MISSING_TOOLS[tool] + printWarning(" %-8s needed by %d package(s), first: %s" + % (tool, len(users), users[0])) + _provs = sorted({p for p in (_provider_of(t) for t in _MISSING_TOOLS) if p}) + if _provs: + printWarning(" Install them first, with their own dependencies:") + printWarning(" packagemanager install %s --run --recursive" + % " ".join(_provs)) + _noprov = sorted(t for t in _MISSING_TOOLS if not _provider_of(t)) + if _noprov: + printWarning(" No BLFS page provides %s -- these come from LFS;" + % ", ".join(_noprov)) + printWarning(" check they were built and are on PATH.") + print() + + def resolve_script(user, name_version, script_dir, assume_yes=False, force_regen=False): """Path to the install script to use for `user`. @@ -867,7 +1512,23 @@ def resolve_script(user, name_version, script_dir, assume_yes=False, engine's runtime copy) are never treated as sources. --regenerate forces a fresh build from the book.""" local = os.path.join(pkgusr_home(user), f"install_{name_version}") - if (not force_regen and os.path.isfile(local) + # A STALE SCRIPT IS REFRESHED, NOT JUST REPORTED. + # + # Telling the user to pass --regenerate has cost several rounds each on + # brotli, rust, xcb-utilities and ffmpeg: the fix existed, the warning + # printed, and the build still ran the old script -- because spotting one + # warning in a 200-package plan and acting on it per package is not a + # reasonable thing to ask. A script older than the generator and NOT + # edited by the user is simply regenerated below. + _stale_unedited = False + if os.path.isfile(local): + _lgv, _cgv = _script_gen_version(local), _blfs_gen_version() + _stale_unedited = bool(_lgv and _cgv and _lgv != _cgv + and not _script_is_user_edited(local)) + if _stale_unedited: + printWarning(f" {os.path.basename(local)}: generated by blfs " + f"{_lgv}, this is {_cgv} -- regenerating") + if (not force_regen and not _stale_unedited and os.path.isfile(local) and _script_name_version(local) == name_version): printInfo(f" using LOCAL script (your edits, not the book): {local}") _warn_if_wrong_file_edited(user, local) @@ -884,6 +1545,48 @@ def resolve_script(user, name_version, script_dir, assume_yes=False, f"book with --regenerate.") else: printInfo(f" -- pass --regenerate to rebuild {name_version} from the book") + # A SCRIPT FROM AN OLDER GENERATOR CARRIES THE OLD BUGS. + # + # brotli failed twice on the same `pip3 install --find-links dist` + # after the generator had been fixed: the local script was written by + # the previous blfs, resolve_script prefers it, and the only notice + # was a one-line hint that is easy to miss in a 200-package run. + # Say it loudly, and offer the command that fixes it. + # A STALE SCRIPT IS REFRESHED, NOT JUST REPORTED. + # + # Telling the user to pass --regenerate has now cost several rounds + # each on brotli, rust, xcb-utilities and ffmpeg: the fix existed, + # the message was printed, and the build still ran the old script -- + # because noticing one warning in a 200-package plan and acting on it + # per package is not a reasonable thing to ask. + # + # A script older than the generator is regenerated automatically. + # Nothing is lost: if the file differs from what the generator + # produces, the old one is kept as .edited (1.13.4) and named here. + _lg, _cg = _script_gen_version(local), _blfs_gen_version() + if _lg and _cg and _lg != _cg: + printWarning(f" ^ generated by blfs {_lg}; the generator is now " + f"{_cg}, and this file has YOUR EDITS -- keeping it") + printWarning(f" to take the newer generator instead (your copy " + f"is saved as install_{name_version}.edited):") + printWarning(f" packagemanager install {user} --run --regenerate") + # THE OPTIONS APPLY TO A LOCAL SCRIPT TOO. + # + # They were merged only when a script was GENERATED, so a package + # whose script already sat in its home -- every package that has + # failed once -- was reused untouched, and libassuan kept building + # its TeX manual with the fix sitting in package-fixes.conf. The + # merge is idempotent, so applying it on every resolve is safe. + apply_machine_opts(user, local) + # THE CHECKS BELONG TO EVERY PATH OUT OF HERE. + # + # They were on the generate path (1.14.9 added the cache path), and a + # script already sitting in the package user's home returned before + # either -- which is EVERY package that has been attempted once. So a + # plan whose libaom needed a missing cmake said nothing, built + # twenty-two other packages, and then stopped exactly where the check + # would have pointed. + _check_script_prereqs(user, local) return local # reuse a script a PRIOR DRY-RUN already generated into script_dir, so a @@ -891,14 +1594,565 @@ def resolve_script(user, name_version, script_dir, assume_yes=False, cached = os.path.join(script_dir, f"install_{name_version}") if (not force_regen and os.path.isfile(cached) and _script_name_version(cached) == name_version): - return cached + # "version-keyed = safe" was only half true: it caught a moved BOOK, + # not a moved GENERATOR. A pre-fix cached docbook-xsl script ran the + # optional doc block a newer blfs comments out. A cache from an + # older generator is regenerated. + _cv, _gv = _script_gen_version(cached), _blfs_gen_version() + if _cv and _gv and _cv != _gv: + printInfo(f" cached script is from blfs {_cv}, this is {_gv} " + f"-- regenerating") + else: + _check_script_prereqs(user, cached) + return cached rc, out, err = run_blfs(["script", user, "-o", script_dir]) m = re.search(r"^Created (.+)$", out, re.M) if rc != 0 or not m: printError(f"{name_version}: blfs script failed -- {(err or out).strip()}") return None - return m.group(1).strip() + path = m.group(1).strip() + # the machine's options, merged the moment the script is born + apply_machine_opts(user, path) + _check_script_prereqs(user, path) + return path + + + +# --------------------------------------------------------------------------- # +# machine option overrides +# --------------------------------------------------------------------------- # +# The BOOK decides how a package is built; the MACHINE decides its options. +# Mesa's gallium drivers belong to the GPU in the case, not to the stack file +# or the book -- so /etc/pkgusr/stacks/machine.conf carries them: +# +# [mesa] +# gallium-drivers = radeonsi +# vulkan-drivers = amd +# +# Every key of an [anchor] section is merged into the generated script's +# `meson setup` line as -Dkey=value -- an existing -Dkey is replaced, a new +# one is appended right after `meson setup`. Only meson for now (cmake +# pending); only freshly generated scripts are touched, and the change makes +# the script a LOCAL edit, which resolve_script then deliberately keeps. + +MACHINE_CONF = "/etc/pkgusr/stacks/machine.conf" + +# Two files, and the difference matters. +# +# package-fixes.conf -- workarounds the TOOLS ship (cairo's ctime_r, +# libassuan's TeX manual). Refreshed with the other +# stack files, so a fix reaches an existing system. +# machine.conf -- the MACHINE'S own choices (mesa's drivers). Never +# overwritten, and it wins where both mention a key. +# +# They were one file, which meant a workaround shipped in machine.conf could +# never arrive anywhere: the sync refuses to overwrite it, correctly, because +# it is the user's. libassuan kept failing on a fix that was in the release +# and not on the system. +PACKAGE_FIXES = "/etc/pkgusr/stacks/package-fixes.conf" + + +def _read_conf_section(path, anchor): + if not os.path.isfile(path): + return {} + import configparser + # strict=False: A REPEATED SECTION IS NOT A BROKEN FILE. + # + # Appending the same block twice -- exactly what `cat >> machine.conf` + # does when a recipe is followed twice -- made configparser raise, and + # this discarded the ENTIRE file. So [webrtc-audio-processing] did + # nothing, and so did [mesa]: the driver selection quietly reverted to + # "build everything" because of a duplicate heading further down. One + # typo disabling every option in the file is not a proportionate answer. + # Later keys win, which is what someone appending a correction expects. + cp = configparser.ConfigParser(strict=False) + # CMAKE VARIABLES ARE CASE-SENSITIVE. + # + # configparser lower-cases keys by default, so + # [webkitgtk] + # ENABLE_JOURNALD_LOG = OFF + # reached cmake as -Denable_journald_log=OFF -- a different variable, + # which cmake accepted and ignored, and webkitgtk went on building + # against systemd/sd-journal.h on a system that has no systemd. meson's + # options are lower-case anyway, which is why this went unnoticed for as + # long as the file only held [mesa]. + cp.optionxform = str + try: + cp.read(path) + except configparser.Error as e: + printWarning(f"{os.path.basename(path)} is broken ({e})") + printWarning(" NO options from that file were applied -- not for this" + " package and not for any other.") + printWarning(f" Fix it and re-run: {path}") + return {} + for sect in cp.sections(): + if sect.lower() == anchor.lower(): + return dict(cp[sect]) + return {} + + +def _machine_opts_for(anchor): + opts = _read_conf_section( + os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), anchor) + opts.update(_read_conf_section( + os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), anchor)) + return opts + + +_ENV_GUARD_RE = re.compile(r'needs ([A-Z][A-Z0-9_]{2,}) to be set') + +def _warn_unset_env(anchor, script_path): + """Say it BEFORE the build, not in the log afterwards. + + The generated script refuses if a variable the book expects is unset -- + but it refuses after unpacking, patching and configuring, with the + failure buried in a log. The script is generated here, so the question + can be asked here: which variables does it need, and are they set? + + Applies to every package, not just TeX Live: whatever the book expects + the user to export, this finds it the moment the script exists.""" + try: + body = open(script_path, encoding="utf-8", errors="replace").read() + except OSError: + return + # SET WHERE THE BUILD LOOKS, NOT WHERE WE HAPPEN TO BE. + # + # This asked os.environ only, so after `packagemanager env --set + # XORG_CONFIG=...` -- which writes /etc/pkgusr/build.env and IS what the + # build reads -- every Xorg package in the plan was still announced as + # missing it. And the advice it gave, `export`, is the one thing that + # does NOT work: builds run as the package user through `su -`, which + # drops the caller's environment. A warning that is wrong and then + # recommends the wrong cure teaches people to ignore warnings. + _build_env = _read_build_env() + missing = [n for n in dict.fromkeys(_ENV_GUARD_RE.findall(body)) + if not (_build_env.get(n) or os.environ.get(n))] + if not missing: + return + # once per VARIABLE, like the build tools -- the rest goes in the summary + fresh = [n for n in missing if n not in _MISSING_ENV] + for n in missing: + _MISSING_ENV.setdefault(n, []).append(anchor) + if not fresh: + return + printWarning("%s needs %s set before it can build:" + % (anchor, ", ".join(fresh))) + for n in fresh: + _sugg = _ENV_SUGGESTIONS.get(n) + printWarning(" packagemanager env --set %s=%s" % (n, _sugg or "...") + + ("" if _sugg else " (the book's page for %s says what)" + % anchor)) + printWarning(" that writes %s, which every build reads;" % BUILD_ENV_FILE) + printWarning(" exporting it in your shell does not reach a package user.") + printWarning(" the build will stop at that check rather than configure with" + " an empty value") + + +# A build-system driver the script CALLS but the page never lists. libheif's +# BLFS page has no Required section at all: CMake reaches it only through +# x265, so on a system where x265 was already installed nothing pulled cmake +# into the plan, and the build died 33 lines in with +# install_libheif: line 33: cmake: command not found +# after three other packages had been built. The dependency list is the +# book's opinion; the COMMANDS are the fact, and they are right here. +_MISSING_TOOLS = {} # tool -> [packages in this plan that call it] +_MISSING_ENV = {} # variable -> [packages in this plan that need it] +_CYCLE_PAIRS = [] # (dependant, dependency) the cycle forced out of order +_IGNORED_REQUIRED = [] # (wanted package, its REQUIRED dep that is ignored) + +# tool -> the BLFS ANCHOR that provides it. These are checked against the +# book at use time (_provider_in_book), because a suggestion naming a package +# the book does not have is the 1.14.10 mistake again: `cargo` was reported +# with "packagemanager install rustc --run --recursive", and the book's anchor +# is `rust`, so the command would have answered "not in the book". +# Meson, Ninja, gperf and libtool are LFS packages, not BLFS ones -- they have +# no anchor here, and are reported without a command rather than a wrong one. +_BUILD_TOOLS = { + "cmake": "cmake", "scons": "scons", "qmake6": "qt6", + "cargo": "rust", "rustc": "rust", + "ant": "apache-ant", "b2": "boost", "bjam": "boost", + "itstool": "itstool", "xsltproc": "libxslt", "doxygen": "doxygen", + "valac": "vala", "node": "nodejs", "npm": "nodejs", "rake": "ruby", + "yasm": "yasm", "nasm": "nasm", + # a stack entry of kind=git clones with it; nothing in the book's command + # text mentions it, so only the tool map can catch it before the build + "git": "git", + # no BLFS anchor: LFS builds these, or the book has no page + "meson": "", "ninja": "", "gperf": "", "libtoolize": "", + "autoreconf": "", "go": "", "mvn": "", "gradle": "", "qmake": "", +} + + +_TOOL_SEEN = {} + + +def _tool_available(tool): + """Is the tool there FOR THE BUILD -- not just for us? + + `shutil.which` reads OUR PATH. A build runs through `su -`, a LOGIN + shell, which reads /etc/profile and /etc/profile.d -- and that is exactly + where BLFS puts the packages it installs outside /usr: + + /opt/rustc/bin (rustc.sh) -> cargo, rustc + /opt/qt6/bin (qt6.sh) -> qmake6 + + So Rust was installed, cargo was on the build's PATH, and this reported + it missing and refused to start. Worse, the plan could not repair it: + rust WAS installed, so adding it changed nothing and the same stop came + back on the second pass -- the loop the user saw. + + Asked the way the build will ask. + """ + if tool in _TOOL_SEEN: + return _TOOL_SEEN[tool] + found = bool(shutil.which(tool)) + if not found: + try: + found = subprocess.run(["bash", "-lc", "command -v " + shlex.quote(tool)], + capture_output=True, timeout=20).returncode == 0 + except (OSError, subprocess.SubprocessError): + found = False + if not found: + found = any(os.access(os.path.join(d, tool), os.X_OK) + for d in glob.glob("/opt/*/bin")) + _TOOL_SEEN[tool] = found + return found + + +def _build_tools_used(script_path): + """The build-system drivers a script's own commands call.""" + try: + body = open(script_path, encoding="utf-8", errors="replace").read() + except OSError: + return [] + used = [] + for line in body.splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + word = line.split()[0] + while word in ("env", "sudo", "nice", "time") and len(line.split()) > 1: + line = line.split(None, 1)[1] + word = line.split()[0] + if "/" in word or word not in _BUILD_TOOLS or word in used: + continue + used.append(word) + return used + + +def _order_by_build_tools(steps): + """Build a package AFTER the tool its commands call. + + The dependency graph is the book's, and the book does not say that libaom + builds with cmake -- libaom's page lists no Required at all. So a plan + could contain BOTH, correctly, and still schedule libaom at position 89 + and CMake at 151, and stop at 89 with `cmake: command not found`. The + resolver was right; it was ordering on the wrong facts. + + The CONSUMER moves down to just after its provider, never the provider up: + moving a package later keeps every dependency it already has in front of + it, while moving one earlier could jump it over its own. + """ + if not steps: + return steps, [] + moved = [] + for _round in range(len(steps)): # bounded: each pass fixes >= 1 + index = {} + for i, (anchor, nv, _r, _p) in enumerate(steps): + index[anchor.lower()] = i + index[nv.lower()] = i + index[nv.rsplit("-", 1)[0].lower()] = i + for i, (anchor, nv, reason, path) in enumerate(steps): + for tool in _build_tools_used(path): + if _tool_available(tool): + continue # already on the system + j = index.get((_BUILD_TOOLS.get(tool) or "").lower()) + if j is None or j <= i: + continue # absent, or already earlier + steps.insert(j + 1, steps.pop(i)) + moved.append((nv, steps[j][1], tool)) + break + else: + continue + break + else: + break + return steps, moved + + +def _provider_of(tool): + """The BLFS package that provides a tool, if the book actually has one.""" + name = _BUILD_TOOLS.get(tool) or "" + if not name: + return "" + try: + return name if book_version(name) else "" + except Exception: + return "" + + +def _install_cmd_for(tool): + prov = _provider_of(tool) + if prov: + return "packagemanager install %s --run --recursive" % prov + return "" + + +def _warn_missing_build_tools(anchor, script_path): + """Name the tool the build will die on, before the build starts.""" + try: + body = open(script_path, encoding="utf-8", errors="replace").read() + except OSError: + return [] + missing = [] + # A CLONE IS NOT A COMMAND IN THE SCRIPT. + # + # `is_git="1"` tells the runner to clone $link, so `git` never appears in + # the command text and the scan could not see it. elogind's entry died on + # `git: command not found` after the account had been created and the + # phase started. + if re.search(r'^is_git=[\'"]?1', body, re.M) and not _tool_available("git"): + missing.append("git") + for line in body.splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + # the command word: skip env prefixes and anything path-qualified + # (./configure is the package's own, not something to install) + word = line.split()[0] + while word in ("env", "sudo", "nice", "time") and len(line.split()) > 1: + line = line.split(None, 1)[1] + word = line.split()[0] + if "/" in word or word not in _BUILD_TOOLS: + continue + if _tool_available(word) or word in missing: + continue + missing.append(word) + if not missing: + return [] + # ONCE PER TOOL, not once per package: a 151-package plan would otherwise + # print the same three lines about cmake thirty-seven times, and a warning + # repeated thirty-seven times is scrollback, not a warning. The rest are + # collected for the summary below the plan. + for w in missing: + first = w not in _MISSING_TOOLS + _MISSING_TOOLS.setdefault(w, []).append(anchor) + if not first: + continue + printWarning("%s builds with `%s`, which is not installed:" % (anchor, w)) + # --recursive, ALWAYS. A tool from the book has its own tree, and + # `packagemanager install cmake --run` builds cmake alone, which then + # fails at "CMAKE_USE_SYSTEM_CURL is ON but a curl is not found" -- + # cURL is one of its four recommended deps. A suggested command that + # does not work is worse than no suggestion, which is also why a tool + # with no BLFS page gets no command at all. + _cmd = _install_cmd_for(w) + printWarning(" " + (_cmd if _cmd else + "(no BLFS page provides `%s` -- it comes from LFS; " + "check it is built and on PATH)" % w)) + printWarning(" (the book's page does not list it as a dependency, so" + " the plan cannot know -- its own commands do)") + return missing + + +def _check_script_prereqs(anchor, script_path): + """Everything knowable from a script before it runs. + + Called for a script whether it was just generated or came from the cache: + the checks belong to USING a script, not to writing one, and pre-generating + a whole plan in one pass (1.14.8) would otherwise have skipped them. + """ + _warn_unset_env(anchor, script_path) + _warn_missing_build_tools(anchor, script_path) + + +def _insert_on_command(body, cmd, arg): + """Add an argument to the first REAL invocation of `cmd`, skipping + comments -- a book NOTE that mentions the command by name is prose.""" + out, done = [], False + for line in body.split("\n"): + if (not done and not line.lstrip().startswith("#") + and re.search(r"(?:^|\s|&&|;)%s\b" % re.escape(cmd), line)): + line = re.sub(r"(%s)\b" % re.escape(cmd), r"\1 " + arg, line, count=1) + done = True + out.append(line) + return "\n".join(out) + + +def apply_machine_opts(anchor, script_path): + """Merge machine.conf's [anchor] options into the script's meson line.""" + opts = _machine_opts_for(anchor) + if not opts or not script_path or not os.path.isfile(script_path): + return + try: + body = open(script_path).read() + except OSError: + return + # AUTOTOOLS TOO. This only knew `meson setup`, so a package built with + # ./configure had no way to take a machine option -- and libassuan needs + # one: its `make` builds a PDF manual that wants TeX, which this system + # deliberately does not have. configure_args is appended to the + # ./configure line; every other key stays a meson -Dkey=value. + # `needs` is for the DEPENDENCY GRAPH, not the build command. blfs reads + # it from the same file (1.13.69) to add an edge the book omits; this + # function turns every remaining key into -Dkey=value, so it passed + # `-Dneeds=glslang` to meson and mesa died with + # ERROR: Unknown option: "needs". + # One file, two readers: each must ignore what belongs to the other. + opts.pop("needs", None) + # ...and REMOVE it if an earlier version already wrote it into the script. + # 1.13.69 stopped adding -Dneeds=, but every script generated while the + # bug existed still carries it, and merging is idempotent so nothing took + # it out again. A key that is not a build option has no business on a + # meson line, whoever put it there. + if "-Dneeds=" in body: + body = re.sub(r"\s*-Dneeds=\S+", "", body) + with open(script_path, "w") as f: + f.write(body) + printInfo(" removed a stale -Dneeds= from %s (it is a dependency " + "key, not a build option)" % os.path.basename(script_path)) + # WRAP MODE, PER PACKAGE. + # + # blfs adds --wrap-mode=nofallback to every `meson setup` so a package + # cannot silently build and install a bundled copy of a dependency (cairo + # once installed its own freetype that way, and the planner then thought + # freetype2 was done). For a package whose subproject is NOT a BLFS + # package at all, that flag is fatal instead of protective: + # gegl: ERROR: Dependency 'libnsgif' is required but not found. + # Use of fallback dependencies is disabled. + # libnsgif and poly2tri-c have no book page; the fallback is the only way + # to get them. So the flag stays the default and becomes overridable: + # [gegl] + # wrap_mode = default + # NINJA DOES NOT READ MAKEFLAGS. + # + # WebKitGTK's page: "some source files require more than 4 GiB of RAM to + # be built ... you should pass -j<N> to ninja (N = RAM / 4 GiB) to avoid + # the job being killed by the kernel OOM killer". MAKEFLAGS cannot do + # that -- ninja ignores it and defaults to nproc+2 -- so there was no way + # to follow the book's advice short of editing the script by hand, and + # the failure it prevents arrives hours in, as a killed compiler. + _jobs = opts.pop("jobs", None) + if _jobs: + _nb = re.sub(r"(?m)^(\s*ninja\b(?![^\n]*\s-j))", + r"\1 -j" + str(_jobs).lstrip("-j"), body) + if _nb != body: + body = _nb + with open(script_path, "w") as f: + f.write(body) + printInfo(" machine.conf [%s]: ninja -j%s" % (anchor, _jobs)) + _wrap = opts.pop("wrap_mode", None) + if _wrap: + if "meson setup" not in body: + printWarning(" machine.conf has [%s] wrap_mode but the script has" + " no `meson setup` line -- not applied" % anchor) + else: + new_body = re.sub(r"--wrap-mode=\S+", "--wrap-mode=" + _wrap, body) + if new_body == body: # none there yet: add one + new_body = re.sub(r"(\bmeson\s+setup\b)", + r"\1 --wrap-mode=" + _wrap, body, count=1) + if new_body != body: + body = new_body + with open(script_path, "w") as f: + f.write(body) + printInfo(" machine.conf [%s]: wrap-mode %s" % (anchor, _wrap)) + _conf_args = opts.pop("configure_args", None) + if _conf_args: + if "./configure" not in body: + printWarning(" machine.conf has [%s] configure_args but the script" + " has no ./configure line -- not applied" % anchor) + else: + if _conf_args in body: + pass # already applied; merging is idempotent + else: + # AN --enable-X MUST BEAT THE BOOK'S --disable-X. + # + # configure_args was PREPENDED, and autoconf takes the last + # occurrence: `[iptables] configure_args = --enable-nftables` + # produced + # ./configure --enable-nftables --prefix=/usr \ + # --disable-nftables ... + # so the book's flag won and the option silently did nothing. + # Drop the opposite form of each flag we are adding first. + # ...line by line, so a continuation that held nothing but + # the removed flag goes with it. Deleting just the token left + # ./configure --enable-nftables --prefix=/usr \ \ + # -- a backslash-escaped space, which is an argument. + for _flag in _conf_args.split(): + _m = re.match(r"^--(enable|disable|with|without)-(.+)$", _flag) + if not _m: + continue + _opp = {"enable": "disable", "disable": "enable", + "with": "without", "without": "with"}[_m.group(1)] + _pat = re.compile(r"--%s-%s\b[^\s\\]*" + % (_opp, re.escape(_m.group(2)))) + _out = [] + for _line in body.split("\n"): + if not _pat.search(_line): + _out.append(_line) + continue + _stripped = _pat.sub("", _line) + if _stripped.strip() in ("", "\\"): + continue # the line was only that flag + _out.append(_stripped) + body = "\n".join(_out) + body = re.sub(r"(\./configure\b)", r"\1 " + _conf_args, body, count=1) + if _conf_args not in open(script_path).read(): + with open(script_path, "w") as f: + f.write(body) + printInfo(" machine.conf [%s]: configure %s" % (anchor, _conf_args)) + if not opts: + return + # CMAKE TAKES -D OPTIONS TOO. + # + # This only knew `meson setup`, so for a cmake package it printed + # machine.conf has [webkitgtk] but the script has no `meson setup` + # line -- options NOT applied + # and the person had nowhere to put -D ENABLE_SPELLCHECK=OFF. Both build + # systems take -Dkey=value; only the command word differs. (autotools is + # already handled above, by configure_args.) + _anchor_cmd = "meson setup" if "meson setup" in body else ( + "cmake" if re.search(r"^\s*cmake\b", body, re.M) else "") + if not _anchor_cmd: + printWarning(f" machine.conf has [{anchor}] but the script has no " + f"`meson setup` or `cmake` line -- options NOT applied") + printWarning(" (an autotools package takes configure_args instead)") + return + new = body + applied = [] + for k, v in opts.items(): + # `-D key=value` WITH A SPACE is how the book writes them. + # + # This matched only "-Dkey=", so the book's own `-D gallium-drivers=auto` + # was left in place and OUR value was merely prepended -- and meson + # takes the LAST occurrence. mesa built for every driver instead of + # radeonsi and then failed on a dependency it should never have wanted: + # ERROR: Dependency "libdrm_intel" not found + # A VALUE WITH A SPACE IS ONE VALUE. + # + # The fix for the commonest GCC-15 breakage is + # [webrtc-audio-processing] + # cpp_args = -include cstdint + # and this pasted it in unquoted, so the shell split it and meson saw + # `-Dcpp_args=-include` plus a stray `cstdint`. Every option here that + # takes a flag LIST -- c_args, cpp_args, c_link_args -- has this shape; + # the single-word examples in the help text hid it. + _v = v if not re.search(r"\s", v) else "'%s'" % v.replace("'", "'\\''") + pat = re.compile(r"-D\s*%s\s*=(?:'[^']*'|\S+)" % re.escape(k)) + if pat.search(new): + new = pat.sub("-D%s=%s" % (k, _v), new) + else: + # ...and put it on the COMMAND, not in the first line that happens + # to contain the word. WebKitGTK's page opens with a NOTE + # mentioning "the cmake command", so every option landed inside + # that comment and none of them reached the build. + new = _insert_on_command(new, _anchor_cmd, "-D%s=%s" % (k, _v)) + applied.append("%s=%s" % (k, v)) + if new != body: + with open(script_path, "w") as f: + f.write(new) + printInfo(" machine.conf [%s]: %s" % (anchor, ", ".join(applied))) def _which_program(prog): @@ -929,6 +2183,89 @@ def _find_library(lib): return None +_STACK_TARGETS = None # name -> (prog, have) from every stack file, once +_STACK_TARGET_SOURCE = {} # name -> the stack file the declaration came from + + +def _stack_declared_targets(): + """What every git/tar entry in every stack file says proves it installed. + + A stack entry's `prog=` and `have=` are its own statement of what to + check. Until 1.14.87 they were not carried into the generated script, and + install_last -- the record of the script AS IT RAN -- keeps whatever it + was given at the time. Reading the stack files directly means verify + checks the right thing for an entry built under any version, without + rewriting a record. Read once per run; a dozen small files. + """ + global _STACK_TARGETS + if _STACK_TARGETS is not None: + return _STACK_TARGETS + out = {} + dirs = [os.environ.get("PM_STACK_DIR") or "/etc/pkgusr/stacks", + os.path.join(os.path.dirname(os.path.abspath(__file__)), "stacks")] + for d in dirs: + try: + names = sorted(os.listdir(d)) + except OSError: + continue + for f in names: + if not f.endswith(".stack"): + continue + try: + rows = _parse_stack_file(os.path.join(d, f)) + except (OSError, ValueError): + continue + for r in rows: + if r.get("kind") not in ("git", "tar"): + continue + o = r["opts"] + if not (o.get("prog") or o.get("have")): + continue + # ALL declarations, not the first: the user's stack in + # /etc/pkgusr/stacks is never overwritten (1.14.58), so a + # corrected prog=/have= shipped in a newer stack sits in a + # .new file beside it -- and the first-found rule meant the + # stale one always won, and the user was asked to hand-merge + # a metadata fix. Read every copy; prefer the one whose + # proof is actually on disk. + out.setdefault(r["target"], []).append( + (o.get("prog", ""), o.get("have", ""), os.path.join(d, f))) + # the .new files a protected install leaves beside the user's copies + # carry the shipped fixes; they count as declarations too + for f in names: + if not f.endswith(".stack.new"): + continue + try: + rows = _parse_stack_file(os.path.join(d, f)) + except (OSError, ValueError): + continue + for r in rows: + if r.get("kind") not in ("git", "tar"): + continue + o = r["opts"] + if o.get("prog") or o.get("have"): + out.setdefault(r["target"], []).append( + (o.get("prog", ""), o.get("have", ""), os.path.join(d, f))) + chosen = {} + for name, alts in out.items(): + pick = None + for prog, have, src in alts: + ok = True + if prog and not shutil.which(prog): + ok = False + if have and not os.path.exists(have): + ok = False + if ok: + pick = (prog, have, src) + break + if pick is None: + pick = alts[0] # nothing satisfied: report the first + chosen[name] = (pick[0], pick[1]) + _STACK_TARGET_SOURCE[name] = pick[2] + _STACK_TARGETS = chosen + return chosen + + def install_targets(meta): """Collect concrete things the install script says it installs: programs, libraries, directories (from installed_program(s)/-libraries/ @@ -941,6 +2278,19 @@ def install_targets(meta): if meta.get("installed_directory"): dirs.append(meta["installed_directory"]) dirs += meta.get("installed_directories", []) + # A STACK ENTRY'S OWN DECLARATION WINS OVER A STALE RECORD. + # + # If the stack says what proves this package installed, use that: it is + # current, it is the entry's own word, and it is what a rebuilt script + # would carry anyway. A self-named program left in an old record + # (installed_program="libsrtp") is dropped in favour of it. + name = (meta.get("name") or "").strip() + decl = _stack_declared_targets().get(name) if name else None + if decl: + prog, have = decl + progs = [p for p in progs if p != name] + ([prog] if prog else []) + if have and have not in dirs: + dirs.append(have) return ([p for p in progs if p], [l for l in libs if l], [d for d in dirs if d]) @@ -974,8 +2324,11 @@ def assess(u, expected=None, run_cmd=True): if not u.home and not u.has_user: return "not installed", "no package-user and no home" if not (u.home and u.has_user): - return "broken", ("has a home dir but no user" if u.home - else "has a user but no home dir") + # NAME THE DIRECTORY. "root broken has a home dir but no user" sent + # someone looking for /usr/src/root, which is not where it was; the + # path is right here and is the whole of what they need to act. + return "broken", (("no account for the directory %s" % u.home) + if u.home else "has a user but no home dir") m = u.meta if not (m.get("name_version") or u._manifest_has_real_paths()): return "not installed", "user + home exist, but nothing installed yet" @@ -1011,26 +2364,65 @@ def assess(u, expected=None, run_cmd=True): missing.append(f"lib {l}") dir_version_ok = None for d in dirs: - if not os.path.isdir(d): + # exists, not isdir: a stack entry's `have=` is whatever proves the + # package landed -- often a .pc file or a header, not a directory -- + # and it is recorded here (1.14.87). A path that is there is proof + # either way; a book's installed_directories are directories anyway. + if not os.path.exists(d): missing.append(f"dir {d}") elif expected: # versioned dir check want = _version_number(expected) if want and re.search(r"\d", os.path.basename(d)): dir_version_ok = (want in d) if progs or libs or dirs: + # A TARGET THAT WAS NEVER A REAL TARGET. + # + # Until 1.14.87 a stack entry's script got installed_program=<its own + # name>, so libsrtp, tllist and dejavu-fonts were checked for binaries + # that do not exist. Their install_last files still say that -- it is + # the record of the script AS IT RAN, and rewriting it would be a lie + # about history -- so verify would report "missing: program libsrtp" + # for a package that is perfectly installed. When the ONLY thing + # missing is a program named exactly like the package, and the + # manifest's files are on disk, the script named no real check. + _pkgname = (m.get("name") or "").strip() + if (missing and len(missing) == 1 and _pkgname + and missing[0] == f"program {_pkgname}"): + return "unvalidated", ("its script names no verifiable target" + " (rebuilt scripts declare one)") if missing: + # SAY WHERE THE EXPECTATION CAME FROM. "missing: dir + # /usr/include/wlr" sent the user to look for a directory; the + # useful fact was that /etc/pkgusr/stacks/sway.stack still said + # have=/usr/include/wlr while sway.stack.new beside it -- the + # shipped fix -- said the .pc file. The user's copy is never + # overwritten (1.14.58), so the place to fix it must be named. + _src = _STACK_TARGET_SOURCE.get(_pkgname) + _where = (" (declared by %s)" % _src) if _src else "" return "unvalidated", "missing: " + ", ".join(missing[:4]) + ( - " ..." if len(missing) > 4 else "") + " ..." if len(missing) > 4 else "") + _where if dir_version_ok is False: return "unvalidated", "installed dir version differs from the book" return "installed", "all installed_* targets present" + ( " (dir version matches)" if dir_version_ok else "") - # 3) evidence but nothing concrete to check + # 3) NOTHING TO CHECK IS NOT THE SAME AS A FAILED CHECK. + # + # Most pages name no installed_programs/libraries/directories, so this + # answered "unvalidated" for 226 of 242 packages on a healthy system -- + # and "unvalidated" reads like a problem. A health check whose normal + # output is two hundred warnings is one nobody reads, and the twelve lines + # that DID matter were buried in it. + # + # The manifest has already been checked at this point: its paths exist + # (that is what got us here). So the honest answer is installed, with the + # limit stated -- and anything genuinely wrong still comes out above as + # "not installed", "broken" or a missing target. if m.get("name_version"): - return "unvalidated", (f"install_last says {m['name_version']} but no " - f"installed_* targets / validate_cmd to confirm") - return "unvalidated", "pkg.lst has real files, but no targets to validate" + return "installed", (f"{m['name_version']}, recorded files present " + f"(the page names no targets to verify against)") + return "installed", ("recorded files present " + "(the page names no targets to verify against)") def _install_evidence(u, expected=None): @@ -1083,19 +2475,86 @@ def cmd_verify(args): f"exist / validate_cmd matches);\n" f" unvalidated = looks installed but couldn't confirm; " f"not installed = nothing built{colors.normal}\n") - counts = {} + counts, _states = {}, {} for name in sorted(set(names)): u = gather_user(name) anchor = u.meta.get("name") or name st, why = assess(u, versions.get(anchor)) counts[st] = counts.get(st, 0) + 1 + _states[name] = st color = _STATE_COLOR.get(st, "") print(f" {name:<24} {color}{st:<12}{colors.normal} {why}") + # A PIP MODULE INSTALLED BEFORE 1.14.37 HAS NO RECORD AT ALL. + # + # `packagemanager pip` wrote neither pkg.lst nor install_last, so the + # bootstrap set reads "nothing installed yet" while being in daily use. + # If the module IMPORTS, the account is real and only its paperwork is + # missing -- say so, and offer the one command that fixes it. + if st == "not installed" and u.has_user and u.home \ + and not u.meta.get("name_version") and _python_module_here(name): + st, why = "unvalidated", ( + "installed as a Python module, but with no record -- run:" + " packagemanager pip record %s" % name) + counts["not installed"] -= 1 + counts[st] = counts.get(st, 0) + 1 + _states[name] = st + color = _STATE_COLOR.get(st, "") + print(f" {name:<24} {color}{st:<12}{colors.normal} {why}") + continue if args.fix and st == "not installed" and u.has_user and u.home \ and not u.meta.get("name_version"): printInfo(f" fixing: regenerating pkg.lst for {name}...") write_pkg_list(name, background=False) print("\n " + " ".join(f"{k}: {v}" for k, v in sorted(counts.items()))) + # + # `unvalidated` is the normal state for most of a system -- it means only + # that the page named no file to check, not that anything is wrong -- and + # on a real tree it is 226 of 242 lines. The dozen that need action + # scroll past inside it. Repeat them at the end, where the eye lands, + # with the command that fixes them. + # ONE PASS, NOT TWO. This re-assessed every package to find the ones to + # repeat -- a second validate command per package on a 242-package tree. + # The states were already computed in the loop above. + _act = [n for n, st in _states.items() if st in ("not installed", "broken")] + if _act: + print() + printWarning("%d package(s) need attention (everything else is fine " + "-- 'unvalidated' just means" % len(_act)) + printWarning("the book named no file to check):") + for n in _act: + u = gather_user(n) + st, why = assess(u, versions.get(u.meta.get("name") or n)) + printWarning(" %-24s %-14s %s" % (n, st, why)) + # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. + # + # The list was pasted straight into an install command that could not + # work: collector groups and a stray `root` are not packages at all, + # and pkgusr / linux-headers are LFS steps with no BLFS page. The one + # bad name then killed the whole run (fixed in 1.14.36), but even + # without that, offering a command that cannot succeed is worse than + # offering none. + _vers = book_versions_all() or {} + _book = [n for n in _act if _vers.get(n) or _vers.get(n.lower())] + _other = [n for n in _act if n not in _book] + if _book: + printInfo(" Rebuild the BLFS ones:") + printInfo(" packagemanager install %s --run --recursive" + % " ".join(_book)) + if _other: + printInfo(" Not BLFS packages, and not rebuilt by that command:") + printInfo(" " + ", ".join(_other)) + printInfo(" LFS steps (linux-headers, pkgusr, ...) belong to the" + " chroot build: lfs-helper build <name> --force") + # NOT `packagemanager remove root`. For a directory with no + # account the fix is to delete the DIRECTORY, and suggesting a + # remove command against a name like `root` is an invitation to + # an accident. + printInfo(" Anything else is a directory or account with no" + " package behind it -- the line") + printInfo(" above names it; a leftover directory can simply be" + " removed.") + else: + printSuccess(" nothing needs attention.") def cmd_info(args): @@ -1131,6 +2590,8 @@ def cmd_info(args): anchor = u.meta.get("name") or u.name or u.search st, _ = assess(u, versions.get(anchor), run_cmd=False) counts[st] = counts.get(st, 0) + 1 + _states[name] = st + _states[name] = st print(f"\nTotal: {len(users)} " + " ".join(f"{k}: {v}" for k, v in sorted(counts.items()))) @@ -1147,13 +2608,62 @@ def cmd_info(args): # add : usermod -a -G <g> <pkg> # remove : gpasswd -d <pkg> <g> +# Collector groups live at 90000+, by convention and by the sanity check. +COLLECTOR_GID_MIN = 90000 +COLLECTOR_GID_MAX = 99000 + + def nimgnu_name(name): + """The collector group for a package or directory name. + + ONE ACCOUNT, ONE PREFIX -- and a group named after a package must not carry + the package prefix as well. Given the owner `p_openssl` this returned + `nimgnu_p_openssl`, which the prompt did not even offer (it showed + `nimgnu_openssl`) and which the sanity report flags as outside every + convention. Strip the owner's prefix before adding this one. + """ cp = collector_prefix() if name.startswith(cp) or name.startswith(user_prefix()) or name == "audio": return name + for pfx in (pkgusr_prefix(), cfguser_prefix()): + if pfx and name.startswith(pfx): + name = name[len(pfx):] + break return cp + name +def _next_collector_gid(): + """The first free gid in the collector range. + + `groupadd <name>` with no -g takes the next free system gid, which put a + collector group at 10093 -- in the middle of the package-user accounts, + where nothing expects it: + !! group nimgnu_p_openssl (gid 10093) is outside every convention + """ + used = set() + if grp is not None: + try: + used = {g.gr_gid for g in grp.getgrall()} + except Exception: + used = set() + for gid in range(COLLECTOR_GID_MIN, COLLECTOR_GID_MAX): + if gid not in used: + return gid + return None + + +def _create_collector_group(group): + """Create a collector group IN ITS RANGE. True if it exists afterwards.""" + if _group_exists(group): + return True + gid = _next_collector_gid() + cmd = [_cmd_path("groupadd")] + if gid is not None: + cmd += ["-g", str(gid)] + cmd.append(group) + return _run(cmd) or _group_exists(group) + + def find_add_package_user(): return os.environ.get("ADD_PACKAGE_USER_BIN", "add_package_user").split() @@ -1361,6 +2871,29 @@ def create_package_user(name, home=None): name, "10000", "20000"] printInfo(f"creating package-user {name}") if _run(cmd): + # THE RECORDED HOME MUST BE THE REAL ONE. + # + # add_package_user takes no home argument, so the hint's script + # uses its own default -- /usr/src/<name>. init_package_user_home + # then created and chowned the home this tool computes, + # /usr/src/pkgusr/p_name, and the two never met. A real run said + # both in the same report: + # $ chown -R -h p_urllib3:p_urllib3 /usr/src/pkgusr/p_urllib3 + # home directory created: /usr/src/p_urllib3 + # An account whose passwd entry points somewhere its files are not + # lands `su - p_urllib3` in a directory that does not exist. + # + # And repointing is only HALF the repair. usermod -d moves the + # entry, not the directory -- so every bootstrap left a husk + # beside the roots (/usr/src/p_wget, /usr/src/p_requests, ...) + # holding the skeleton add_package_user had copied in. lfs-helper + # fix-home is the one implementation of the whole repair: repoint, + # merge the stray into the real home, delete nothing that differs. + if shutil.which("lfs-helper"): + _run(["lfs-helper", "fix-home", name, "--run"]) + else: + _relocate_stray_home(name, home) + _fix_recorded_home(name, home) init_package_user_home(name, home) return True printWarning("add_package_user failed -- falling back to useradd") @@ -1385,6 +2918,50 @@ def create_package_user(name, home=None): return ok +def _relocate_stray_home(name, home): + """Move the directory add_package_user just created onto the real home. + + Fallback only -- lfs-helper fix-home is the full repair (it can merge). + This handles the clean case: the wrong home exists, the right one does + not yet, so the whole directory simply moves. Anything messier is left + for fix-home and reported by verify.""" + try: + recorded = pwd.getpwnam(name).pw_dir + except (KeyError, TypeError): + return + if os.path.normpath(recorded) == os.path.normpath(home): + return + if os.path.isdir(recorded) and not os.path.exists(home): + try: + os.makedirs(os.path.dirname(home), exist_ok=True) + shutil.move(recorded, home) + printInfo(f" {name}: moved {recorded} -> {home}") + except OSError as e: + printWarning(f" {name}: could not move {recorded} -> {home}: {e}") + elif os.path.isdir(recorded): + printWarning(f" {name}: both {recorded} and {home} exist -- " + f"merge with: lfs-helper fix-home {name} --run") + + +def _fix_recorded_home(name, home): + """Point the account's passwd entry at the home this tool actually uses.""" + try: + recorded = pwd.getpwnam(name).pw_dir + except (KeyError, TypeError): + return + if os.path.normpath(recorded) == os.path.normpath(home): + return + if not _have_cmd("usermod"): + printWarning(f" {name}: home is recorded as {recorded}, but its files " + f"are in {home} -- and usermod is not available to fix it") + return + if _run([_cmd_path("usermod"), "-d", home, name]): + printInfo(f" {name}: home corrected to {home} (was {recorded})") + else: + printWarning(f" {name}: could not correct the recorded home " + f"({recorded} -> {home})") + + def _group_exists(name): try: grp.getgrnam(name) @@ -1394,11 +2971,16 @@ def _group_exists(name): def _user_exists(name): - try: - pwd.getpwnam(name) - return True - except (KeyError, TypeError): - return False + """Does this account exist? Accepts an account name or a package name -- + the raw name is tried first, so a real account ('root', 'lfs') is never + reinterpreted as a package.""" + for n in (name, pkgusr_name(name)): + try: + pwd.getpwnam(n) + return True + except (KeyError, TypeError): + continue + return False def _in_group(user, group): @@ -1645,13 +3227,39 @@ def is_shared_dir(path): # burned ~90 seconds of CPU and made the machine crawl. So: only read the tail # (errors are at the end), skip any line without the words we care about, and # anchor the patterns per line so there is nothing to backtrack over. -_PERM_HINT = ("Permission denied", "Operation not permitted", "PermissionError") +# p11-kit reports errno 13 as a NUMBER -- "Unknown error 13" -- so a filter +# looking for the word "denied" skips the line entirely: +# p11-kit: couldn't create file: /etc/ssl/certs/X.pem: Unknown error 13 +# 13 is EACCES. make-ca then exits 0 having written the other stores, so +# nothing else flags it either. +_PERM_HINT = ("Permission denied", "Operation not permitted", "PermissionError", + "Unknown error 13", "Errno 13") + +# THE QUOTES ARE NOT ALWAYS APOSTROPHES. +# +# coreutils quotes paths with the locale's quoting style, which under a UTF-8 +# locale is U+2018/U+2019 -- so the real failure +# +# install: cannot change permissions of '/usr/sbin': Operation not permitted +# +# matched nothing, no directory was extracted, and the auto-grant that exists +# precisely for this case never fired. packagemanager printed the manual +# command instead, as though it had no opinion. +_Q = r"['\u2018\u2019\"\u201c\u201d`]" _PERM_PATTERNS = [ - re.compile(r"PermissionError[^']*'([^']+)'"), - re.compile(r"(?:Permission denied|Operation not permitted)[:,]?\s*'([^']+)'"), - re.compile(r"cannot (?:create|remove|touch|open)[^']*'([^']+)'"), + re.compile(r"PermissionError[^'\u2018]*" + _Q + r"([^'\u2018\u2019\"]+)" + _Q), + re.compile(r"(?:Permission denied|Operation not permitted)[:,]?\s*" + _Q + + r"([^'\u2018\u2019\"]+)" + _Q), + # `change permissions` was missing, which is the one install(1) uses when + # the directory already exists and only its mode cannot be set -- the + # commonest way a package meets a directory another package owns. + re.compile(r"cannot (?:create|remove|touch|open|change permissions of)" + + r"[^'\u2018]*" + _Q + r"([^'\u2018\u2019\"]+)" + _Q), re.compile(r"^\s*(/[^\s:]+):\s*Permission denied"), + # p11-kit, and anything else that prints "<verb>: /path: <errno text>" + re.compile(r"couldn't (?:create|write|open)[^/]*(/[^\s:]+)"), + re.compile(r"(/[^\s:]+):\s*(?:Unknown error 13|Errno 13)"), ] _LOG_TAIL_BYTES = 512 * 1024 # the error is never at the start @@ -1730,6 +3338,26 @@ def auto_grant_from_log(user, log_path, verbose=True): except (KeyError, OSError): pass + # ALREADY IN THE GROUP: granting again cannot be the answer. + # + # /usr/sbin is root:install and the user is in install, so it could + # create files there -- the failure was `install -vdm755` trying to + # CHMOD a directory root owns, which is the wrappers' job, not the + # group's. Granting a group the user already has, twice, looks like + # progress and is not. + try: + if cur and cur in user_supplementary_groups(user): + printWarning(f" {user} is already in group '{cur}' on {d} --") + printWarning( " this is not a group problem. A book command like") + printWarning( " install -vdm755 " + d) + printWarning( " chmods a directory root owns, which no package user") + printWarning( " may do; the package-user wrappers make it succeed and") + printWarning( " change nothing. Check them with:") + printWarning( " lfs-helper make-wrappers --run") + continue + except Exception: + pass + if is_shared_dir(d) and cur != "install": # shared by nature: it belongs to the install group, and every # package user is already in that -- no collector group needed @@ -1755,7 +3383,7 @@ def auto_grant_from_log(user, log_path, verbose=True): if not _group_exists(group): if verbose: printInfo(f" creating collector group {group}") - if not _run([_cmd_path("groupadd"), group]): + if not _create_collector_group(group): continue if owner not in ("root", "") and _user_exists(owner): _run([_cmd_path("usermod"), "-a", "-G", group, owner]) @@ -1787,8 +3415,30 @@ def _latest_log(name): def _run_pm_install_once(name, path, env): - """Run the install engine exactly once, with no recovery.""" - cmd = find_pm_install() + [name, path] + """Run the install engine exactly once, with no recovery. + + THE ENGINE IS HANDED AN ACCOUNT NAME, NOT A PACKAGE NAME. + + Argument 1 is used by packagemanager_install for exactly one thing: the + user to create, own the files and run the build as. This passed the bare + package name, so the engine did + + id wget || addUser wget + + and created an account literally called `wget`, living in /usr/src/wget -- + no prefix, no subdirectory, colliding with any real account of that name. + A fresh tree proved it: `id wget` said no such user until the install ran. + + Half this codebase asks for the name -- pkgusr_name, owner-name, + pkgusr-home -- and this one call assumed it. Same species as + `chown: invalid user: 'wget:wget'` and `usermod: user 'requests' does not + exist`. pkgusr_name is idempotent, so a caller that already resolved is + unaffected. + """ + # Staging the downloaded tarball used to happen here, and could not: for a + # new package the home does not exist yet, because the engine creates it. + # It lives in packagemanager_install now, after the account is made. + cmd = find_pm_install() + [pkgusr_name(name), path] printInfo(" $ " + " ".join(shlex.quote(c) for c in cmd)) try: return subprocess.run(cmd, env=env) @@ -1814,7 +3464,12 @@ def run_pm_install(name, path, env, auto_fix=True, max_rounds=4): rounds = 0 while result.returncode != 0 and auto_fix and rounds < max_rounds: log = _latest_log(name) - granted = auto_grant_from_log(name, log) + # THE ACCOUNT, NOT THE PACKAGE. This passed `make-ca`, so the grant + # it worked out was correct and then applied to a user that does not + # exist: + # usermod: user 'make-ca' does not exist + # Third time this species has surfaced in this file. + granted = auto_grant_from_log(pkgusr_name(name), log) if not granted: break # nothing to grant: a real build error rounds += 1 @@ -2024,21 +3679,36 @@ def cmd_search(args): # optional) and the parent package that required it. --tree shows blfs's tree # verbatim; --no-recommended / --optional pass through. -def blfs_order_anchors(name, no_recommended=False, optional=False): +def blfs_order_anchors(name, no_recommended=False, optional=False, ignore=None): """Return (order, missing): order is [(anchor, name_version, kind, parent)], - missing is [(name, kind, parent)], via blfs order --anchors.""" + missing is [(name, kind, parent)], via blfs order --anchors. + + `ignore` is passed to the WALK, not applied to the result: a package that + is not built should not drag its own dependencies in behind it. + """ argv = ["order", name, "--anchors"] if no_recommended: argv.append("--no-recommended") if optional: argv.append("--optional") + if ignore: + argv += ["--ignore", ",".join(sorted(ignore))] rc, out, err = run_blfs(argv) if rc == 127: printError("blfs not found (set BLFS_BIN or put blfs on PATH).") sys.exit(1) if rc != 0: + # ONE BAD NAME MUST NOT KILL THE WHOLE COMMAND. + # + # `blfs order` exits 1 for a name the book does not have, and this + # turned that into sys.exit(1) for the entire run. So + # packagemanager install <12 names> --run --recursive + # died on the first name that is not a BLFS package -- and since the + # exit came before any plan was printed, the person saw the book note + # and nothing else. Report it and hand back an empty order; the + # caller records it in `not_found` and carries on with the rest. printError((err or out).strip() or f"blfs order {name} failed") - sys.exit(1) + return [], [] order, missing = [], [] for line in out.splitlines(): if "\t" not in line: # skip any stray output @@ -2047,7 +3717,13 @@ def blfs_order_anchors(name, no_recommended=False, optional=False): cols += [""] * (4 - len(cols)) left, nv, kind, parent = cols[:4] left = left.strip() - if left == "#missing": + if left == "#ignored-required": + _IGNORED_REQUIRED.append((nv.strip(), kind.strip())) + elif left == "#cycle": + # a dependant the walker had to build BEFORE something it + # recommends, because the cycle admits no order satisfying both + _CYCLE_PAIRS.append((nv.strip(), kind.strip())) + elif left == "#missing": missing.append((nv.strip(), kind, parent)) elif left and " " not in left and nv.strip(): order.append((left, nv.strip(), kind, parent)) @@ -2101,7 +3777,7 @@ def cmd_dependencies(args): # Resolve the recursive install order (via blfs), work out which packages # actually need building (state != installed, or --reinstall), generate an # install script for each with `blfs script`, and either print the plan -# (default, safe) or run packagemanager_install for each in order (--run). +# (default, safe) or run lfs-helper pm-install for each in order (--run). # # -f/--force just the named package(s), skip dependency resolution # -r/--reinstall rebuild even packages that are already up to date @@ -2132,6 +3808,70 @@ def _blfs_book_available(): return False +def _download_tool_available(): + return bool(shutil.which("wget") or shutil.which("curl")) + + +def _explain_no_blfs_book(target): + """No book, and say so in terms of what can actually be done from here. + + "run `blfs fetch`" is sound advice on a system that can reach the network. + On a fresh chroot it is the circle itself: fetching needs the download + tool, the download tool is looked up in the book, and the book is what is + missing. Telling someone to close a circle by going round it once more is + worse than saying nothing. + + The host resolves wget's URL FROM THE BOOK too (`blfs sources wget`, via + lfs's bootstrap_sources), so a host with no cached book staged no wget + tarball either -- both halves are missing together, and both are fixed in + the same trip. Say which of them is actually absent rather than making + the person find out one at a time. + """ + srcdir = setup_sources_dir() + have_tool = _download_tool_available() + try: + have_tarball = any(f.startswith("wget-") and ".tar" in f + for f in os.listdir(srcdir)) + except OSError: + have_tarball = False + + printError(f"{target}: there is no BLFS book on this system to look " + f"it up in.") + printInfo("") + if have_tool: + printInfo(" Fetch one:") + printInfo(" blfs fetch") + printInfo(" or import a copy:") + printInfo(" blfs import /path/to/BLFS-BOOK-...-nochunks.html") + printInfo("") + return + + printInfo(" There is no download tool on this system either, so") + printInfo(" `blfs fetch` cannot work from in here -- that circle is what") + printInfo(" bootstrap exists to break, and it is broken from the HOST.") + printInfo("") + printInfo(" The host stages both halves into the tree's own sources") + printInfo(f" directory, which it sees as $LFS{srcdir}:") + printInfo("") + printInfo(" # on the host, which still has a network") + printInfo(" blfs fetch # or: blfs import <book.html>") + if not have_tarball: + printInfo(" blfs sources wget # the URL to download") + printInfo(f" wget -P $LFS{srcdir} <that URL>") + printInfo(f" cp <book.html> $LFS{srcdir}/") + printInfo("") + printInfo(" # back in here") + printInfo(f" blfs import {srcdir}/<book.html>") + printInfo(" packagemanager bootstrap --run") + printInfo("") + printInfo(" Already on this system: " + + ("the wget tarball is in %s" % srcdir if have_tarball + else "no wget tarball in %s either" % srcdir)) + printInfo(" (`lfs build-system install-tools` copies a book the host has") + printInfo(" cached; it said so at the time if there was none.)") + printInfo("") + + def _explain_not_in_book(target): """Say why a package could not be found, and what to do about it. @@ -2139,14 +3879,7 @@ def _explain_not_in_book(target): book -- a fresh chroot has the LFS book but not the BLFS one, and every lookup fails the same way.""" if not _blfs_book_available(): - printError(f"{target}: there is no BLFS book on this system to look " - f"it up in.") - printInfo("") - printInfo(" Download it (needs working networking):") - printInfo(" blfs fetch") - printInfo(" or copy one in from the host and import it:") - printInfo(" blfs import /path/to/BLFS-BOOK-...-nochunks.html") - printInfo("") + _explain_no_blfs_book(target) return printError(f"{target}: not in the BLFS book.") rc, out, _e = run_blfs(["search", target]) @@ -2165,6 +3898,12 @@ def build_install_plan(args): anchor -> (kind, parent) so we can show 'required by <parent>'.""" ignore = {x for x in (args.ignore or "").split(",") if x} plan, seen, missing_all, why = [], set(), [], {} + # WHAT IS ALREADY DONE, counted as we go. + # + # A re-run recomputes the whole tree and prints only what is LEFT, so a + # plan that has shrunk 181 -> 151 -> 135 across three runs reads as three + # identical fresh starts. The progress was real and invisible. + build_install_plan.already = set() _not_found = [] # Reading the book can take many seconds the first time (it is indexed # once per version). Without a word on screen that looks like a hang, so @@ -2180,6 +3919,8 @@ def build_install_plan(args): u = gather_user(anchor) up_to_date = (u.state == "installed" and (not u.version or u.version == nv)) + if up_to_date: + build_install_plan.already.add(anchor) if args.reinstall: plan.append((anchor, nv, "reinstall" if up_to_date else ("update" if u.state == "installed" else "new"))) @@ -2199,8 +3940,40 @@ def build_install_plan(args): continue consider(target, nv, "requested", "") else: + # THE PRUNE BELONGS TO THE WALK, AND NOW IT IS THERE (1.14.24). + # + # 1.12.82 pruned HERE, on the finished list, and was reverted for + # a good reason: `blfs order` reports ONE parent per package, so + # graphite2 -- recorded under an ignored parent but genuinely + # needed by harfbuzz -- was dropped, and the build failed with + # Dependency 'graphite2' is required but not found + # The note then read "pruning needs every edge, which this output + # does not carry". Right, so it moved to where the edges are: + # walk_deps does not descend into an ignored package, and anything + # another package needs is still reached through that package. + # + # Until then --ignore dropped only the named package and kept + # everything it alone wanted: harfbuzz with libreoffice ignored + # planned 230 packages, of which PostgreSQL, unixODBC, CLucene, + # Redland, OpenJDK, apache-ant and Cups were nobody else's. It + # comes to 46 now, with icu, libxml2, fontconfig, curl and + # graphite2 all still present. sway.stack's 80-name avoid list + # was the workaround. order, missing = blfs_order_anchors( - target, args.ignore_recommended, args.optional) + target, args.ignore_recommended, args.optional, ignore=ignore) + # A NAME THE BOOK DOES NOT HAVE, IN THE RECURSIVE PATH. + # + # The non-recursive branch explains and records it; this one got + # an empty order back and moved on WITHOUT A WORD. So + # packagemanager install <12 names> --run --recursive + # printed the book note and nothing else, and the person could not + # tell whether it had worked, refused, or crashed. Silence is the + # one answer a command must never give. + if not order: + clear_status() + _explain_not_in_book(target) + _not_found.append(target) + continue missing_all.extend(m for m, _, _ in missing) for anchor, nv, kind, parent in order: consider(anchor, nv, @@ -2289,7 +4062,7 @@ def install_local(name, path, run, yes, verb="install", test=False, clean=False) print(f"\n script : {path}") if not run: printInfo("\nDry run -- to proceed, re-run with --run, or execute:\n" - f" packagemanager_install {name} {path}") + f" lfs-helper pm-install {name} {path}") return _acct = pkgusr_name(name) if not _user_exists(_acct) and (yes or ask_yes_no( @@ -2303,7 +4076,7 @@ def install_local(name, path, run, yes, verb="install", test=False, clean=False) env = {**os.environ, "PM_YES": "1" if yes else "0", "PM_MODE": "update" if verb == "update" else "all"} if test: - env["PM_TEST"] = "1" + env["PM_TEST"] = env["PKGUSR_TESTS"] = "1" old_paths = _pkglist_paths(name) if clean else None t0 = time.time() if clean: @@ -2318,7 +4091,7 @@ def install_local(name, path, run, yes, verb="install", test=False, clean=False) printSuccess(f"Done -- {name} {verb}ed. logs: {_pkg_log_dir(name)}") -def cmd_install(args): +def cmd_install(args, _replanned=False): # ---- local package: install a custom script directly, no BLFS ---- if args.local or (len(args.packages) == 1 and book_version(args.packages[0]) is None @@ -2365,8 +4138,39 @@ def cmd_install(args): script_dir = args.outdir or os.path.join(TMP_DIR, "install_files") os.makedirs(script_dir, exist_ok=True) + # A ONE-PACKAGE PLAN WITH MISSING DEPENDENCIES IS A TRAP. + # + # `packagemanager install ffmpeg --run` (no --recursive) plans exactly + # one package and says "1 to build" -- then the build dies on + # ERROR: libass >= 0.11.0 not found using pkg-config + # because libass was never installed and was never going to be. The + # plan knows; it just did not say. + if not getattr(args, "recursive", False) and len(plan) == 1: + try: + _deps, _ = blfs_order_anchors(plan[0][0], ignore=ignore) + _missing = [nv for a, nv, _k, parent in _deps + if parent and gather_user(a).state != "installed"] + except (SystemExit, Exception): + _missing = [] + if _missing: + printWarning(" %d dependenc%s of %s %s NOT installed:" + % (len(_missing), "y" if len(_missing) == 1 else "ies", + plan[0][1], "is" if len(_missing) == 1 else "are")) + printWarning(" " + ", ".join(_missing[:8]) + + (" ..." if len(_missing) > 8 else "")) + printWarning(" this plan builds ONLY %s and will fail on them." + % plan[0][1]) + printWarning(" to build them too: packagemanager install %s " + "--run --recursive" % plan[0][0]) + _done = len(getattr(build_install_plan, "already", ())) printHeader(f"Install plan ({len(plan)} to build)") + if _done: + printSuccess(" %d of the %d packages in this tree are already" + " installed -- they are skipped." + % (_done, _done + len(plan))) print() + _prewarm_scripts(plan, script_dir, + force_regen=getattr(args, "regenerate", False)) steps = [] # (anchor, nv, reason, path) for anchor, nv, reason in plan: path = resolve_script(anchor, nv, script_dir, @@ -2375,17 +4179,63 @@ def cmd_install(args): if not path: sys.exit(1) steps.append((anchor, nv, reason, path)) + + # A TOOL THE PLAN DOES NOT HAVE AT ALL: add it and plan again. + # + # 1.14.12 orders a package after the tool it calls when both are in the + # plan. When the provider is absent entirely the run just stopped: + # cargo needed by 3 package(s), first: rust-bindgen + # and rust-bindgen's page does not list Rust, so no amount of reordering + # would have helped. The provider is a book package like any other, so + # add it to what was asked for and rebuild the plan -- once, so a tool + # whose own provider is missing cannot loop. + _add = sorted({_provider_of(t) for t in _MISSING_TOOLS if _provider_of(t)} + - {a.lower() for a, _n, _r, _p in steps}) + # A provider that is ALREADY INSTALLED cannot be added -- the plan would + # come back identical and stop again, which is what happened with rust: + # installed in /opt/rustc, invisible to our PATH, so "add it" produced no + # change and the same refusal appeared twice. + _add = [a for a in _add if gather_user(a).state != "installed"] + if _add and getattr(args, "recursive", False) and not _replanned: + printInfo(" adding to the plan: %s -- %s call %s, which the book's" + " pages do not list" + % (", ".join(_add), + "their commands" if len(_add) > 1 else "its commands", + ", ".join("`%s`" % t for t in sorted(_MISSING_TOOLS)))) + args.packages = list(args.packages) + _add + _MISSING_TOOLS.clear() + _MISSING_ENV.clear() + return cmd_install(args, _replanned=True) + + # ORDER BEFORE PRINTING. The list is a promise about what happens next, + # so it has to be the order that will actually run -- printing as each + # script resolved and reordering afterwards showed libaom above CMake and + # then built it below. + steps, _moved = _order_by_build_tools(steps) + _in_plan = {a.lower() for a, _n, _r, _p in steps} + for anchor, nv, reason, _path in steps: color = {"new": colors.yellow, "update": colors.cyan, "reinstall": colors.blue}.get(reason, "") w = why_str(anchor) print(f" {nv:<30} {color}{reason:<10}{colors.normal} " f"{colors.dim}{w}{colors.normal}") + for nv, after, tool in _moved: + printInfo(" ordering: %s moved after %s (its commands call `%s`," + " which the book's page does not list)" % (nv, after, tool)) + # a tool the plan itself builds is no longer missing -- it is just later + for _t in list(_MISSING_TOOLS): + if (_BUILD_TOOLS.get(_t) or "").lower() in _in_plan: + _MISSING_TOOLS.pop(_t, None) + _report_missing_tools() + _report_ignored_required(steps) + _report_cycles(steps) + _stop_if_prereqs_missing(args) if not args.run: print() printInfo("Dry run -- to install, re-run with --run, or execute:") for anchor, nv, reason, path in steps: - print(f" packagemanager_install {anchor} {path}") + print(f" lfs-helper pm-install {anchor} {path}") return print() @@ -2397,7 +4247,7 @@ def cmd_install(args): pm = find_pm_install() env = {**os.environ, "PM_MODE": "all", "PM_YES": "1" if args.yes else "0"} if getattr(args, "test", False): - env["PM_TEST"] = "1" + env["PM_TEST"] = env["PKGUSR_TESTS"] = "1" if getattr(args, "reinstall", False): env["PM_REINSTALL"] = "1" total = len(steps) @@ -2419,25 +4269,53 @@ def cmd_install(args): clear_status() printError(f"install failed for {nv} (exit {result.returncode}); " f"stopping ({i - 1}/{total} done).") - printInfo(f" install file: {path}") printInfo(f" logs: {_pkg_log_dir(anchor)} " f"(see all: packagemanager errors)") - # Name the file to EDIT, not just the command to re-run. A - # package user's home holds install_<name>-<version> (the source) - # and install_<user> (a runtime copy overwritten every run), and - # editing the wrong one fails silently -- the install repeats - # unchanged and the edit looks lost. - printInfo(f" To change what it does, edit:\n {path}") printInfo(f" Then re-run ONLY the install step (no recompile):\n" f" packagemanager script install {anchor}") + # THE FILE TO EDIT IS THE LAST THING SAID. + # + # It is what you do next, and it was buried three messages up. A + # path you have to scroll back for is a path you retype wrong. + # + # And it must be the copy in the package user's HOME, not the one + # under /tmp/packagemanager/install_files. That is where a script + # freshly generated from the book lands; the engine copies it into + # the home, and the next run regenerates the /tmp one -- so an edit + # there is silently discarded, which is the exact failure this + # message exists to prevent. + _canon = os.path.join(pkgusr_home(anchor), + os.path.basename(path)) + _edit = _canon if os.path.isfile(_canon) else path + _runtime = os.path.join(pkgusr_home(anchor), + "install_" + pkgusr_name(anchor)) + print() + printInfo(" To change what it does, edit:") + printError(f" {_edit}") + # Since 1.14.16 the canonical script is the one that runs, so + # this only fires for a leftover copy from an older version. + if os.path.isfile(_runtime) and _runtime != _edit: + printInfo(f" ({os.path.basename(_runtime)} is a leftover copy" + f" from an older version;\n" + f" it is no longer used and can be deleted)") + if _edit != path: + printInfo(f" (not {path} --\n" + f" that one is regenerated from the book each run)") sys.exit(result.returncode) if getattr(args, "clean", False): _clean_after_install(anchor, old_paths, t0) _maybe_remove_old_sources(anchor, nv, getattr(args, "keep_sources", False)) clear_status() printSuccess(f"Done -- {total} package(s) installed.") + # THE PATTERN HAS TO BE A REAL PATH. + # + # This built the hint by hand -- os.path.join(BASE_DIR, ...) -- and got it + # wrong twice over: it printed "/usr/src/p_<pkg>/log", missing the pkgusr/ + # component that every package home actually has. Someone following it + # found nothing, twice: the directory does not exist, and until 1.14.71 + # the logs were not there either. _pkg_log_dir is the function that knows. printInfo(f" results: packagemanager errors logs: " - f"{os.path.join(BASE_DIR, pkgusr_name('<pkg>'), 'log')}") + f"{_pkg_log_dir('<name>')}") # =========================================================================== # @@ -2527,6 +4405,177 @@ def _is_git_package(name): or re.search(r'\bgit\s+(clone|pull|fetch)\b', t)) +# A plan entry whose "script" is this is built by `lfs`, not by us: the LFS +# book owns it, and lfs already knows how to build one of its packages. +_LFS_STEP = ":lfs-book:" + +_TOOLCHAIN_RISK = ("glibc", "binutils", "gcc", "libstdcpp", "ld", "linux-headers") + + +def _lfs_stale_list(include_current=False): + """[(name, installed, book)] for packages the LFS book has newer. + + With include_current, the packages whose version did NOT move are listed + too (installed == book). After glibc changes, "up to date" and "built + against the current system" stop being the same statement -- so a full + rebuild in book order is a thing a person may legitimately ask for. + + Read from `lfs update --porcelain` -- one tab-separated fact per line. + Parsing the human report instead would break the first time a word in it + changed, and this tool has been bitten by exactly that before.""" + lfs_bin = shutil.which("lfs") or ("/usr/bin/lfs" if os.access( + "/usr/bin/lfs", os.X_OK) else None) + if not lfs_bin: + return [] + try: + _cmd = [lfs_bin, "update", "--porcelain"] + if include_current: + _cmd.append("--include-current") + r = subprocess.run(_cmd, capture_output=True, text=True, timeout=600) + except Exception: + return [] + out = [] + for ln in (r.stdout or "").splitlines(): + parts = ln.split("\t") + if len(parts) == 3 and parts[0]: + out.append((parts[0], parts[1], parts[2])) + return out + + +def _lfs_update_report(names): + """Ask `lfs update` about the packages the BLFS book cannot answer for, + and relay WHAT IT SAYS. + + Returns (kind, payload): + ("current", None) -- lfs said so, in those words + ("lines", [...]) -- lfs's own report, verbatim + ("failed", why) -- no answer; the caller must say so, not assume + + The previous version returned [] both for "everything is current" and for + "the call produced nothing", and the caller printed "all current with the + LFS book" for both -- a clean bill of health manufactured out of silence, + on a system with 30 stale packages. Empty-as-pass, in the very code that + was written to stop packages disappearing quietly. + + It is called with no package names: `lfs update` already covers every + installed LFS package and labels what belongs to the other book, so this + asks the same question the user would ask by hand, and shows the same + answer. Passing 77 names asked the same tool a longer question with more + ways to go wrong.""" + lfs_bin = shutil.which("lfs") or ("/usr/bin/lfs" if os.access( + "/usr/bin/lfs", os.X_OK) else None) + if not lfs_bin: + return ("failed", "lfs is not installed here") + try: + r = subprocess.run([lfs_bin, "update"], + capture_output=True, text=True, timeout=600) + except Exception as e: + return ("failed", str(e)) + out = (r.stdout or "").strip() + if not out: + tail = (r.stderr or "").strip().splitlines() + why = tail[-1] if tail else f"no output (exit {r.returncode})" + return ("failed", why) + if "Everything checked is current" in out and "Out of date" not in out: + return ("current", None) + return ("lines", [ln for ln in out.splitlines() if ln.strip()]) + + +def _report_no_book(no_book, consult_lfs, no_record=(), unjudged=(), + uptodate_n=None, total=None, named=()): + """Account for every package the BLFS plan could not judge, and let the + LFS book answer for them. + + Listing them only when the plan was non-empty made the commonest state + (BLFS subset current, glibc stale) read as a clean bill of health -- and + two silent `continue`s meant the numbers never had to add up. They do + now: checked = planned + current + these.""" + rest = sorted(set(no_book) | set(no_record) | set(unjudged)) + if uptodate_n is not None and total is not None: + print() + printInfo(f"Of {total} checked: {uptodate_n} current with the BLFS " + f"book, {len(rest)} it cannot judge.") + if not rest: + return + if not (uptodate_n is not None and total is not None): + print() + if no_record: + printInfo(f" {len(no_record)} installed, but no recorded version " + f"(built before packagemanager, or its record was lost):") + printInfo(" " + ", ".join(sorted(no_record)[:12]) + + (" ..." if len(no_record) > 12 else "")) + if unjudged: + printInfo(f" {len(unjudged)} with no completed-install record here " + f"(the LFS build tracks these in its own manifests):") + printInfo(" " + ", ".join(sorted(unjudged)[:12]) + + (" ..." if len(unjudged) > 12 else "")) + # SAY WHAT TO DO INSTEAD. Asking to update a package this system has + # never installed is a reasonable thing to try, and "cannot judge" is + # a true answer to a question the user did not mean to ask. If they + # NAMED the packages, tell them the command that does what they want. + _named = list(named or ()) + if _named: + _want = [u for u in sorted(unjudged) + if any(u.lower() == n.lower() for n in _named)] + if _want: + printInfo(" These have no record here because they are NOT " + "INSTALLED. To build them:") + printInfo(" packagemanager install %s --run --recursive" + % " ".join(_want[:6])) + if no_book: + printInfo(f" {len(no_book)} not in the BLFS book at all.") + printInfo(" The LFS book answers for these:") + if not consult_lfs: + printInfo(" check them with: lfs update") + return + kind, payload = _lfs_update_report(rest) + if kind == "failed": + printWarning(f" could not ask lfs ({payload}) -- these are UNCHECKED.") + printInfo(" run it yourself: lfs update") + elif kind == "current": + printInfo(" all current with the LFS book.") + else: + for ln in payload: + print(" " + ln) + + +def _systemctl_users(steps): + """Planned packages whose install script runs systemctl. + + Generated scripts shim a missing systemctl into a visible skip (1.11.30), + so a systemd book builds on a SysV system -- but "skipped" means the + package's service was NOT enabled, and only the person can decide whether + that matters for this package. So: name them, once, rather than leaving + the notices to scroll past mid-build.""" + if os.path.isdir("/usr/lib/systemd/system") or shutil.which("systemctl"): + return [] # this system HAS systemd: nothing to say + out = [] + for _anchor, name, _iv, _bv, path in steps: + cands = [] + if path and path != _LFS_STEP: + cands.append(path) + # the LFS book's script for this package, if one has been generated + cands.append(os.path.join("/usr/src/lfs-pkgusr/scripts", name + ".sh")) + cands.append(os.path.join(pkgusr_home(name), "install_" + name)) + for c in cands: + try: + with open(c, encoding="utf-8", errors="replace") as f: + body = f.read() + except OSError: + continue + # the shim definition itself is not a use of systemctl + uses = [ln for ln in body.splitlines() + if "systemctl" in ln + and "command -v systemctl" not in ln + and not ln.strip().startswith("#")] + if uses: + # the PATH, not just the name: the next thing anyone does with + # this list is open the file and add the SysV equivalent + out.append((name, len(uses), c)) + break + return out + + def cmd_update(args): # --local: update a single package from a custom/local install script # (e.g. a git-based package). @@ -2574,6 +4623,8 @@ def cmd_update(args): kind_of = {} # anchor -> required/recommended/... cached = use_stored_plan("update", plan_key, assume_yes=args.yes) if args.run else None + no_book, no_record, unjudged, uptodate = [], [], [], [] + _checked = 0 if cached is not None: plan = [tuple(x) for x in cached] @@ -2605,9 +4656,17 @@ def cmd_update(args): vlookup = book_versions_all() names = list_all_users() - printInfo(f"Checking {len(names)} package(s) against the book...") + _bk = _current_blfs_book() + _checked = len(names) + printInfo(f"Checking {len(names)} package(s) against the book" + + (f" ({_bk})" if _bk else "") + "...") named = set(args.packages) if args.packages else set() - plan, not_installed, uptodate = [], [], [] + plan, not_installed = [], [] + # Two silent exits used to swallow most of a real system. Every + # package now lands in a NAMED bucket and the counts add up, because + # "97 checked, nothing to do" while `lfs update glibc` says 2.42 -> + # 2.43 is the plan lying about its own scope. + no_record, unjudged = [], [] for name in names: u = gather_user(name) anchor = u.meta.get("name") or name @@ -2615,12 +4674,28 @@ def cmd_update(args): if u.state != "installed": if name in named or anchor in named: not_installed.append(name) + else: + # NOT silent: a package built by lfs-helper has no + # install_last, so it reads as "not installed" here and + # simply disappeared from every update-all. + unjudged.append(name) continue if not bv: + # not silently: an update-all that never mentions these reads + # as "everything is covered", and the LFS-book packages are + # exactly the ones that are not + no_book.append(name) + continue + if not u.version: + # An UNKNOWN installed version used to be folded into the + # same test as the comparison, where it came out falsy -- + # and the package was filed as up to date. Not knowing is + # not the same as being current. + no_record.append(name) continue - outdated = u.version and bv != u.version + outdated = not _same_version(bv, u.version) if outdated: - plan.append((anchor, name, u.version or "?", bv)) + plan.append((anchor, name, u.version, bv)) elif getattr(args, "reinstall", False): plan.append((anchor, name, "(reinstall)", bv)) else: @@ -2649,8 +4724,17 @@ def cmd_update(args): plan.append((anchor, name, "(reinstall)", bv)) if not plan: - printSuccess("Nothing to update -- all matching installed packages are " - "current.") + _bk = _current_blfs_book() + printSuccess("Nothing to update from the BLFS book" + + (f" ({_bk})" if _bk else "") + + " -- every package it carries is current.") + # consulted on --run too: with no BLFS work to do, the run has time, + # and "check them with: lfs update" is a worse answer than the answer + _report_no_book(no_book, consult_lfs=True, + named=getattr(args, "packages", None) or [], + no_record=no_record, unjudged=unjudged, + uptodate_n=len(uptodate), + total=_checked or None) clear_plan("update", plan_key) return @@ -2719,6 +4803,107 @@ def cmd_update(args): "book (discards edits).") print(f"{bar}") + # ---- conflicts, LAST, where they are read ------------------------------ + # A version bump generates a fresh script; local edits to the installed + # version's script never carry over on their own. Collected here and said + # at the END of the plan -- one block to work through -- and --run below + # STOPS at the first one rather than silently discarding the edits. + conflicts = [] + if not getattr(args, "regenerate", False): + for anchor, name, iv, bv, path in steps: + if _pending_edit_conflict(pkgusr_home(name), iv, bv): + conflicts.append((name, iv, bv, path)) + if conflicts: + print() + printWarning(f"!! {len(conflicts)} package(s) carry LOCAL EDITS the " + f"update would leave behind:") + for name, iv, bv, path in conflicts: + home = pkgusr_home(name) + printWarning(f"!! {name}: install_{iv} differs from install_last") + printInfo(f" your edits : {os.path.join(home, 'install_' + iv)}") + printInfo(f" fresh {bv}: {path}") + printInfo(f" carry them : merge into " + f"{os.path.join(home, 'install_' + bv)} -- the update " + f"then uses that file") + printWarning("!! --run stops at the first of these. Merge the edits " + "(above), or add --regenerate to discard them all.") + # ---- the LFS book covers what BLFS does not -------------------------- + # BLFS is PREFERRED: anything it carries was planned above, from its own + # script. Everything else falls back to the LFS book, so one `update` + # covers the whole system instead of planning two packages and printing a + # command for the other twenty-nine. + _blfs_planned = {n.lower() for _a, n, _i, _b, _p in steps} + _lfs_steps = [] + if not args.packages: + for _n, _have, _want in _lfs_stale_list( + include_current=getattr(args, "reinstall", False)): + if _n.lower() in _blfs_planned: + continue # BLFS has it: preferred, done + _lfs_steps.append((_n, _n, _have, _want, _LFS_STEP)) + if _lfs_steps: + print() + printInfo("From the LFS book (BLFS does not carry these):") + for _a, _n, _iv, _bv, _p in _lfs_steps: + _risk = " (toolchain)" if _n.lower() in _TOOLCHAIN_RISK else "" + if _iv == _bv: + print(f" {_n:<22} {_iv} {colors.dim}(rebuild, same version)" + f"{colors.normal}{_risk}") + else: + print(f" {_n:<22} {_iv} -> {colors.cyan}{_bv}" + f"{colors.normal}{_risk}") + steps += _lfs_steps + total = len(steps) + + # Rebuilding the toolchain on a running system is not the same act as + # rebuilding `less`: get it wrong and there is no compiler left to fix it + # with. They are planned and shown, but running them has to be asked for. + _risky = [n for _a, n, _i, _b, p in steps + if p == _LFS_STEP and n.lower() in _TOOLCHAIN_RISK] + if _risky: + print() + if args.run and not getattr(args, "toolchain", False): + printWarning("!! toolchain packages in this plan: %s" + % ", ".join(_risky)) + printInfo(" Rebuilding those on the running system can leave it") + printInfo(" without a working compiler. SKIPPED unless asked:") + printInfo(" packagemanager update --run --toolchain") + steps = [s for s in steps if s[1] not in _risky] + total = len(steps) + elif not args.run: + printInfo("Toolchain packages above (%s) need --toolchain with " + "--run." % ", ".join(_risky)) + + # A systemd book on a SysV system: the scripts call systemctl, the shim + # turns each call into a visible skip, and the build succeeds -- but the + # service was not enabled. Worth knowing BEFORE the build, not as + # notices scrolling past during it. + _sysd = _systemctl_users(steps) + if _sysd: + print() + printWarning("%d package(s) in this plan run systemctl, which this " + "system does not have:" % len(_sysd)) + for _n, _c, _path in _sysd[:12]: + printInfo(" %-22s %d line(s) %s" % (_n, _c, _path)) + if len(_sysd) > 12: + printInfo(" ... and %d more" % (len(_sysd) - 12)) + printInfo(" Those lines are SKIPPED (and say so), so the build") + printInfo(" succeeds -- but the service is not enabled. Edit the") + printInfo(" file above to add the SysV equivalent; it is the file") + printInfo(" the build runs, and an edited script is kept.") + + # The report below covers what the plan does NOT: listing a package as + # "the LFS book answers for this" directly above the same package in the + # plan printed every stale package twice. + _planned_now = {n.lower() for _a, n, _i, _b, _p in steps} + _report_no_book([n for n in no_book if n.lower() not in _planned_now], + named=getattr(args, "packages", None) or [], + consult_lfs=False, + no_record=[n for n in no_record + if n.lower() not in _planned_now], + unjudged=[n for n in unjudged + if n.lower() not in _planned_now]) + _conflict_names = {c[0] for c in conflicts} + if not args.run: print() if args.select: @@ -2755,7 +4940,7 @@ def cmd_update(args): env = {**os.environ, "PM_MODE": "install" if install_only else "update", "PM_YES": "1" if args.yes else "0"} if getattr(args, "test", False): - env["PM_TEST"] = "1" + env["PM_TEST"] = env["PKGUSR_TESTS"] = "1" if getattr(args, "reinstall", False): env["PM_REINSTALL"] = "1" remaining = list(plan) @@ -2763,10 +4948,50 @@ def cmd_update(args): errors = 0 run_log = start_run_log("update", plan_key) for i, (anchor, name, iv, bv, path) in enumerate(steps, 1): + if name in _conflict_names: + save_plan("update", plan_key, remaining) # resume-able + clear_status() + home = pkgusr_home(name) + printError(f"stopping BEFORE {name}: install_{iv} carries local " + f"edits the fresh {bv} script would leave behind " + f"({i - 1}/{total} done).") + printInfo(f" your edits : {os.path.join(home, 'install_' + iv)}") + printInfo(f" fresh {bv}: {path}") + printInfo(f" Merge them into {os.path.join(home, 'install_' + bv)}" + f" -- the update then uses that file -- and re-run to " + f"resume; or re-run with --regenerate to discard the " + f"edits.") + return set_status(f"[{i}/{total}] updating {name} {iv} -> {bv}" + (f" ({errors} error(s) so far)" if errors else "")) verb = "Installing" if install_only else "Updating" printHeader(f"({i}/{total}) {verb} {name}: {iv} -> {bv}") + if path == _LFS_STEP: + # The LFS book owns this package, and `lfs` already knows how to + # build one of its packages -- fetching the tarball, generating + # the script, handing over to lfs-helper. Reimplementing that + # here would be a second builder for the same book, which is the + # whole disease this project keeps treating. + _lfs_bin = shutil.which("lfs") or "/usr/bin/lfs" + _cmd = [_lfs_bin, "install", "--reinstall", name] + if args.yes: + _cmd.append("--yes") + printInfo(" via the LFS book: %s" % " ".join(_cmd)) + try: + _rc = subprocess.run(_cmd).returncode + except Exception as e: + printError(f" could not run lfs: {e}") + _rc = 1 + if _rc != 0: + errors += 1 + printError(f"{name}: the LFS build failed (exit {_rc}); " + f"stopping ({i - 1}/{total} done).") + save_plan("update", plan_key, remaining) + clear_status() + return + remaining = [p for p in remaining if p[0] != anchor] + save_plan("update", plan_key, remaining) + continue cleaning = getattr(args, "clean", False) old_paths = _pkglist_paths(name) if cleaning else None t0 = time.time() @@ -2780,7 +5005,6 @@ def cmd_update(args): clear_status() printError(f"update failed for {name} (exit {result.returncode}); " f"stopping ({i - 1}/{total} done, {errors} error(s)).") - printInfo(f" install file: {path}") printInfo(f" logs: {_pkg_log_dir(name)} " f"(see all: packagemanager errors)") base = ("packagemanager update " + " ".join(list(args.packages))) @@ -2792,6 +5016,12 @@ def cmd_update(args): f"{name})\n" f" - to REBUILD from scratch: {base} --run " f"(add --regenerate to rebuild the script from the book)") + # Last, and the copy that survives -- see the install path above. + _canon = os.path.join(pkgusr_home(name), os.path.basename(path)) + _edit = _canon if os.path.isfile(_canon) else path + print() + printInfo(" To change what it does, edit:") + printError(f" {_edit}") sys.exit(result.returncode) if getattr(args, "clean", False): _clean_after_install(name, old_paths, t0) @@ -2802,8 +5032,15 @@ def cmd_update(args): clear_plan("update", plan_key) printSuccess(f"Done -- {total} package(s) updated" + (f", {errors} error(s)." if errors else ".")) + # THE PATTERN HAS TO BE A REAL PATH. + # + # This built the hint by hand -- os.path.join(BASE_DIR, ...) -- and got it + # wrong twice over: it printed "/usr/src/p_<pkg>/log", missing the pkgusr/ + # component that every package home actually has. Someone following it + # found nothing, twice: the directory does not exist, and until 1.14.71 + # the logs were not there either. _pkg_log_dir is the function that knows. printInfo(f" results: packagemanager errors logs: " - f"{os.path.join(BASE_DIR, pkgusr_name('<pkg>'), 'log')}") + f"{_pkg_log_dir('<name>')}") # =========================================================================== # # command: remove [IMPLEMENTED] # =========================================================================== # @@ -2949,6 +5186,34 @@ def clear_status(): sys.stdout.flush() +def _versioned_script_in(home): + """The editable install_<name>-<version> in a package user's home. + + That is the file a person is told to edit, and the one the copy that + actually runs is made from -- so a failure has to be able to name it. + """ + try: + for f in sorted(os.listdir(home)): + if not f.startswith("install_") or f.endswith((".edited", ".bak")): + continue + # install_wget-1.25.0 -- a version after the last dash + if re.search(r"-\d[\w.]*$", f): + return os.path.join(home, f) + except OSError: + pass + return None + + +def _unpacked_source_dir(home): + """Where the package unpacked, when there is exactly one candidate.""" + src = os.path.join(home, "src") + try: + dirs = [d for d in os.listdir(src) if os.path.isdir(os.path.join(src, d))] + except OSError: + return None + return os.path.join(src, dirs[0]) if len(dirs) == 1 else None + + def _pkg_log_dir(user): return os.path.join(pkgusr_home(user), "log") @@ -3185,7 +5450,14 @@ def remove_one(name, args): # safety: for --purge, never remove a user whose home isn't under /usr/src if args.purge and u.has_user: - pw_dir = pwd.getpwnam(name).pw_dir if pwd else "" + # `u.has_user` was computed for the ACCOUNT; this looked the ANCHOR up + # in passwd, so `packagemanager remove cargo-c --purge` died with + # KeyError: getpwnam(): name not found: 'cargo-c' + # A missing account is a normal answer here, not an exception. + try: + pw_dir = pwd.getpwnam(pkgusr_name(name)).pw_dir if pwd else "" + except KeyError: + pw_dir = "" if not pw_dir.startswith(BASE_DIR.rstrip("/") + "/"): printError(f"{name}: user home {pw_dir} is not under {BASE_DIR}; " f"refusing to purge (not a package-user).") @@ -3245,12 +5517,23 @@ def remove_one(name, args): printInfo(f" uninstalled {total} files owned by {name}") if args.purge: + # THE ACCOUNT, NOT THE ANCHOR -- AND CHECK IT WORKED. + # + # `remove --purge mako` ran `userdel mako`. There is no such user; + # the account is p_mako, which stayed in /etc/passwd while the tool + # printed "mako purged." The files and the home were gone, so the + # system was left with an account owning nothing and a report saying + # otherwise. + _acct = pkgusr_name(name) if u.has_user: - _run(["userdel", name]) + if not _run(["userdel", _acct]): # _run returns True on success + printError(f" userdel {_acct} did not succeed -- the account " + f"is still there") + printInfo(f" check with: grep {_acct} /etc/passwd") if grp is not None: try: - grp.getgrnam(name) - _run(["groupdel", name]) + grp.getgrnam(_acct) + _run(["groupdel", _acct]) except KeyError: pass if has_home and os.path.isdir(home): @@ -3258,7 +5541,20 @@ def remove_one(name, args): shutil.rmtree(home) except OSError as e: printError(f" rmtree {home}: {e}") - printSuccess(f"{name} purged.") + _left = [] + if pwd is not None: + try: + pwd.getpwnam(_acct) + _left.append("the account " + _acct) + except KeyError: + pass + if os.path.isdir(home): + _left.append(home) + if _left: + printWarning(f"{name}: files removed, but {', '.join(_left)} " + f"remain -- NOT fully purged") + else: + printSuccess(f"{name} purged.") else: for fname in ("pkg.lst", "pkg.lst.new"): # manifest now meaningless try: @@ -3297,10 +5593,31 @@ def _can_su(user): return False +def _python_module_here(name): + """Is a module of this name importable by the system python?""" + mod = name.replace("-", "_") + try: + return subprocess.run( + ["python3", "-c", + "import importlib.metadata as m,sys;m.version(sys.argv[1])", mod], + capture_output=True, timeout=20).returncode == 0 + except (OSError, subprocess.SubprocessError): + return False + + def write_pkg_list(user, background=True): """Regenerate /usr/src/<user>/pkg.lst. Runs list_package AS the package user (su -) when possible, so it only scans directories that user owns -- much faster than scanning the whole filesystem as root.""" + # THE ACCOUNT NAME, NOT THE PACKAGE NAME. + # + # `packagemanager reload-pkg-list mpv` passed "mpv" straight through. + # pkgusr_home() accepts either, so the home resolved; _can_su("mpv") did + # not, because the account is p_mpv -- so it took the root branch and ran + # `list_package mpv`, which found no such user, produced nothing, and left + # an empty root-owned pkg.lst.new behind. And it printed "regeneration + # started". + user = pkgusr_name(user) home = pkgusr_home(user) if not os.path.isdir(home): printError(f"{user}: no package home at {home}") @@ -3308,14 +5625,50 @@ def write_pkg_list(user, background=True): lp = " ".join(shlex.quote(x) for x in find_list_package()) quser = shlex.quote(user) + # THE HOME IS NOT INSTALLED FILES. + # + # list_package lists everything the account owns, and that includes its + # own home -- sources, build trees, the install script. For the kernel + # that is the whole tree: + # # recorded 266621 file(s) for p_linux + # A manifest is what the package put ON THE SYSTEM, which is what + # `verify` checks and what `remove` deletes; 266k source files in it + # make both meaningless. The build's own tracking has always excluded + # the home (it compares snapshots of the tree outside it); this is the + # same rule for the rescan. awk, not grep, because a filter that + # removes every line must still exit 0. + # ...AND NOT THROUGH A SYMLINK INTO IT. + # + # Excluding the home was not enough: modules_install leaves + # /lib/modules/<ver>/build as a SYMLINK to the source tree, the scan + # follows it, and the same quarter-million files come back recorded as + # /lib/modules/<ver>/build/... -- outside the home, so the home filter + # never saw them. The symlink itself belongs in the manifest; what is + # on the far side of it does not. + # + # The prefix match (index()==1) was too strict: it assumed every line is + # an absolute path starting exactly at the home, and the manifest stayed + # at 268k entries with the filter in place. Matching the home ANYWHERE + # in the line cannot be fooled by a leading ./ or a relative listing, + # and no installed file has a package home in its path. + _filter = ("awk -v h=%s -v m=/var/mail/ -v s=/var/spool/mail/ " + "'index($0,h) { next } " + " index($0,m)==1 { next } " + " index($0,s)==1 { next } " + " /\\/lib\\/modules\\/[^/]+\\/build\\// { next } " + " /\\/lib\\/modules\\/[^/]+\\/source\\// { next } " + " { print }'" + % shlex.quote(home.rstrip("/") + "/")) if _can_su(user): # su - starts in the user's home, so pkg.lst.new/pkg.lst are ~/... - inner = f"{lp} {quser} > pkg.lst.new 2>/dev/null && mv pkg.lst.new pkg.lst" + inner = (f"{lp} {quser} 2>/dev/null | {_filter} > pkg.lst.new" + f" && mv pkg.lst.new pkg.lst") cmd = ["su", "-", user, "-c", inner] else: tmp = shlex.quote(os.path.join(home, "pkg.lst.new")) dst = shlex.quote(os.path.join(home, "pkg.lst")) - cmd = ["sh", "-c", f"{lp} {quser} > {tmp} 2>/dev/null && mv {tmp} {dst}"] + cmd = ["sh", "-c", f"{lp} {quser} 2>/dev/null | {_filter} > {tmp}" + f" && mv {tmp} {dst}"] if background: # detached / disowned subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, @@ -3325,11 +5678,39 @@ def write_pkg_list(user, background=True): def cmd_reload_pkg_list(args): + # SAY WHAT HAPPENED, NOT THAT IT STARTED. + # + # A disowned task that dies in its first second reported "regeneration + # started" and left the user watching an empty pkg.lst.new and an empty + # process table. This command is asked for by a person, so it runs in the + # foreground by default and reports the RESULT; --background is there for + # anyone who wants the old behaviour. + rc = 0 for user in args.packages: - ok = write_pkg_list(user, background=not args.foreground) - if ok: - where = "foreground" if args.foreground else "disowned background task" - printInfo(f"{user}: pkg.lst regeneration started ({where}).") + acct = pkgusr_name(user) + if getattr(args, "background", False): + if write_pkg_list(user, background=True): + printInfo(f"{acct}: pkg.lst regeneration started" + f" (disowned background task).") + continue + printInfo(f"{acct}: scanning for files it owns ...") + if not write_pkg_list(user, background=False): + printError(f"{acct}: could not regenerate pkg.lst") + rc = 1 + continue + manifest = os.path.join(pkgusr_home(acct), "pkg.lst") + try: + with open(manifest) as f: + n = sum(1 for line in f if line.strip()) + except OSError: + n = 0 + if n: + printSuccess(f"{acct}: {n} file(s) recorded in {manifest}") + else: + printWarning(f"{acct}: no files found -- it owns nothing on disk.") + printInfo(" Either it was never installed, or its files ended up" + " owned by someone else.") + return rc # =========================================================================== # @@ -3339,11 +5720,21 @@ def cmd_reload_pkg_list(args): def report_created_user(name, kind="package-user"): """After creating a user, tell the person exactly what now exists / changed on the system (uid/gid, home, and the databases that were written).""" + # THE ACCOUNT THAT WAS CREATED, NOT THE NAME IT WAS ASKED FOR. + # + # `add-user n76310` created p_n76310 and then reported on "n76310" -- + # the HUMAN login of the same name: its uid, its /home/n76310, its + # audio/video/wheel groups. Every line was true of the wrong account, + # which is the most misleading kind of report there is. try: pw = pwd.getpwnam(name) except (KeyError, TypeError): - printWarning(f" {name}: created, but not found in the user database?") - return + try: + name = pkgusr_name(name) + pw = pwd.getpwnam(name) + except (KeyError, TypeError): + printWarning(f" {name}: created, but not found in the user database?") + return printInfo(f" {kind} '{name}': uid {pw.pw_uid}, gid {pw.pw_gid}") printInfo(f" recorded in: /etc/passwd, /etc/group, /etc/shadow") home = pw.pw_dir @@ -3432,7 +5823,7 @@ def pip_install_module(mod, site, grp, find_links=None, upgrade=False, else: if not _group_exists(grp): printInfo(f"creating collector group {grp}") - _run([_cmd_path("groupadd"), grp]) + _create_collector_group(grp) _run([_cmd_path("usermod"), "-a", "-G", grp, user]) if _group_exists(grp): _run(["chgrp", "-R", grp, site]) @@ -3456,12 +5847,93 @@ def pip_install_module(mod, site, grp, find_links=None, upgrade=False, " ".join(shlex.quote(x) for x in env + pip_args)] printInfo(" ".join(pip_args) + f" (as {user})") if _run(cmd, show_output=True): + # RECORD IT LIKE EVERY OTHER INSTALL. + # + # This installed the module and wrote nothing: no pkg.lst, no + # install_last. So beautifulsoup4, certifi, idna, requests, urllib3 + # and the rest of the bootstrap set -- installed, importable, in daily + # use by these very tools -- read as + # not installed user + home exist, but nothing installed yet + # and were offered for rebuild by `verify`, with a command that could + # not work because they are not in the book either. An install that + # leaves no record is invisible to everything downstream of it. + _record_pip_install(user, mod, target) printSuccess(f"{mod} installed as package-user {user}") return True printError(f"{mod} failed -- see the output above") return False +def _record_pip_install(user, mod, target=None): + """The manifest and the version stamp a pip install owes the system.""" + home = pkgusr_home(user) + if not os.path.isdir(home): + return + ok = write_pkg_list(user, background=False) + # the version pip actually landed, asked of the module itself + ver = "" + try: + out = subprocess.run( + ["python3", "-c", + "import importlib.metadata as m,sys;print(m.version(sys.argv[1]))", + mod], capture_output=True, text=True, timeout=30) + if out.returncode == 0: + ver = out.stdout.strip() + except (OSError, subprocess.SubprocessError): + pass + # install_last is what the tools read for name_version; a pip module has no + # script, so write the minimum that makes it a package like the others + try: + with open(os.path.join(home, "install_last"), "w") as f: + f.write("### installed by `packagemanager pip` -- no book page\n") + f.write("name=%s\n" % shlex.quote(mod)) + f.write("name_version=%s\n" % shlex.quote( + "%s-%s" % (mod, ver) if ver else mod)) + f.write("validate_cmd=%s\n" % shlex.quote( + "python3 -c 'import importlib.metadata as m;" + "print(m.version(\"%s\"))'" % mod)) + os.chmod(os.path.join(home, "install_last"), 0o444) + except OSError: + pass + if not ok: + printWarning(" (could not record %s's file list; `packagemanager " + "verify` may report it as not installed)" % user) + + +def cmd_pip_record(names): + """Give an already-installed pip module the record it never got. + + Installs before 1.14.37 wrote neither pkg.lst nor install_last, so the + modules these tools themselves depend on -- beautifulsoup4, certifi, idna, + requests, urllib3 -- read as "not installed" forever. Reinstalling them + would work, and is the wrong answer: they ARE installed, and nothing but + the paperwork is missing. With no argument, every account in that state. + """ + if not names: + names = [n for n in list_all_users() + if not gather_user(n).meta.get("name_version") + and _python_module_here(n)] + if not names: + printSuccess("Nothing to record -- every python module has one.") + return 0 + printInfo("Recording: " + " ".join(names)) + rc = 0 + for name in names: + user = pkgusr_name(name) + if not os.path.isdir(pkgusr_home(user)): + printError(f"{name}: no package home -- nothing to record.") + rc = 1 + continue + if not _python_module_here(name): + printError(f"{name}: python cannot import it, so it is not " + f"installed -- record refused.") + rc = 1 + continue + _record_pip_install(user, name.replace("-", "_")) + printSuccess(f"{name}: recorded.") + return rc + + def cmd_pip(args): """Install or update a Python module AS A PACKAGE USER. @@ -3491,20 +5963,154 @@ def cmd_pip(args): def cmd_add_user(args): for name in args.packages: - if _user_exists(name): - printInfo(f"{name}: already exists.") + # THE ACCOUNT, NOT THE PACKAGE NAME. + # + # `add-user n76310` asked whether "n76310" exists. It did -- it is + # the human's login account -- so this said "already exists" and + # created nothing, and the account p_n76310 was then made on the fly + # by the install runner, with the hint's default home + # (/usr/src/p_n76310, no pkgusr/ component). verify scans + # /usr/src/pkgusr and never saw it. Same fault as 1.14.85: a package + # name is not an account name. + acct = pkgusr_name(name) + if _user_exists(acct): + printInfo(f"{acct}: already exists.") continue if create_package_user(name): - printSuccess(f"Created package-user {name}.") - report_created_user(name) + printSuccess(f"Created package-user {acct}.") + report_created_user(acct) # =========================================================================== # # first-run setup + config command # =========================================================================== # +def _adopt_existing_settings(): + """What can be answered without asking anybody. + + Three sources, in order of authority: + + 1. the lfs config -- the interview's own answers, which is what the + system was actually built with; + 2. the environment `lfs` exports into the chroot; + 3. the system itself -- the collector groups and accounts that are + already there. + + Returns {key: (value, where it came from)} for everything that is known. + """ + found = {} + shared = _shared_lfs_config() + for key in _SHARED_WITH_LFS + ("user_prefix",): + val = shared.get(key) + if val: + found[key] = (str(val).rstrip("_"), "the lfs config") + + for key, env in (("collector_prefix", "LFS_COLLECTOR_PREFIX"), + ("pkgusr_prefix", "LFS_PKGUSR_PREFIX"), + ("cfguser_prefix", "LFS_CFGUSR_PREFIX"), + ("main_user", "LFS_MAIN_USER")): + if key in found: + continue + val = os.environ.get(env) + if val: + found[key] = (val.rstrip("_"), "the chroot environment ($%s)" % env) + + # THE SYSTEM ITSELF. Collector groups live at gid 90000+, and every one of + # them carries the prefix -- so on a built system the answer is written in + # /etc/group whether or not any config survived. + if "collector_prefix" not in found: + pfx = _prefix_from_groups() + if pfx: + found["collector_prefix"] = (pfx, "the collector groups on this system") + if "pkgusr_prefix" not in found: + pfx = _prefix_from_accounts(("p", "pkg")) + if pfx: + found["pkgusr_prefix"] = (pfx, "the package users on this system") + if "main_user" not in found: + mu = _guess_main_user() + if mu: + found["main_user"] = (mu, "the login account on this system") + return found + + +def _prefix_from_groups(): + """The prefix the collector groups already carry, if they agree on one.""" + if grp is None: + return "" + seen = {} + try: + groups = grp.getgrall() + except Exception: + return "" + for g in groups: + if not (90000 <= g.gr_gid < 99000) or "_" not in g.gr_name: + continue + seen[g.gr_name.split("_", 1)[0]] = seen.get(g.gr_name.split("_", 1)[0], 0) + 1 + if not seen: + return "" + # One prefix, or the commonest: a tree that changed prefix mid-life has + # both, and the one on most groups is the one in use. + return max(seen.items(), key=lambda kv: kv[1])[0] + + +def _prefix_from_accounts(candidates): + """Which of these prefixes the package users actually wear.""" + if pwd is None: + return "" + counts = {} + try: + users = pwd.getpwall() + except Exception: + return "" + for u in users: + if not (10000 <= u.pw_uid < 65000) or "_" not in u.pw_name: + continue + pfx = u.pw_name.split("_", 1)[0] + if pfx in candidates: + counts[pfx] = counts.get(pfx, 0) + 1 + if not counts: + return "" + return max(counts.items(), key=lambda kv: kv[1])[0] + + def first_run_setup(): + """Write packagemanager's config, asking only for what nothing knows. + + This asked for all three settings every time the config file was absent -- + on a system built by these tools, where `lfs build-system session` had + already collected every one of them, written them into config.json, and + copied that into the chroot. Being asked for a value the machine is + holding teaches you that the answer does not matter, which is the opposite + of true: a collector prefix that disagrees with the built system gives you + two parallel sets of groups over the same directories, with no error. + """ cfg = load_config() + known = _adopt_existing_settings() + + for key, (val, _src) in known.items(): + cfg[key] = val + + missing = [k for k in ("collector_prefix", "user_prefix") + if not cfg.get(k)] + + if not missing: + printHeader("Settings") + print() + printInfo("Taken from what this system was built with -- nothing to answer:") + for key in ("collector_prefix", "pkgusr_prefix", "cfguser_prefix", + "user_prefix", "main_user"): + if not cfg.get(key): + continue + src = known.get(key, (None, "the default"))[1] + print(" %-18s %-12s (%s)" % (key, cfg[key], src)) + if save_config(cfg): + printSuccess("Saved %s -- change any of it with: packagemanager config" + % CONFIG_PATH) + else: + printWarning("Could not save config; using these values for this run only.") + print() + return + printHeader("First-run setup") print() printInfo("packagemanager needs a few settings (saved so you're only asked once):") @@ -3559,13 +6165,74 @@ def cmd_script(args): """Run one phase of a package's install script AS the package user, e.g. 'packagemanager script nano install' to re-run only the install step after fixing permissions (no recompile).""" - name = args.package + # THE ACCOUNT, NOT THE ANCHOR. This took args.package verbatim, so + # packagemanager script install rust + # ran `su - rust -c 'bash ~/install_rust install'` -- an account that + # does not exist, and a script named after the anchor rather than the + # package user. Everything else in the tool goes through pkgusr_name. + # TWO NAMES, AND THEY ARE NOT INTERCHANGEABLE. + # anchor -- what the BOOK calls it (rust); blfs generates from this + # name -- the ACCOUNT (p_rust); su and the script path use this + # It used the anchor for both (su - rust: no such account), and the first + # fix used the account for both (blfs script failed for p_rust). + anchor = args.package + name = pkgusr_name(anchor) phase = args.phase + + # fetch: get the script, run NOTHING. It lands where edits live -- the + # package user's home when there is one, /tmp's script dir otherwise -- + # which is exactly the file resolve_script prefers, so "fetch, edit, + # install" and the 1.11.31 conflict flow ("merge into install_<version>") + # are the same motion. + if phase == "fetch": + bv = book_version(name) + if not bv: + printError(f"{name} is not in the BLFS book.") + printInfo(f" LFS-book packages have their script fetched by: " + f"lfs script fetch {name}") + sys.exit(1) + home = pkgusr_home(name) + target = os.path.join(home, f"install_{bv}") + if os.path.isfile(target) and not args.regenerate: + printSuccess(f"already there: {target}") + printInfo(" edit that file -- install and update use it. " + "Rebuild it from the book with --regenerate.") + return + outdir = home if os.path.isdir(home) \ + else os.path.join(TMP_DIR, "install_files") + os.makedirs(outdir, exist_ok=True) + path = resolve_script(name, bv, outdir, assume_yes=True, + force_regen=args.regenerate) + if not path: + sys.exit(1) + if os.path.isdir(home) and os.path.dirname( + os.path.abspath(path)) != os.path.abspath(home): + shutil.copy(path, target) + path = target + # the canonical script belongs to the package user, like every other + # file in its home; a not-yet-created account simply keeps root's copy + try: + import pwd + pw = pwd.getpwnam(pkgusr_name(name)) + os.chown(path, pw.pw_uid, pw.pw_gid) + except (KeyError, OSError): + pass + printSuccess(f"script: {path}") + printInfo(f" edit it, then: packagemanager install {name} --run") + return + home = pkgusr_home(name) if not _user_exists(name): printError(f"{name}: no such package user.") return - home_script = os.path.join(home, f"install_{name}") + # THE CANONICAL FILE, NOT A COPY OF IT. + # + # This used install_<user> -- the redundant runtime copy 1.14.16 removed + # everywhere else -- so a failure reported "ran: install_p_glib2" while + # telling the person to edit install_GLib-2.86.4, and an edit to the file + # they were pointed at did not affect the file that ran. + home_script = (_versioned_script_in(home) + or os.path.join(home, f"install_{name}")) # (re)generate if asked, missing, or the on-disk script predates the current # phased format (old scripts can't find the build dir for install). @@ -3583,39 +6250,154 @@ def cmd_script(args): printInfo(f"Generating install script for {name}...") script_dir = os.path.join(TMP_DIR, "install_files") os.makedirs(script_dir, exist_ok=True) - rc, out, err = run_blfs(["script", name, "-o", script_dir]) + rc, out, err = run_blfs(["script", anchor, "-o", script_dir]) m = re.search(r"^Created (.+)$", out, re.M) if rc != 0 or not m: printError(f"blfs script failed for {name} -- {(err or out).strip()}") return gen = m.group(1).strip() + # NEVER DESTROY AN EDIT IN SILENCE. + # + # This overwrites the script in the package's home whenever the tools + # move on -- and a user who had fixed something in it (a dead download + # URL, say) loses the fix with no message. Keep a copy and say so. + try: + _dest = os.path.join(home, os.path.basename(gen)) + if (os.path.isfile(_dest) + and open(_dest).read() != open(gen).read()): + home_script = _dest + bak = home_script + ".edited" + shutil.copy(home_script, bak) + printWarning(" your version of %s differed -- saved as %s" + % (os.path.basename(home_script), + os.path.basename(bak))) + # Say what actually happens. This claimed the versioned file + # "is never regenerated" while regenerating it, which is how a + # person loses an edit twice: once to the overwrite, once to + # the sentence that said it could not happen. + printInfo(" (this file IS the one you edit; it was rewritten" + " because you asked for") + printInfo(" --regenerate or its format was too old." + " Your copy is the .edited file.)") + except OSError: + pass + # a regenerated script keeps the book's own name for the file + home_script = os.path.join(home, os.path.basename(gen)) try: shutil.copy(gen, home_script) - if pwd is not None: - pw = pwd.getpwnam(name) - os.chown(home_script, pw.pw_uid, pw.pw_gid) + os.chmod(home_script, 0o755) + # `pwd` is imported inside a try in ANOTHER branch, so this + # referenced a name that may never have been bound: + # UnboundLocalError: cannot access local variable 'pwd' + # Import it here, where it is used, and treat a missing account + # as "leave it owned by root" rather than an error. + try: + import pwd as _pw_mod + _pw = _pw_mod.getpwnam(name) + os.chown(home_script, _pw.pw_uid, _pw.pw_gid) + except (ImportError, KeyError, OSError): + pass except OSError as e: printError(f"could not place script in {home}: {e}") return printHeader(f"{name}: {phase}") - inner = f"cd ~ && bash {shlex.quote(f'install_{name}')} {shlex.quote(phase)}" - cmd = ["su", "-", name, "-c", inner] if _can_su(name) else ["sh", "-c", inner] - printInfo(f" $ su - {name} -c 'bash ~/install_{name} {phase}'") - env = {**os.environ} + # THE BUILD ENVIRONMENT HAS TO REACH THE BUILD. + # + # `packagemanager env --set XORG_CONFIG=...` writes /etc/pkgusr/build.env, + # and lfs-helper's run_phase_as pastes it in front of the command it runs. + # This path built its own `su` and passed nothing, so the same package + # failed here with + # this package needs XORG_CONFIG to be set + # right after the person had set it and been shown it in the table. + # `su -` starts a login shell and drops the caller's environment, so the + # variables have to travel INSIDE the command, not in subprocess's env. + _bootenv = dict(_read_build_env()) if getattr(args, "test", False): - env["PM_TEST"] = "1" + _bootenv["PM_TEST"] = _bootenv["PKGUSR_TESTS"] = "1" + _pairs = "".join("%s=%s " % (k, shlex.quote(v)) + for k, v in sorted(_bootenv.items())) + # THE SAME DOOR AS EVERY OTHER BUILD. + # + # This ran the script itself, and so had none of what lfs-helper's + # cmd_build wraps around one: the compiler wrappers, the file tracking, + # the failure box -- and above all the GRANT-AND-RETRY. A glib2 install + # that stopped on + # PermissionError: [Errno 13] '/usr/share/gettext/its' + # was repaired automatically when the same script ran through lfs-helper, + # and repeated the identical traceback forever when it ran through here. + # `packagemanager script install X` is the command the failure box itself + # tells people to use, so it must be the same door. + _helper = shutil.which("lfs-helper") + if _helper and _can_su(name) and phase != "fetch": + cmd = [_helper, "build", name, "--script", home_script, + "--phase", phase, "--force"] + printInfo(" $ " + " ".join(cmd[1:])) + else: + _run = f"bash {shlex.quote(f'install_{name}')} {shlex.quote(phase)}" + inner = f"cd ~ && {('env ' + _pairs) if _pairs else ''}{_run}" + cmd = (["su", "-", name, "-c", inner] if _can_su(name) + else ["sh", "-c", inner]) + printInfo(f" $ su - {name} -c 'bash ~/install_{name} {phase}'") + if _bootenv: + printInfo(" with: " + " ".join(sorted(_bootenv)) + " (from %s)" + % BUILD_ENV_FILE) + _records = phase in ("install", "all", "update") + env = {**os.environ} old_paths = (_pkglist_paths(name) if getattr(args, "clean", False) and phase in ("all", "install", "update") else None) t0 = time.time() result = subprocess.run(cmd, env=env) if result.returncode != 0: + # NAME THE FILES. This printed the exit code and a log directory and + # left the person to work out which of the three copies of the script + # had just run, where it built, and what to do next -- while the same + # failure through lfs-helper prints all of it. A failure report that + # does not say where is barely a report. printError(f"phase '{phase}' failed for {name} (exit {result.returncode}).") - printInfo(f" logs: {_pkg_log_dir(name)}") + printInfo(f" ran : {home_script}") + _edit = _versioned_script_in(home) + if _edit: + printInfo(f" edit : {_edit}") + printInfo(" (that one is yours and is never regenerated;" + " the file above is a copy of it)") + _src = _unpacked_source_dir(home) + if _src: + printInfo(f" source : {_src}") + printInfo(f" logs : {_pkg_log_dir(name)}") + printInfo(f" retry : packagemanager script {phase} " + f"{getattr(args, 'package', name)}") + if phase in ("install", "configure", "test"): + # "No rule to make target 'install'" means the source is there and + # unconfigured -- the build phase never finished, and running one + # later phase cannot make up for it. + printInfo(" If the build phase never finished here, no later" + " phase can work.") + printInfo(f" packagemanager script build " + f"{getattr(args, 'package', name)}") sys.exit(result.returncode) if old_paths is not None: _clean_after_install(name, old_paths, t0) + # A PHASE THAT INSTALLED MUST BE RECORDED. + # + # This runs the script directly, and only pm-install writes the manifest + # -- so after `packagemanager script install rust` succeeded, the planner + # still called Rustc "new" and the next stack run started building it + # again from scratch. An install nobody recorded did not happen, as far + # as every other command is concerned. + if _records: + try: + _rr = subprocess.run(["lfs-helper", "record-install", name], + capture_output=True, text=True) + for _l in (_rr.stdout or "").splitlines(): + if _l.strip(): + printInfo(" " + _l.strip()) + if _rr.returncode != 0: + printWarning(" could not record the install -- the planner will" + " treat %s as not installed" % name) + except (OSError, subprocess.SubprocessError) as e: + printWarning(f" could not record the install ({e})") printSuccess(f"{name}: {phase} done. logs: {_pkg_log_dir(name)}") @@ -3698,9 +6480,199 @@ def _book_in_use(tool): return f"{ver} (not cached here)" +def _accounts_with_prefix(prefix): + """Package-user accounts currently using this prefix. + + Used to refuse a prefix change that would orphan a built tree: the + accounts are not renamed by changing the setting. + """ + if pwd is None: + return [] + pfx = (prefix.rstrip("_") + "_") if prefix else "" + out = [] + for entry in pwd.getpwall(): + if not pfx or entry.pw_name.startswith(pfx): + if entry.pw_dir.startswith(BASE_DIR.rstrip("/") + "/"): + out.append(entry.pw_name) + return out + + +def _config_sources(): + """Every prefix and directory, with the reason it has that value. + + The user could not find where these live -- fairly: prefixes come from + /usr/share/lfs/config.json (authoritative), then + /etc/pkgusr/packagemanager.conf, then the environment, then a default; + and the DIRECTORIES were environment-only, documented nowhere. + """ + shared = _shared_lfs_config() or {} + conf = {} + try: + import configparser + cp = configparser.ConfigParser() + with open(CONFIG_PATH) as f: + cp.read_string("[c]\n" + f.read()) + conf = dict(cp["c"]) + except Exception: + conf = {} + + def src(key, env, value): + if os.environ.get(env): + return "the environment ($%s)" % env + if key in shared: + return "the lfs config (%s)" % _lfs_config_path() + if key in conf: + return "packagemanager.conf" + return "the built-in default" + + rows = [ + ("package-user prefix", pkgusr_prefix(), + src("pkgusr_prefix", "LFS_PKGUSR_PREFIX", None)), + ("config-user prefix", cfguser_prefix(), + src("cfguser_prefix", "LFS_CFGUSR_PREFIX", None)), + ("collector prefix", collector_prefix(), + src("collector_prefix", "LFS_COLLECTOR_PREFIX", None)), + ("shared-user prefix", user_prefix(), + src("user_prefix", "LFS_USER_PREFIX", None)), + ("base directory", BASE_DIR, + "the environment ($PKGUSR_BASE)" if os.environ.get("PKGUSR_BASE") + else "the built-in default"), + ("package homes", os.path.join(BASE_DIR, PKGUSR_SUBDIR) if PKGUSR_SUBDIR + else BASE_DIR, + src("pkgusr_subdir", "PKGUSR_SUBDIR", None)), + ("config-user homes", os.path.join(BASE_DIR, CFGUSR_SUBDIR) if CFGUSR_SUBDIR + else BASE_DIR, + src("cfguser_subdir", "CFGUSR_SUBDIR", None)), + ] + return rows + + +def _lfs_config_path(): + """The lfs config actually in use -- not the default path. + + The report named /usr/share/lfs/config.json whatever it had read, which + is wrong wherever LFS_STORE points somewhere else. + """ + for cand in (os.environ.get("LFS_STORE"), "/usr/share/lfs", + os.path.expanduser("~/.local/share/lfs")): + if cand and os.path.isfile(os.path.join(cand, "config.json")): + return os.path.join(cand, "config.json") + return "/usr/share/lfs/config.json" + + +_LFS_CONFIG_HINT = "/usr/share/lfs/config.json" + + def cmd_config(args): + # --show: the whole picture, no questions. `config` on its own asks for + # the main user before it prints anything, which is not what somebody + # checking their prefixes wants. + if getattr(args, "show", False): + printHeader("Prefixes and directories") + print() + for label, value, source in _config_sources(): + print(" %-20s %-28s %s" % (label, value or "(none)", source)) + print() + # Say what the keys ARE. This told people to export variables, which + # was the only way until 1.13.69 -- and then kept telling them so, + # which is why the report looked unchanged after the keys were added. + printInfo(" every value below can be set in either file; the lfs " + "config wins for the") + printInfo(" prefixes, and the environment wins over both.") + print() + printInfo(" %s" % _lfs_config_path()) + printInfo(' "pkgusr_prefix": "p" "cfguser_prefix": "cfg"') + printInfo(' "collector_prefix": "nimgnu"') + printInfo(' "pkgusr_subdir": "pkgusr" "cfguser_subdir": "cfg"') + print() + printInfo(" %s" % CONFIG_PATH) + printInfo(" pkgusr_home=/usr/src/pkgusr (the base; the " + "subdir is taken off)") + printInfo(" pkgusr_subdir=pkgusr cfguser_subdir=cfg") + printInfo(" user_prefix=u main_user=<login>") + printWarning(" a prefix is chosen BEFORE the first account exists; " + "changing it on a built") + printWarning(" tree makes every tool look for accounts that are not " + "there.") + return + return _cmd_config_interactive(args) + + +def _cmd_config_interactive(args): cfg = load_config() changed = False + # THE PACKAGE-USER PREFIX HAD NO FLAG AT ALL. + # + # It could only be changed by editing config.json by hand -- and it is + # the one prefix that renames every account the system has. So: a flag, + # and a refusal when accounts already exist under the current one. + for _flag, _key in (("pkgusr_prefix", "pkgusr_prefix"), + ("cfguser_prefix", "cfguser_prefix")): + _new = getattr(args, _flag, None) + if _new is None: + continue + _new = _new.rstrip("_") + _old = cfg.get(_key, _DEFAULT_CONFIG.get(_key, "")) + if _new == _old: + printInfo(" %s is already %r" % (_key, _new or "")) + continue + _have = _accounts_with_prefix(_old) + if _have and not getattr(args, "force", False): + printError("%d account(s) already use the '%s_' prefix: %s%s" + % (len(_have), _old, ", ".join(sorted(_have)[:4]), + " ..." if len(_have) > 4 else "")) + printWarning(" Changing it now does NOT rename them. Every tool " + "would look for accounts") + printWarning(" named %r and find nothing: the packages would read " + "as not installed," % (_new + "_" if _new else "<none>")) + printWarning(" their manifests would orphan, and a rebuild would " + "make a second set.") + printWarning(" This belongs on a tree with no package users yet.") + printInfo(" If you are certain (a fresh tree, or you will rename " + "the accounts yourself):") + printInfo(" packagemanager config --%s %s --force" + % (_flag.replace("_", "-"), _new or "''")) + return + cfg[_key] = _new + changed = True + printWarning(" %s -> %r%s" % (_key, _new or "(no prefix)", + " [--force]" if _have else "")) + # ...and put it where it will actually be read + _save_shared_key(_key, _new) + + # THE DIRECTORIES NEED FLAGS TOO. + # + # They became configurable in 1.13.52/53 and could still only be changed + # by editing a file. Same guard as the prefixes: moving the layout does + # not move the homes, so it belongs on a tree that has none yet. + for _flag, _key in (("base_dir", "pkgusr_home"), + ("pkgusr_subdir", "pkgusr_subdir"), + ("cfguser_subdir", "cfguser_subdir")): + _new = getattr(args, _flag, None) + if _new is None: + continue + _new = _new.rstrip("/") + _homes = [h for h in (os.path.join(BASE_DIR, PKGUSR_SUBDIR), + os.path.join(BASE_DIR, CFGUSR_SUBDIR)) + if os.path.isdir(h) and os.listdir(h)] + if _homes and not getattr(args, "force", False): + printError("account homes already exist under the current layout:") + for _h in _homes: + printError(" %s (%d entries)" % (_h, len(os.listdir(_h)))) + printWarning(" Changing the layout does NOT move them. Every " + "tool would look in the new") + printWarning(" place and find nothing. This belongs on a tree " + "with no accounts yet.") + printInfo(" If you are certain (or you will move the homes " + "yourself):") + printInfo(" packagemanager config --%s %s --force" + % (_flag.replace("_", "-"), _new or "''")) + return + cfg[_key] = _new + changed = True + printWarning(" %s -> %r" % (_key, _new or "(none)")) + _save_shared_key(_key, _new) + if args.collector_prefix: cfg["collector_prefix"] = args.collector_prefix.rstrip("_") changed = True @@ -3936,8 +6908,22 @@ def _wheel_installed(site, dist, version): os.path.join(site, "%s-%s.dist-info" % (dist, version))) -def _setup_state(): - """What is already true, so setup can be run twice and do nothing twice.""" +def _have_ca_bundle(): + return any(os.path.isfile(c) and os.path.getsize(c) > 0 for c in _CA_BUNDLES) + + +def _make_ca_installed(): + return bool(shutil.which("make-ca")) or os.path.isfile("/usr/sbin/make-ca") + + +def _bootstrap_state(): + """What is already true, so bootstrap can be run twice and do nothing twice. + + Five stages, in the order they unblock each other. Each is a question this + tool already knew how to answer on its own; what was missing was the + SEQUENCE -- a freshly booted system where every check says something + different and nothing says what to do first. + """ site = _pip_site_dir() wheels = [] for url in _SETUP_WHEELS: @@ -3947,9 +6933,21 @@ def _setup_state(): "site": site, "wget": shutil.which("wget"), "wheels": wheels, + "book": _blfs_book_available(), + "make_ca": _make_ca_installed(), + "ca": _have_ca_bundle(), + "unverified": _wgetrc_is_weakened(), } +def _wgetrc_is_weakened(): + """Is certificate checking still turned off in /etc/wgetrc?""" + try: + return _WGETRC_MARK in open("/etc/wgetrc").read() + except OSError: + return False + + def _setup_install_wget(yes): """Install wget through the ordinary BLFS path. @@ -3958,53 +6956,118 @@ def _setup_install_wget(yes): manifest, and there is no version of "just the wget part" of that which is not a second implementation of it. """ - cmd = [sys.executable, os.path.abspath(__file__), "install", "wget", "--run"] + return _install_via_self("wget", yes) + + +def _install_via_self(pkg, yes, recursive=False, required_only=False, ignore=None, + ignore_prereqs=False): + cmd = [sys.executable, os.path.abspath(__file__), "install", pkg, "--run"] + if recursive: + cmd.append("--recursive") + if ignore_prereqs: + # a stack is a long unattended run: a decision to push past a missing + # tool has to reach the install it delegates to + cmd.append("--ignore-prereqs") + if required_only: + cmd.append("--ignore-recommended") + if ignore: + # ONE list, already joined by the caller. A second block here + # re-joined the STRING -- ",".join("texlive,...") is one comma per + # LETTER -- and the command line grew a second --ignore reading + # "t,e,x,l,i,v,e,...". + cmd += ["--ignore", ignore] if yes: cmd.append("--yes") printInfo(" $ " + " ".join(cmd[2:])) try: return subprocess.run(cmd).returncode == 0 except (OSError, subprocess.SubprocessError) as e: - printError("could not install wget: %s" % e) + printError("could not install %s: %s" % (pkg, e)) return False +# The stages, in the order they unblock each other. Kept as data so the +# report, the work and the tests all read the same list. +_BOOTSTRAP_STAGES = [ + # The modules come first: looking a package up in the book needs them, and + # installing the download tool IS a book lookup. + ("wheels", "Python modules for reading the books", + "`blfs` and `packagemanager` cannot parse a book"), + ("wget", "a download tool", + "nothing can be fetched at all"), + ("book", "the BLFS book on disk", + "every package lookup fails the same way: not found"), + ("make_ca", "make-ca, for certificates", + "downloads stay unverified"), + ("ca", "a CA certificate bundle", + "make-ca is installed but has generated nothing"), +] + + def cmd_setup(args): if args.sources: for u in setup_sources(): print(u) return - st = _setup_state() + st = _bootstrap_state() srcdir = getattr(args, "sources_dir", None) or setup_sources_dir() force = getattr(args, "force", False) + yes = getattr(args, "yes", False) todo_wget = not st["wget"] or force todo_wheels = [w for w in st["wheels"] if force or not w[4]] + n_wheels = len(st["wheels"]) - len(todo_wheels) - printHeader("packagemanager setup") + def mark(done): + return " ok " if done else " -- " + + printHeader("packagemanager bootstrap") + print() + print(" A freshly booted LFS system has no download tool, no Python") + print(" modules to read a book with, no BLFS book and no certificates.") + print(" These are the five things between here and a working `blfs`:") + print() + print("%s1. book-parsing modules %d of %d installed" + % (mark(not todo_wheels), n_wheels, len(st["wheels"]))) + print("%s2. a download tool %s" % (mark(st["wget"]), + st["wget"] or "wget is not installed")) + print("%s3. the BLFS book %s" % (mark(st["book"]), + "found" if st["book"] else "no book on disk")) + print("%s4. make-ca %s" % (mark(st["make_ca"]), + "installed" if st["make_ca"] else "not installed")) + print("%s5. CA certificates %s" % (mark(st["ca"]), + "present" if st["ca"] else "no bundle generated yet")) print() print(" site-packages : %s" % (st["site"] or "NOT FOUND -- is Python installed?")) print(" sources : %s" % srcdir) - print() - print(" wget : %s" % ("installed at %s" % st["wget"] if st["wget"] - else "MISSING -- will build from the BLFS book")) - for dist, ver, fn, url, have in st["wheels"]: - print(" %-13s %s" % (dist, "%s installed" % ver if have - else "%s to install" % ver)) + if st["unverified"]: + print() + printWarning(" Certificate checking is OFF in /etc/wgetrc so that") + printWarning(" make-ca can be fetched at all. It is turned back on") + printWarning(" automatically once stage 5 is done.") if not st["site"]: printError("\nNo system site-packages directory: nothing can be installed.") sys.exit(1) + # Stages 1 and 2 need no network -- only the files get-sources already put + # on disk. Stages 3-5 need a working download, which is exactly what 1 and + # 2 buy you, so they run straight after in the same command. if not todo_wget and not todo_wheels: print() - printSuccess("Nothing to do -- wget and every module are already installed.") - printInfo("Re-do it anyway with: packagemanager setup --run --force") + printSuccess("Stages 1 and 2 are done -- `blfs` and `packagemanager` can read a book.") + if not args.run: + _bootstrap_next_steps(st) + printInfo("\nDry run -- to do these, re-run with --run.") + return + failed = _bootstrap_finish(st, yes) + _bootstrap_report(failed) return if not args.run: print() - printInfo("Dry run -- to apply, re-run with --run.") + _bootstrap_next_steps(st) + printInfo("\nDry run -- to apply all of this, re-run with --run.") return # The wheels are already on disk and pip must not reach for an index: this @@ -4021,9 +7084,42 @@ def cmd_setup(args): sys.exit(1) failed = [] + + # MODULES FIRST, THEN THE DOWNLOAD TOOL. + # + # Installing wget means looking it up in the BLFS book, and reading the + # book needs requests and beautifulsoup4. With the tool installed first, + # the very first run on a fresh system could not parse the book it was + # holding and said the package did not exist: + # + # $ install wget --run + # wget: not in the BLFS book. + # searching the book for wget Nothing similar found either. + # + # ...and it only ever succeeded on a SECOND run, once stage 2 had put the + # parsers in place -- which looked like flakiness rather than an order. + # The wheels need neither the book nor a network, so they come first and + # nothing is lost. + if todo_wheels: + printHeader("1. book-parsing modules") + grp = collector_prefix() + _pip_owner_of_python() + # IN ORDER, with --no-deps: pip cannot reach an index to resolve + # anything, so the order in _SETUP_WHEELS is the dependency order and + # each module must already have what it imports. + for dist, ver, fn, url, _have in todo_wheels: + if not pip_install_module(dist, st["site"], grp, + find_links=srcdir, no_deps=True, + target=os.path.join(srcdir, fn)): + failed.append(dist) + if todo_wget: - printHeader("wget") - if not _setup_install_wget(getattr(args, "yes", False)): + printHeader("2. a download tool") + if failed: + printWarning("The book parsers did not install, so the book cannot") + printWarning("be read and the download tool cannot be looked up.") + printWarning("Fix those first; this stage would fail for that reason") + printWarning("rather than for one of its own.") + elif not _setup_install_wget(yes): failed.append("wget") else: # A system with no certificates cannot verify anything it fetches. @@ -4031,25 +7127,270 @@ def cmd_setup(args): # here means the very next download works. ensure_wget_workaround() - if todo_wheels: - grp = collector_prefix() + _pip_owner_of_python() - # IN ORDER, with --no-deps: pip cannot reach an index to resolve - # anything, so the order in _SETUP_WHEELS is the dependency order and - # each module must already have what it imports. - for dist, ver, fn, url, _have in todo_wheels: - if not pip_install_module(dist, st["site"], grp, - find_links=srcdir, no_deps=True, - target=os.path.join(srcdir, fn)): - failed.append(dist) + print() + if failed: + # Stop here: everything after this point needs what just failed. + _bootstrap_report(failed) + printSuccess("Stages 1 and 2 done.") + failed = _bootstrap_finish(_bootstrap_state(), yes) + _bootstrap_report(failed) + + +def _bootstrap_report(failed): + print() + if failed: + printError("bootstrap stopped with %d failure(s): %s" + % (len(failed), ", ".join(failed))) + printInfo("Fix them and run it again -- it skips what is already done.") + sys.exit(1) + st = _bootstrap_state() + if st["book"] and st["make_ca"] and st["ca"]: + printSuccess("Bootstrap complete -- all five stages done.") + # THE STARTUP HELPERS BELONG TO THE BOOTSTRAP. BLFS packages drop + # /etc/profile.d snippets that call pathprepend, which LFS does not + # provide -- rust's snippet failed on every package-user login. + # lfs-helper writes them; this is the command people run on a fresh + # system, so it asks for them here. + try: + _pr = subprocess.run(["lfs-helper", "install-profile"], + capture_output=True, text=True) + for _l in (_pr.stdout or "").splitlines(): + if _l.strip(): + printInfo(" " + _l.strip()) + except (OSError, subprocess.SubprocessError): + printWarning(" could not run `lfs-helper install-profile`; BLFS" + " profile.d snippets may fail with" + " 'pathprepend: command not found'") + printInfo("Try it: blfs order <pkg> / packagemanager install <pkg>") + else: + printSuccess("Nothing left that this command can do.") + + +def _bootstrap_finish(st, yes): + """Do stages 3, 4 and 5 -- the ones that need a working download. + + These used to be PRINTED, on the reasoning that each has to be seen to work + before the next is worth trying. That was a poor trade: `bootstrap --run` + is a person saying "get this system ready", and answering with three + commands to type is refusing the request while looking like an answer. + Each stage still reports what it did, and a failure still stops the ones + that depend on it -- which is the part that actually mattered. + """ + failed = [] + + # 3. the book. Only reachable now: fetching it needs the download tool + # that stage 2 just installed. + if not st["book"]: + printHeader("3. the BLFS book") + blfs = _find_blfs_cmd() + if not blfs: + printError(" blfs is not installed, so the book cannot be fetched.") + failed.append("the BLFS book") + else: + printInfo(" $ blfs fetch") + try: + rc = subprocess.run(blfs + ["fetch"]).returncode + except (OSError, subprocess.SubprocessError) as e: + printError(" could not run blfs: %s" % e) + rc = 1 + if rc != 0 or not _blfs_book_available(): + printError(" no book was fetched -- this needs a network.") + printInfo(" Or copy one in: blfs import /path/to/BLFS-BOOK-...html") + failed.append("the BLFS book") + + # 4. make-ca, through the ordinary install path so BLFS resolves what it + # needs. Without a book there is nothing to look it up in, so this is + # skipped rather than attempted and blamed on itself. + if not st["make_ca"]: + printHeader("4. make-ca") + if "the BLFS book" in failed: + printWarning(" Skipped: there is no book to look make-ca up in.") + failed.append("make-ca") + else: + printInfo(" This fetches over an UNVERIFIED connection -- that is") + printInfo(" what the /etc/wgetrc marker is for, and the last time") + printInfo(" it happens.") + if not _install_via_self("make-ca", yes, recursive=True): + failed.append("make-ca") + + # 5. the bundle. `make-ca -g` alone reports "No update required!" and + # writes nothing, because its copy of certdata.txt looks current right + # after installing -- so the install looks fine while the system still has + # no certificates. + # THE TRIGGER MUST MATCH WHAT COUNTS AS DONE. + # + # This ran only when there was NO bundle at all, while 1.11.19 made the + # finishing condition "a bundle AND no empty store". So on a tree with the + # GnuTLS bundle written and /etc/ssl/certs empty, stage 5 was skipped + # entirely and then the check below reported the empty store as a failure + # -- correctly, and without ever having tried to fix it. Reported twice, + # attempted never. + if not _have_ca_bundle() or _empty_cert_stores(): + printHeader("5. the certificate bundle") + if "make-ca" in failed: + printWarning(" Skipped: make-ca is not installed.") + failed.append("certificates") + else: + if not _run_make_ca(): + failed.append("certificates") + + # Turn verification back on only when there is something that can actually + # verify. make-ca writes several stores and EXITS 0 having failed some of + # them, so "a bundle exists" is not the same as "verification works". + if _have_ca_bundle(): + empty = _empty_cert_stores() + if empty: + printWarning("") + printWarning(" Some certificate stores were NOT written, even after") + printWarning(" granting and generating again:") + for d, why in empty: + printWarning(" %-28s %s" % (d, why)) + printWarning(" A tool that reads one of those still cannot verify") + printWarning(" anything. OpenSSL reads /etc/ssl/certs by default,") + printWarning(" so this matters to wget itself.") + printWarning(" Grant it by hand and generate once more:") + printWarning(" lfs-helper grant-dir %s %s --run" + % (_OPENSSL_CERT_DIR, pkgusr_name("make-ca"))) + printWarning(" /usr/sbin/make-ca -g --force") + if "certificates" not in failed: + failed.append("certificates") + else: + restore_wget_verification() + return failed + + +# OpenSSL's default CApath, and the file make-ca writes beside it. These are +# what `wget` itself consults, so an empty one means verification fails even +# though _have_ca_bundle found a GnuTLS bundle elsewhere and said yes. +_OPENSSL_CERT_DIR = "/etc/ssl/certs" + + +def _empty_cert_stores(): + """Stores that exist but hold nothing, with why -- best guess.""" + out = [] + d = _OPENSSL_CERT_DIR + if os.path.isdir(d): + try: + n = len([f for f in os.listdir(d) + if f.endswith((".pem", ".crt", ".0"))]) + except OSError: + n = 0 + if n == 0: + why = "empty" + try: + owner = pwd.getpwuid(os.stat(d).st_uid).pw_name + if owner != "root": + why = "empty -- owned by %s" % owner + except (KeyError, OSError): + pass + out.append((d, why)) + return out + + +def _run_make_ca(retry=True): + """Generate the certificate stores, granting access if that is what stops it. + + make-ca writes into several directories owned by other packages -- + /etc/ssl/certs belongs to p_openssl -- and it EXITS 0 having failed some of + them: + + p11-kit: couldn't create file: /etc/ssl/certs/...pem: Unknown error 13 + Failed!!! + ... + Extracting GNUTLS server auth certificates to: ...Done! + + A zero exit and one bundle written is what made bootstrap report success + with an empty OpenSSL trust store. So: read the output, grant what the + errors name, and run it again. + """ + mk = shutil.which("make-ca") or "/usr/sbin/make-ca" + printInfo(" $ %s -g --force" % mk) + try: + r = subprocess.run([mk, "-g", "--force"], capture_output=True, text=True) + except (OSError, subprocess.SubprocessError) as e: + printError(" could not run make-ca: %s" % e) + return False + out = (r.stdout or "") + (r.returncode and (r.stderr or "") or (r.stderr or "")) + sys.stdout.write(r.stdout or "") + sys.stderr.write(r.stderr or "") + + if r.returncode == 0 and not _empty_cert_stores(): + return _have_ca_bundle() + + if not retry: + if not _have_ca_bundle(): + printError(" no certificate bundle was generated.") + return False + + # Which directories did it fail on? Written to a file so the one parser + # both this and an install failure go through can read it. + tmp = os.path.join(TMP_DIR, "make-ca.log") + try: + os.makedirs(TMP_DIR, exist_ok=True) + with open(tmp, "w") as f: + f.write(out) + except OSError: + return False + account = pkgusr_name("make-ca") + if not _user_exists(account): + return _have_ca_bundle() + granted = auto_grant_from_log(account, tmp) + if not granted: + printWarning(" make-ca could not write every store, and nothing in its") + printWarning(" output names a directory to grant. Check by hand:") + printWarning(" ls -ald %s" % _OPENSSL_CERT_DIR) + return False + printInfo(" granted; generating again ...") + return _run_make_ca(retry=False) + + +def _find_blfs_cmd(): + if shutil.which("blfs"): + return ["blfs"] + here = os.path.join(os.path.dirname(os.path.abspath(__file__)), "blfs") + if os.path.isfile(here): + return [sys.executable, here] + return None + +def _bootstrap_next_steps(st): + """What is left, and the command for it -- for a DRY RUN. + + `--run` does these (see _bootstrap_finish); this is what to expect. + """ + if st["book"] and st["make_ca"] and st["ca"]: + print() + printSuccess("Bootstrap complete -- nothing is left to do.") + printInfo("Try it: blfs order <pkg> / packagemanager install <pkg>") + return print() - if failed: - printError("setup finished with %d failure(s): %s" - % (len(failed), ", ".join(failed))) - printInfo("Fix them and run it again -- it skips what is already done.") - sys.exit(1) - printSuccess("setup complete.") - printInfo("Check it: python3 -c 'import requests, bs4' && wget --version | head -1") + printHeader("what --run would do next") + n = 3 + if not st["book"]: + print() + print(" %d. The BLFS book. Without it every lookup fails as" % n) + print(" 'not found in book', whatever you ask for.") + print(" blfs books --discover") + print(" blfs fetch <version>") + print(" Or copy a book you already have into /usr/share/blfs/books/.") + n += 1 + if not st["make_ca"]: + print() + print(" %d. make-ca. Until this is installed every download is" % n) + print(" unverified -- which is why /etc/wgetrc has verification") + print(" turned off: make-ca itself has to be fetched somehow.") + print(" packagemanager install make-ca --run") + n += 1 + if not st["ca"]: + print() + print(" %d. The certificate bundle. `make-ca -g` generates nothing" % n) + print(" when it thinks its copy of certdata.txt is current, which") + print(" it is right after installing -- so force it once:") + print(" /usr/sbin/make-ca -g --force") + print(" The tools re-enable certificate checking on their own once") + print(" a bundle exists.") + print() + pass def cmd_user_create(args): @@ -4435,14 +7776,15 @@ def cmd_user_delete(args): # =========================================================================== # # # Write an install-script skeleton for a package that isn't in the BLFS book, -# in the exact shape packagemanager_install expects (header vars + install_pkg() +# in the exact shape lfs-helper pm-install expects (header vars + install_pkg() # [+ configure_pkg()] + the sourced-safe install/update dispatcher). _TEMPLATE = '''\ #!/bin/bash ############################################ ### {name_version} -### phased install script: all|unpack|build|install|configure|update +### script format v17 origin=packagemanager +### phases: all (default) | unpack | build | test | install | configure | update url="" # optional: where this came from name="{name}" @@ -4471,92 +7813,45 @@ installed_directories=() validate_cmd="" # install_groups=('{collector}_...') # if it installs into collector dirs -BUILD_ROOT="${{BUILD_ROOT:-$PWD}}" -pkg_dir="" - -_enter_build() {{ -\tcd "$BUILD_ROOT" || exit 1 -\tif [ -z "$pkg_dir" ]; then -\t\tpkg_dir="$name_version" -\t\t[ -d "$pkg_dir" ] || pkg_dir="$(ls -d */ 2>/dev/null | head -n1)" -\tfi -\tif [ -n "$pkg_dir" ] && [ -d "$pkg_dir" ]; then cd "$pkg_dir" || exit 1 -\telse echo "no source in $BUILD_ROOT -- run: $0 unpack" && exit 1; fi -}} - -unpack_pkg() {{ -#### UNPACK #### -\tcd "$BUILD_ROOT" || exit 1 -\tif [ "$is_git" = "1" ]; then -\t\tpkg_dir="$name_version" -\t\tif [ -d "$pkg_dir/.git" ]; then -\t\t\tgit -C "$pkg_dir" fetch --all --tags --prune -\t\telse -\t\t\trm -rf "$pkg_dir" && git clone --recurse-submodules "$link" "$pkg_dir" -\t\tfi -\t\tif [ -n "$git_ref" ]; then git -C "$pkg_dir" checkout "$git_ref"; fi -\t\tgit -C "$pkg_dir" submodule update --init --recursive -\telse -\t\t[ -f "$pkg" ] || {{ [ -n "$link" ] && wget -4 "$link"; }} -\t\tif [ -n "$md5_sum" ] && ! grep -wq "$md5_sum" <<< "$(md5sum "$pkg")"; then -\t\t\techo "ERROR: md5sum check failed" && exit 1 -\t\tfi -\t\trm -rf tmp && mkdir tmp && cd tmp && tar -xf ../"$pkg" -\t\tif [ "$(ls | wc -w)" = "1" ]; then pkg_dir="$(ls)"; else pkg_dir="$name_version"; fi -\t\tcd .. && [ -d "$pkg_dir" ] && rm -rf "$pkg_dir" -\t\tif [ "$(ls tmp | wc -w)" = "1" ]; then mv tmp/* . ; else mv tmp "$pkg_dir"; fi -\t\trm -rf tmp -\tfi -\tfor li in "${{additional_links[@]}}"; do wget -4 "$li"; done -\techo "source ready in $BUILD_ROOT/$pkg_dir" -#### UNPACK DONE #### -}} - build_pkg() {{ -#### BUILD #### -\t_enter_build -\tfor _d in build builddir _build; do [ -d "$_d" ] && rm -rf "$_d"; done -\t# >>> your CONFIGURE + COMPILE commands go here (no install yet) <<< -\t# meson setup build --prefix=/usr && ninja -C build -\t# ./configure --prefix=/usr && make -\tpwd > "$BUILD_ROOT/.pm_build_cwd" 2>/dev/null || true -#### BUILD DONE #### +# >>> your CONFIGURE + COMPILE commands go here (no install yet) <<< +# meson setup build --prefix=/usr && ninja -C build +# ./configure --prefix=/usr && make +: }} install_pkg() {{ -#### INSTALL #### -\t_enter_build -\tif [ -f "$BUILD_ROOT/.pm_build_cwd" ] && [ -d "$(cat "$BUILD_ROOT/.pm_build_cwd" 2>/dev/null)" ]; then -\t\tcd "$(cat "$BUILD_ROOT/.pm_build_cwd")" -\telif [ ! -e build.ninja ] && [ ! -e Makefile ]; then -\t\tfor _d in build builddir _build; do [ -e "$_d/build.ninja" ] || [ -e "$_d/Makefile" ] && cd "$_d" && break; done -\tfi -\t# >>> your INSTALL commands go here (the step that writes into system dirs) <<< -\t# ninja install / make install -#### INSTALL DONE #### +# >>> your INSTALL commands go here (the step that writes into system dirs) <<< +# ninja -C build install / make install +: }} configure_pkg() {{ -#### CONFIGURE #### -\techo "## Configuration" -\t# >>> optional post-install configuration goes here <<< -#### CONFIGURE DONE #### +# >>> optional post-install configuration goes here <<< +: }} -if [ "${{BASH_SOURCE[0]}}" = "${{0}}" ]; then -case "${{1:-all}}" in -\tall) unpack_pkg && build_pkg && install_pkg && configure_pkg ;; -\tunpack) unpack_pkg ;; -\tbuild) build_pkg ;; -\tinstall) install_pkg ;; -\tconfigure) configure_pkg ;; -\tupdate) unpack_pkg && build_pkg && install_pkg ;; -\t*) echo "usage: $0 {{all|unpack|build|install|configure|update}}" ;; -esac -fi +# fetch, unpack, run each phase, resume in the build dir: all in the runner +. "${{PKGUSR_LIB:-lfs-phases}}" && pkgusr_run "$@" ''' +_FN_RE = r"^%s\(\)\s*\{\n(.*?)^\}[ \t]*$" + + +def _phase_body(text, fn): + """The body of a phase function (build_pkg, install_pkg, ...) in a script + of the shared-runner format, None when the script does not define it.""" + m = re.search(_FN_RE % re.escape(fn), text, re.S | re.M) + return m.group(1) if m else None + + +def _set_phase_body(text, fn, body): + """Replace a phase function's body in a template-shaped script.""" + return re.sub(_FN_RE % re.escape(fn), lambda m: "%s() {\n%s\n}" % (fn, body), + text, count=1, flags=re.S | re.M) + + def _between(text, a, b): m = re.search(re.escape(a) + r"\n?(.*?)" + re.escape(b), text, re.S) return m.group(1) if m else None @@ -4589,6 +7884,11 @@ def extract_phase_cmds(text, phase): """Pull the user's actual commands (scaffolding removed) out of a phased install script for build|install|configure. None if that phase isn't in this script format.""" + if phase in ("build", "install", "configure"): + # shared-runner format: the function IS the user's commands + r = _phase_body(text, phase + "_pkg") + if r is not None: + return _strip_lines(r, _CONFIG_SCAFFOLD if phase == "configure" else ()) if phase == "build": r = _between(text, "#### BUILD ####", "#### BUILD DONE ####") return _strip_lines(r, _BUILD_SCAFFOLD) if r is not None else None @@ -4750,9 +8050,38 @@ def _restamp_script(path): return False +def _script_origin(text): + """Which generator wrote this script, per its own stamp ('' if unstamped). + + The shared contract (item 6, step 3): every generated script carries + `script format vN origin=<tool>`. Before it, this tool knew only blfs's + wording, so an LFS-book script failed the format test and was reported as + OLD-format -- with advice that would have rewritten it into a BLFS one. + Recognising the other dialect was the containment; asking is the fix.""" + m = re.search(r"script format v\d+\s+origin=(\w+)", text or "") + if m: + return m.group(1) + # Legacy, before the shared stamp: infer from each tool's own mark. + if "### lfs-crosschain-script v" in (text or ""): + return "lfs" + if "generated by blfs" in (text or ""): + return "blfs" + return "" + + +def _is_lfs_book_script(text): + """Was this script generated by `lfs` from the LFS book?""" + return _script_origin(text) == "lfs" + + def _package_script_outdated(name, current_ver): """True if this package's canonical install_<ver> script predates the current - format (so a reinstall would reuse an old script / risk losing edits).""" + format (so a reinstall would reuse an old script / risk losing edits). + + Only scripts THIS tool's generator owns are judged. The format counters + are per generator and independent -- comparing blfs's v4 with lfs's v17 + would be a number against an unrelated number -- so a script from another + origin is left to the tool that writes it.""" p = _canonical_script_path(name) if not p: return False @@ -4760,6 +8089,9 @@ def _package_script_outdated(name, current_ver): txt = open(p, encoding="utf-8", errors="replace").read(6000) except OSError: return False + origin = _script_origin(txt) + if origin and origin != "blfs": + return False return not _script_is_current(txt, current_ver) @@ -4829,13 +8161,14 @@ def cmd_script_status(args): script_dir = os.path.join(TMP_DIR, "install_files") os.makedirs(script_dir, exist_ok=True) quiet = getattr(args, "quiet", False) - changed_names, counts = [], {} + changed_names, counts, _states = [], {}, {} rows = [] for name in sorted(set(names)): st, cand, _bp, _nv = classify_script(name, script_dir) if st == "no-script": continue counts[st] = counts.get(st, 0) + 1 + _states[name] = st rows.append((name, st, cand)) if st in ("edited", "no-baseline", "outdated"): changed_names.append(name) @@ -4851,6 +8184,56 @@ def cmd_script_status(args): color = _SCRIPT_STATUS_COLOR.get(st, "") print(f" {name:<24} {color}{st:<12}{colors.normal} {cand or ''}") print("\n " + " ".join(f"{k}: {v}" for k, v in sorted(counts.items()))) + # LEAD WITH WHAT NEEDS DOING. + # + # `unvalidated` is the normal state for most of a system -- it means only + # that the page named no file to check, not that anything is wrong -- and + # on a real tree it is 226 of 242 lines. The dozen that need action + # scroll past inside it. Repeat them at the end, where the eye lands, + # with the command that fixes them. + # ONE PASS, NOT TWO. This re-assessed every package to find the ones to + # repeat -- a second validate command per package on a 242-package tree. + # The states were already computed in the loop above. + _act = [n for n, st in _states.items() if st in ("not installed", "broken")] + if _act: + print() + printWarning("%d package(s) need attention (everything else is fine " + "-- 'unvalidated' just means" % len(_act)) + printWarning("the book named no file to check):") + for n in _act: + u = gather_user(n) + st, why = assess(u, versions.get(u.meta.get("name") or n)) + printWarning(" %-24s %-14s %s" % (n, st, why)) + # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. + # + # The list was pasted straight into an install command that could not + # work: collector groups and a stray `root` are not packages at all, + # and pkgusr / linux-headers are LFS steps with no BLFS page. The one + # bad name then killed the whole run (fixed in 1.14.36), but even + # without that, offering a command that cannot succeed is worse than + # offering none. + _vers = book_versions_all() or {} + _book = [n for n in _act if _vers.get(n) or _vers.get(n.lower())] + _other = [n for n in _act if n not in _book] + if _book: + printInfo(" Rebuild the BLFS ones:") + printInfo(" packagemanager install %s --run --recursive" + % " ".join(_book)) + if _other: + printInfo(" Not BLFS packages, and not rebuilt by that command:") + printInfo(" " + ", ".join(_other)) + printInfo(" LFS steps (linux-headers, pkgusr, ...) belong to the" + " chroot build: lfs-helper build <name> --force") + # NOT `packagemanager remove root`. For a directory with no + # account the fix is to delete the DIRECTORY, and suggesting a + # remove command against a name like `root` is an invitation to + # an accident. + printInfo(" Anything else is a directory or account with no" + " package behind it -- the line") + printInfo(" above names it; a leftover directory can simply be" + " removed.") + else: + printSuccess(" nothing needs attention.") if changed_names: printInfo("Migrate the changed/outdated ones: packagemanager " "migrate-scripts " + " ".join(changed_names) + " --run") @@ -5088,7 +8471,997 @@ def cmd_template(args): printInfo("Edit the build/install commands, then:\n" f" packagemanager install {name} --local {path} --run\n" f" (or: packagemanager add-user {name}; " - f"packagemanager_install {name} {path})") + f"lfs-helper pm-install {name} {path})") + + + + +# =========================================================================== # +# command: env [IMPLEMENTED] +# =========================================================================== # +# +# The book asks for a handful of exported variables (TEXLIVE_PREFIX, +# XORG_CONFIG, ...). Exporting them in a shell does not help: builds run as +# a package user through `su -`, which drops the environment. They live in +# one file instead, and lfs-helper passes it to every phase. + +BUILD_ENV_FILE = os.environ.get("LFS_BUILD_ENV", "/etc/pkgusr/build.env") + + +def _read_build_env(): + out = {} + try: + with open(BUILD_ENV_FILE) as f: + for line in f: + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + out[k.strip()] = v.strip() + except OSError: + pass + return out + + +def _write_build_env(vals): + os.makedirs(os.path.dirname(BUILD_ENV_FILE), exist_ok=True) + with open(BUILD_ENV_FILE, "w") as f: + f.write("# Build-time variables the BLFS book expects to be exported.\n") + f.write("# Written by `packagemanager env`; read by every build.\n") + for k in sorted(vals): + f.write("%s=%s\n" % (k, vals[k])) + + +# Values the book states plainly but that the generator cannot lift safely: +# XORG_CONFIG is a quoted, multi-line flag list built from XORG_PREFIX, and a +# half-captured fragment would be worse than asking. Offered as a SUGGESTION +# at the prompt -- the user presses Enter or types their own. +_ENV_SUGGESTIONS = { + "XORG_PREFIX": "/usr", + "KF6_PREFIX": "/usr", + "QT6PREFIX": "/opt/qt6", + "TEXLIVE_PREFIX": "/opt/texlive/2025", + "XORG_CONFIG": "--prefix=/usr --sysconfdir=/etc " + "--localstatedir=/var --disable-static", +} + + +def _env_wanted_by_scripts(): + """Which variables do the generated scripts ask for, and does the book + answer them itself? Returns {name: suggested_value_or_empty}.""" + wanted, stale, asked_by = {}, set(), {} + dirs = [os.path.join(TMP_DIR, "install_files")] if TMP_DIR else [] + dirs += [os.path.join(pkgusr_root(), d, "") + for d in ()] # package homes are scanned below + for d in dirs: + if not d or not os.path.isdir(d): + continue + for fn in os.listdir(d): + try: + body = open(os.path.join(d, fn), encoding="utf-8", + errors="replace").read() + except OSError: + continue + # A SCRIPT FROM AN OLDER GENERATOR ASKS OLDER QUESTIONS. + # These files are whatever the last run produced; after the guard + # was narrowed (1.12.56) the stale ones still listed EUID, + # DOCNAME and the rest, so the fix looked like it had done + # nothing. Only scripts from the CURRENT generator are consulted. + gen = _script_gen_version(os.path.join(d, fn)) + cur = _blfs_gen_version() or TOOL_VERSION + if gen and cur and gen != cur: + stale.add(fn) + continue + for n in _ENV_GUARD_RE.findall(body): + wanted.setdefault(n, "") + asked_by.setdefault(n, set()).add(fn.replace("install_", "")) + # the book's own answer, if the generator found one + for n, v in re.findall(r'^\t([A-Z][A-Z0-9_]{2,})="([^"]*)"\s+# set by', + body, re.M): + wanted[n] = v + _env_wanted_by_scripts.asked_by = asked_by + if stale and not wanted: + # EVERY script was stale, so this returned nothing and `env --ask` + # had nothing to ask about -- which is how XORG_PREFIX went unset + # while XORG_CONFIG was being set from the very same book page. + printWarning("all %d cached script(s) are from an older blfs, so" + " nothing here can say which variables your packages" + " need." % len(stale)) + printWarning(" They are regenerated on the next install; the common" + " ones are:") + for n in sorted(_ENV_SUGGESTIONS): + printWarning(" packagemanager env --set %s=%s" + % (n, _ENV_SUGGESTIONS[n])) + elif stale: + printWarning("ignored %d script(s) from an older blfs -- they ask" + " questions the current one no longer asks; they are" + " regenerated on the next install" % len(stale)) + return wanted + + +# The book's Xorg page sets these TOGETHER: +# export XORG_PREFIX="/usr" +# export XORG_CONFIG="--prefix=$XORG_PREFIX --sysconfdir=/etc ..." +# so someone who sets XORG_CONFIG has read that page and means both. Setting +# only one leaves the other packages to fail later, one at a time -- which is +# what happened: XORG_CONFIG was set, and seventeen packages later libevdev +# stopped on XORG_PREFIX. +_ENV_COMPANIONS = { + "XORG_CONFIG": ["XORG_PREFIX"], + "XORG_PREFIX": ["XORG_CONFIG"], +} + + +def _warn_env_companions(vals): + for name in sorted(vals): + for other in _ENV_COMPANIONS.get(name, ()): + if other in vals: + continue + printWarning("%s is set but %s is not, and the book's page sets" + " both." % (name, other)) + printWarning(" packagemanager env --set %s=%s" + % (other, _ENV_SUGGESTIONS.get(other, "..."))) + + +def cmd_env(args): + vals = _read_build_env() + if args.set: + for pair in args.set: + if "=" not in pair: + printError("not NAME=VALUE: %s" % pair) + sys.exit(1) + k, v = pair.split("=", 1) + vals[k.strip()] = v.strip() + _write_build_env(vals) + printSuccess("wrote %s" % BUILD_ENV_FILE) + _warn_env_companions(vals) + + wanted = _env_wanted_by_scripts() + if args.ask: + for name in sorted(wanted): + if name in vals: + continue + suggested = wanted[name] or _ENV_SUGGESTIONS.get(name, "") + prompt = " %s" % name + if suggested: + prompt += " [%s]" % suggested + who = sorted(getattr(_env_wanted_by_scripts, "asked_by", {}).get(name, ())) + printWarning("%s is needed by: %s" % (name, ", ".join(who) or "a generated script")) + if who: + printInfo(" the book's page for that package explains it:" + " blfs debug %s" % re.sub(r"-[0-9].*$", "", who[0])) + if not suggested: + printInfo(" press Enter to skip -- the build stops at that" + " package rather than using an empty value") + try: + got = input(prompt + ": ").strip() + except EOFError: + got = "" + vals[name] = got or suggested + if not vals[name]: + del vals[name] + _write_build_env(vals) + printSuccess("wrote %s" % BUILD_ENV_FILE) + + if args.refresh: + # STALE SCRIPTS HIDE THE QUESTIONS. They are regenerated per + # install anyway, so deleting them costs nothing and lets the next + # `env` (or the next build) see the current generator's guards. + removed = 0 + d = os.path.join(TMP_DIR, "install_files") if TMP_DIR else None + cur = _blfs_gen_version() or TOOL_VERSION + if d and os.path.isdir(d): + for fn in os.listdir(d): + fp = os.path.join(d, fn) + gen = _script_gen_version(fp) + if gen and cur and gen != cur: + try: + os.unlink(fp) + removed += 1 + except OSError: + pass + printSuccess("removed %d stale script(s); they regenerate on demand" + % removed) + wanted = _env_wanted_by_scripts() + + printHeader("build environment (%s)" % BUILD_ENV_FILE) + if not vals and not wanted: + print(" nothing set, and no generated script asks for anything") + for k in sorted(set(vals) | set(wanted)): + state = vals.get(k) + if state: + print(" %-20s = %s" % (k, state)) + else: + _sugg = wanted.get(k) or _ENV_SUGGESTIONS.get(k, "") + _who = sorted(getattr(_env_wanted_by_scripts, "asked_by", {}).get(k, ())) + print(" %-20s NOT SET (%s%s)" + % (k, + "wanted by " + ", ".join(_who[:2]) if _who else "a script needs it", + "; the book says %s" % _sugg if _sugg else "")) + if any(k not in vals for k in wanted): + printInfo("\n set them with: packagemanager env --ask") + printInfo(" or one at a time: packagemanager env --set NAME=VALUE") + elif not vals: + # An empty list is not a promise. A variable only becomes visible + # once the package's script has been generated, which happens during + # the run -- so say what will happen rather than implying there is + # nothing to do. + printInfo("\n A variable becomes visible when its package's script is") + printInfo(" generated, which happens during the build. The build warns") + printInfo(" before that package starts, and stops rather than passing an") + printInfo(" empty value. Set one now if you already know it:") + printInfo(" packagemanager env --set XORG_CONFIG='--prefix=/usr " + "--sysconfdir=/etc --localstatedir=/var --disable-static'") + + +# =========================================================================== # +# command: stack [IMPLEMENTED] +# =========================================================================== # +# +# One command, one environment. A stack file is the PLAN -- an ordered list +# of targets that together make a working environment (sway + a webkit +# browser, say) -- and `packagemanager stack <name> --run` replays it through +# the exact machinery a full fresh build proved: book targets go through the +# ordinary install planner (BLFS resolves the dependency chains, installed +# packages are skipped), git targets are built from an auto-generated local +# template, and check/cfg targets are root scripts shipped beside the stack. +# It stops at the first failure and skips what is done, so re-running after a +# fix continues instead of starting over. +# +# File format, one entry per line: +# book <target> [key=value ...] a BLFS book package (deps resolved) +# git <name> url=<repo> [ref=<ref>] [prog=<installed-program>] +# tar <name> url=<tarball-url> pkg=<filename> [prog=|have=] [build=|install=] +# check <name> root script <stackdir>/<name>.sh; +# a non-zero exit STOPS the stack +# cfg <name> root script <stackdir>/<name>.sh +# '#' comments and blank lines are ignored. Entries run top to bottom. +# +# Stack files are looked for in: the path as given, /etc/pkgusr/stacks/, +# and stacks/ beside this tool. + +def _find_stack_file(name): + cands = [name] + if not name.endswith(".stack"): + cands.append(name + ".stack") + out = [] + for c in cands: + out.append(c) + out.append(os.path.join("/etc/pkgusr/stacks", c)) + out.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), + "stacks", c)) + for c in out: + if os.path.isfile(c): + return os.path.abspath(c) + return None + + +def _parse_stack_file(path): + """-> list of {kind, target, opts{}} in file order. Raises ValueError + with a line number on anything malformed -- a plan half-understood is + worse than one refused.""" + import shlex + entries = [] + kinds = ("book", "git", "tar", "check", "cfg", "avoid") + # A LINE MAY CONTINUE. + # + # Every entry had to be one line, so sway.stack's are 300 characters wide + # and nobody can read the build= command in them. A trailing backslash + # joins, as it does everywhere else in a shell-shaped file; the line + # NUMBER reported for an error stays the one the entry started on. + _joined, _buf, _start = [], "", 0 + with open(path) as f: + for n, raw in enumerate(f, 1): + _s = raw.rstrip("\n") + if _buf: + _s = _buf + " " + _s.strip() + else: + _start = n + if _s.rstrip().endswith("\\"): + _buf = _s.rstrip()[:-1].rstrip() + continue + _buf = "" + _joined.append((_start, _s)) + if _buf: + raise ValueError("%s:%d: line ends with a backslash and nothing follows" + % (path, _start)) + for n, raw in _joined: + line = raw.strip() + if not line or line.startswith("#"): + continue + # shlex, not split(): build= commands carry spaces, so values are + # quoted -- build="meson setup build; ninja -C build" + try: + parts = shlex.split(line, comments=True) + except ValueError as e: + raise ValueError("%s:%d: %s" % (path, n, e)) + if not parts: + continue + kind = parts[0] + if kind not in kinds: + raise ValueError("%s:%d: unknown entry kind %r (one of %s)" + % (path, n, kind, "/".join(kinds))) + if len(parts) < 2: + raise ValueError("%s:%d: %r needs a target" % (path, n, kind)) + target = parts[1] + opts = {} + for kv in parts[2:]: + if "=" not in kv: + raise ValueError("%s:%d: option %r is not key=value" + % (path, n, kv)) + k, v = kv.split("=", 1) + opts[k] = v + if kind == "avoid": + # a package this stack must never build, however it is + # reached -- texlive and LibreOffice arrive as RECOMMENDED + # dependencies of harfbuzz, three layers down, and a desktop + # does not want either + entries.append({"kind": "avoid", "target": target, "opts": opts}) + continue + if kind == "git" and "url" not in opts: + raise ValueError("%s:%d: a git entry needs url=" % (path, n)) + if kind == "tar" and ("url" not in opts or "pkg" not in opts): + raise ValueError("%s:%d: a tar entry needs url= and pkg=" % (path, n)) + entries.append({"kind": kind, "target": target, "opts": opts}) + if not entries: + raise ValueError("%s: the stack file has no entries" % path) + return entries + + +def _stack_git_script(entry, script_dir): + """Generate the local install script for a git OR tar entry, in the + template's shape, with the family's build convention: make all && make + PREFIX=/usr install. Overridable per entry with build= / install= + (';'-separated).""" + name = entry["target"] + opts = entry["opts"] + is_git = "1" if entry["kind"] == "git" else "" + build_cmds = opts.get("build", "make all").replace(";", "\n") + install_cmds = opts.get("install", "make PREFIX=/usr install").replace(";", "\n") + # `prog=` is the BINARY this entry installs; it is not always the entry's + # own name. Defaulting it to the name made verify check libsrtp for a + # program called `libsrtp`, and report "missing: program libsrtp" for + # every library and data package in the stack -- twelve of them. + prog = opts.get("prog", "") + text = _TEMPLATE.format(name=name, version=opts.get("version", "git"), + name_version="%s-%s" % (name, opts.get("version", "git")), + link=opts["url"], pkg=opts.get("pkg", ""), + is_git=is_git, + collector=collector_prefix().rstrip("_")) + # pin the ref and the doneness signal into the generated header + text = text.replace('git_ref=""', 'git_ref="%s"' % opts.get("ref", ""), 1) + text = text.replace('installed_program=""', 'installed_program="%s"' % prog, 1) + # ...and `have=` is the entry's OWN statement of what proves it installed + # -- the stack already uses it to skip an entry that is done. It is the + # right validation target for a package with no binary: a .pc file, a + # header directory, a font directory. + _have = opts.get("have", "") + if _have: + text = text.replace('installed_directory=""', + 'installed_directory="%s"' % _have, 1) + # the template ships with placeholder build/install bodies; put the + # family's convention in + text = _set_phase_body(text, "build_pkg", build_cmds) + text = _set_phase_body(text, "install_pkg", install_cmds) + os.makedirs(script_dir, exist_ok=True) + path = os.path.join(script_dir, "install_%s" % name) + with open(path, "w") as f: + f.write(text) + os.chmod(path, 0o755) + return path, (prog or name) + + +def _stack_root_script(entry, stack_dir): + return os.path.join(stack_dir, entry["target"] + ".sh") + + +def _stack_state_file(stack_path): + d = os.path.join(os.environ.get("LFS_PKGUSR_DIR", "/usr/src/lfs-pkgusr"), + "progress", "stacks") + base = os.path.basename(stack_path) + if base.endswith(".stack"): + base = base[:-6] + return os.path.join(d, base + ".done") + + +def _stack_done_set(stack_path): + try: + return set(l.strip() for l in open(_stack_state_file(stack_path)) + if l.strip()) + except OSError: + return set() + + +def _stack_mark_done(stack_path, key): + f = _stack_state_file(stack_path) + try: + os.makedirs(os.path.dirname(f), exist_ok=True) + with open(f, "a") as fh: + fh.write(key + "\n") + except OSError as e: + printWarning("could not record stack progress (%s) -- the next run " + "re-judges this entry" % e) + + +def _check_stack_refs(rows): + """Ask every git remote whether the ref this stack pins still exists. + + Two entries in one stack had rotted: usrsctp ref=1.0.0 (the tags are + 0.9.3.0, 0.9.4.0, 0.9.5.0) and foot ref=1.24.1 (1.24.0 exists, 1.24.1 + never did). Each cost a clone, a build attempt and a round trip to find + out. A pinned ref is data that rots, and only the remote can check it -- + so check them all at once, before anything is built. + """ + git = shutil.which("git") + if not git: + printError("git is not installed -- cannot check any ref.") + return 1 + bad = unreachable = 0 + # (unreachable is counted for no-ref entries too, so the summary cannot + # claim an all-clear over a remote nobody asked) + for r in rows: + if r.get("kind") != "git": + continue + url = (r["opts"].get("url") or "").strip() + ref = (r["opts"].get("ref") or "").strip() + target = r.get("target") + if not url: + continue + if not ref: + # AN ENTRY WITH NO REF STILL HAS A URL. + # + # Four entries tracked their default branch and were passed over + # untouched, so the check said "every pinned ref exists" without + # having contacted them at all. A repository that has been + # renamed, made private or moved fails identically to a bad ref -- + # at clone time, after everything before it has built. Ask. + try: + out = subprocess.run([git, "ls-remote", "--heads", url], + capture_output=True, text=True, timeout=120) + except (OSError, subprocess.SubprocessError): + out = None + if out is None or out.returncode != 0: + printWarning(" %-22s no ref=, and %s could not be reached" + % (target, url)) + unreachable += 1 + elif not out.stdout.strip(): + bad += 1 + printError(" %-22s no ref=, and the remote has NO branches" + % target) + printInfo(" %s -- moved, renamed or empty?" % url) + else: + printSuccess(" %-22s no ref= (default branch, reachable)" + % target) + continue + if re.fullmatch(r"[0-9a-f]{7,40}", ref): + printInfo(" %-22s %s (a commit id -- not checked)" % (target, ref)) + continue + try: + out = subprocess.run([git, "ls-remote", "--tags", "--heads", url], + capture_output=True, text=True, timeout=120) + except (OSError, subprocess.SubprocessError) as e: + printWarning(" %-22s could not reach %s (%s)" % (target, url, e)) + unreachable += 1 + continue + if out.returncode != 0: + printWarning(" %-22s could not reach %s" % (target, url)) + unreachable += 1 + continue + names = {l.rsplit("/", 1)[-1] for l in out.stdout.splitlines()} + names = {n[:-3] if n.endswith("^{}") else n for n in names} + if ref in names: + printSuccess(" %-22s %s" % (target, ref)) + continue + bad += 1 + printError(" %-22s %s DOES NOT EXIST" % (target, ref)) + # SORT TAGS AS VERSIONS, NOT AS STRINGS. + # + # foot's real tags include 1.28.0, and a reverse string sort offered + # "1.9.2, 1.9.1, 1.9.0, 1.9, 1.8.2" -- because "9" > "2" one character + # at a time. A suggestion that names an ancient release as the newest + # is worse than no suggestion: it invites a downgrade. + def _vkey(n): + base = n[1:] if n[:1] == "v" else n + parts = re.split(r"[.\-_]", base) + out = [] + for x in parts: + out.append((0, int(x), "") if x.isdigit() else (1, 0, x)) + return out + near = sorted((n for n in names if n[:1].isdigit() or n[:1] == "v"), + key=_vkey, reverse=True)[:5] + if near: + printInfo(" remote has: " + ", ".join(near)) + if bad: + print() + printError("%d entr%s a ref the remote does not have." + % (bad, "y pins" if bad == 1 else "ies pin")) + printInfo(" Fix ref= in the stack file before building.") + else: + print() + # "every pinned ref exists" while five remotes were unreachable is a + # false all-clear: the ones that could not be asked are exactly the + # ones that might be wrong. + if unreachable: + printSuccess("every ref that COULD be checked exists.") + else: + printSuccess("every pinned ref exists.") + if unreachable: + printWarning(" %d remote(s) could not be reached -- those refs are unverified." + % unreachable) + return 1 if bad else 0 + + +def cmd_stack(args): + path = _find_stack_file(args.name) + if not path: + printError("no stack file for %r (looked in ., /etc/pkgusr/stacks, " + "stacks/ beside this tool)" % args.name) + sys.exit(1) + try: + entries = _parse_stack_file(path) + # --yes REACHES THE SCRIPTS THIS RUNS. A cfg step calls `lfs-helper + # pm-install`, which asks "Press enter to continue" unless PM_YES is + # set -- so an unattended stack run stopped at a prompt nobody was + # there to answer. + if getattr(args, "yes", False): + os.environ["PM_YES"] = "1" + os.environ["LFS_ASSUME_YES"] = "1" + # A NEWER VERSION OF THIS FILE IS SITTING BESIDE IT. + # + # `make install` leaves the shipped copy as <stack>.new when the + # user's own differs (1.14.58/1.14.98), and running the stack said + # nothing about it: "3 entries" while the .new had six, twice in a + # row, with no hint that a merge was owed. The moment to mention it + # is the moment someone runs the file. + _new = path + ".new" + if os.path.isfile(_new): + try: + _n_entries = len(_parse_stack_file(_new)) + except (OSError, ValueError): + _n_entries = None + if (getattr(args, "take_new", False) and _n_entries is not None + and _n_entries < len(entries) and not args.force): + # A .new WITH FEWER ENTRIES IS A DOWNGRADE. + # + # The chroot refresh writes <stack>.new from the HOST's copy + # when the chroot's differs (1.14.105). If the host is the + # stale one -- it often is, since `make install` keeps a file + # it thinks the user edited -- then --take-new would replace + # a 6-entry stack with a 3-entry one, and the next refresh + # would offer it again. Refuse the shrink; --force overrides. + printError("that .new file has FEWER entries (%d) than this one" + " (%d) -- not taking it." % (_n_entries, len(entries))) + printInfo(" It is probably the host's older copy, brought in by" + " the chroot refresh.") + printInfo(" Update the host first:") + printInfo(" cp /etc/pkgusr/stacks/%s{.new,} # on the HOST" + % os.path.basename(path)) + printInfo(" ...or take it anyway with --take-new --force.") + sys.exit(1) + if getattr(args, "take_new", False): + # TAKE IT, rather than describe it again. Three runs in a + # row ended with the user reading a note and doing nothing: + # the note was right and the work it asked for was mine to + # do. Theirs is kept as .bak, so nothing is lost. + _bak = path + ".bak" + try: + shutil.copy2(path, _bak) + shutil.copy2(_new, path) + os.remove(_new) + except OSError as e: + printError("could not take the new stack file: %s" % e) + sys.exit(1) + printSuccess("took the shipped stack file; yours is %s" % _bak) + entries = _parse_stack_file(path) + printInfo(" %d entries now." % len(entries)) + print() + _new = None + if _new and os.path.isfile(_new): + printWarning("a newer version of this stack ships beside it:") + printWarning(" %s%s" % (_new, + "" if _n_entries is None else + " (%d entries, this file has %d)" + % (_n_entries, len(entries)))) + printInfo(" diff %s %s" % (path, _new)) + printInfo(" ...and take what you want; this run uses YOUR file.") + print() + if getattr(args, "check_refs", False): + printHeader("Checking pinned git refs") + sys.exit(_check_stack_refs(entries)) + except ValueError as e: + printError(str(e)) + sys.exit(1) + stack_dir = os.path.dirname(path) + script_dir = os.path.join(TMP_DIR, "stack_files") + + avoid = [e["target"] for e in entries if e["kind"] == "avoid"] + entries = [e for e in entries if e["kind"] != "avoid"] + if getattr(args, "no_git", False): + # git and tar entries are the extras -- cloned tools, fonts. Skipping + # them finishes the book part of the desktop first, and matters while + # git itself is not built yet ("git: command not found"). + _n = len(entries) + _skipped_git = [e for e in entries if e["kind"] in ("git", "tar")] + entries = [e for e in entries if e["kind"] not in ("git", "tar")] + if not getattr(args, "status", False): + printInfo(" --no-git: skipping %d cloned/tarball entries\n" + % (_n - len(entries))) + # SOME CLONED ENTRIES ARE DEPENDENCIES, NOT EXTRAS. + # + # libnice and friends (1.13.69) are git entries that book + # packages LINK: gst-plugins-bad compiles webrtcbin only if + # libnice is present. Skipping them with --no-git builds those + # packages without WebRTC and says nothing -- and finding out + # later costs a webkitgtk rebuild. + _webrtc = [e["target"] for e in _skipped_git + if e["target"] in ("libnice", "libsrtp", "usrsctp", + "webrtc-audio-processing")] + if _webrtc: + printWarning(" NOTE: %s are skipped, and gst-plugins-bad / " + "webkitgtk link them." % ", ".join(_webrtc)) + printWarning(" Built without them there is no WebRTC (no " + "camera or screen sharing in the browser),") + printWarning(" and adding them later means recompiling both. " + " Drop --no-git to include them.") + if not getattr(args, "status", False): + printHeader("stack: %s (%d entries)" % (os.path.basename(path), len(entries))) + if avoid: + # AVOID WAS ONLY A MESSAGE. It printed "never built by this + # stack" and then passed nothing to the installer, so the + # packages were built anyway. It is the stack's exclusion list + # now, handed to every entry's resolve. + print(" never built by this stack: %s\n" % ", ".join(avoid)) + print(" file: %s\n" % path) + + import shutil as _sh + + def _git_installed(e): + """prog= on PATH, or have= (a file that only exists once it is + installed -- the signal for libraries, which put nothing on PATH).""" + have = e["opts"].get("have") + if have and os.path.exists(have): + return True + prog = e["opts"].get("prog", e["target"]) + return bool(prog and prog != "-" and _sh.which(prog)) + + # Entries that finished in an earlier run are recorded and SKIPPED, so a + # resume does not re-plan a 200-package dependency tree for a stage that + # is already done. --force ignores the cache. Recorded per stack file + # under $LFS_PKGUSR_DIR/progress/stacks/. + # RUN ONE ENTRY, OR A FEW. + # + # A stack is a list, and after the first pass the only thing anyone wants + # is to redo ONE line of it -- a config step that failed, a package to + # rebuild. --force re-ran all 56. --only names entries (repeatable, or + # comma-separated); it implies --force for those and skips the rest. + donekeys = set() if args.force else _stack_done_set(path) + # RUN JUST ONE PART, AND REALLY RUN IT. + # + # A stack is long and its entries fail one at a time; re-running the + # whole thing to retry `cfg mytools` is a poor answer, and --force + # re-runs everything. --only names entries: they are exempt from the + # cache (so a "done" one is rebuilt) and everything else is skipped. + # (--only was declared long ago and never implemented -- the help + # promised "repeatable or comma-separated"; both work now.) + _only = {x.strip() + for spec in (getattr(args, "only", None) or []) + for x in str(spec).split(",") if x.strip()} + if _only: + _known = {e["target"] for e in entries} + _unknown = sorted(_only - _known) + if _unknown: + printError("not in this stack: %s" % ", ".join(_unknown)) + printInfo(" entries: %s" % ", ".join(sorted(_known))) + # ...BUT IT MAY BE IN THE FILE WAITING BESIDE THIS ONE. + # + # The user asked for `--only mytools` against a stack whose + # newer copy has exactly that entry. "not in this stack" was + # true and useless: the answer was one flag away and the message + # did not say so. + _nf = path + ".new" + if os.path.isfile(_nf): + try: + _nnames = {x["target"] for x in _parse_stack_file(_nf)} + except (OSError, ValueError): + _nnames = set() + _there = sorted(set(_unknown) & _nnames) + if _there: + print() + printWarning("%s IS in the newer file beside this one:" + % ", ".join(_there)) + printInfo(" %s" % _nf) + printInfo(" Take it and run the same command:") + printInfo(" packagemanager stack %s --take-new %s" + % (args.name, " ".join( + "--only " + x for x in sorted(_only)))) + sys.exit(1) + donekeys = {k for k in donekeys + if k.split(" ", 1)[-1] not in _only} + + # ...and SKIP the rest. Exempting the named entries from the cache is + # only half of it: without this the other 55 were still planned and run. + if _only: + entries = [e for e in entries if e["target"] in _only] + printInfo("running only: %s" % ", ".join(e["target"] for e in entries)) + print() + rows = [] + for e in entries: + e["_key"] = "%s %s" % (e["kind"], e["target"]) + # (entries are already filtered to --only above; the old `only` + # variable was removed with the duplicate implementation, and this + # reference outlived it -- NameError on every stack run.) + if e["_key"] in donekeys: + rows.append((e, "done (stack cache)")) + if not getattr(args, "status", False): + print(" %-6s %-28s %s" % (e["kind"], e["target"], "done (stack cache)")) + continue + if e["kind"] in ("git", "tar"): + state = "installed" if _git_installed(e) else "to build" + elif e["kind"] in ("check", "cfg"): + sp = _stack_root_script(e, stack_dir) + state = "script" if os.path.isfile(sp) else "MISSING SCRIPT" + else: + # ASK THE BOOK. A dry run that says "book" without looking is a + # plan half-judged: sway and wlroots sat in a stack as book + # entries, and BLFS carries neither -- the first --run would have + # discovered it one llvm build too late. + # book_version cannot tell "anchor absent" from "no book on + # disk", so probe availability ONCE with an anchor every BLFS + # has; without a book, every entry is honestly unjudged. + if not hasattr(cmd_stack, "_book_ok"): + try: + cmd_stack._book_ok = book_version("dbus") is not None \ + or book_version("DBus") is not None + except Exception: + cmd_stack._book_ok = False + if not cmd_stack._book_ok: + state = "book (no book on disk -- unjudged)" + else: + try: + bv = book_version(e["target"]) + state = ("book %s" % bv) if bv else "NOT IN BOOK" + except Exception: + state = "book (lookup failed -- unjudged)" + # A book entry carrying url= FALLS BACK to git where the book + # does not know it -- one stack file serves a system whose book + # flavour has the package and one whose book does not. + if state == "NOT IN BOOK" and "url" in e["opts"]: + e["kind"] = "git" + state = ("installed" if _git_installed(e) + else "not in book -> git fallback") + rows.append((e, state)) + if not getattr(args, "status", False): + print(" %-6s %-28s %s" % (e["kind"], e["target"], state)) + + notinbook = [e["target"] for e, st in rows if st == "NOT IN BOOK"] + if notinbook: + printWarning("\nnot in the book: " + ", ".join(notinbook) + + "\n fix the entry (rename to the book's anchor, or make it" + "\n a git entry with build=/install= commands)") + if args.run: + printError("refusing to run a plan with unknown targets.") + sys.exit(1) + + missing = [e["target"] for e, st in rows if st == "MISSING SCRIPT"] + if missing: + printError("\nmissing root script(s) beside the stack: " + + ", ".join("%s.sh" % m for m in missing)) + sys.exit(1) + if getattr(args, "status", False): + # WHERE ARE WE IN THE TREE? + # + # Entry counts answered the wrong question: "1 of 39" says nothing + # about the hundreds of packages those entries pull in. Resolve each + # book entry's dependency order and report the PACKAGES -- installed, + # remaining, and what comes next -- using install_tag, the same + # judgement `search` and the install plan use, so all three agree. + seen, pkgs, skipped_by_avoid = set(), [], set() + for e, _st in rows: + if e["kind"] != "book": + continue + try: + # THE SAME GRAPH THE BUILD WILL USE. Asking without the + # entry's deps= showed 272 packages for a plan that builds + # 99 -- a status that does not match the run is worse than + # no status. + order, _missing = blfs_order_anchors( + e["target"], + no_recommended=(e["opts"].get("deps") == "required")) + except SystemExit: + continue + for anchor, nv, _kind, _parent in order: + # THE STATUS MUST NOT COUNT WHAT THE RUN WILL SKIP. The avoid + # list reaches the installer (1.12.78) and never reached this + # display, so it reported 273 packages for a run that skips + # qt6, okular, LibreOffice, CUPS and the rest by name. + if any(anchor.lower() == a.lower() for a in avoid): + skipped_by_avoid.add(anchor) + continue + if anchor not in seen: + seen.add(anchor) + pkgs.append((anchor, nv)) + done_pkgs = [(a, v) for a, v in pkgs if "not installed" not in install_tag(a, v)] + todo = [(a, v) for a, v in pkgs if (a, v) not in set(done_pkgs)] + done_n = sum(1 for _, st in rows if st == "done (stack cache)") + printHeader("stack %s" % args.name) + print(" entries : %d of %d done" % (done_n, len(rows))) + if pkgs: + print(" packages: %d of %d installed" % (len(done_pkgs), len(pkgs))) + if skipped_by_avoid: + print(" avoided : %d named in the stack (%s%s)" + % (len(skipped_by_avoid), + ", ".join(sorted(skipped_by_avoid)[:5]), + " ..." if len(skipped_by_avoid) > 5 else "")) + nxt_entry = next((e["target"] for e, st in rows if st != "done (stack cache)"), None) + print(" next : %s" % (nxt_entry or "nothing left to do")) + if todo: + print(" building next: " + ", ".join(a for a, _ in todo[:6]) + + (" ..." if len(todo) > 6 else "")) + if getattr(args, "full", False): + print() + for a, v in pkgs: + tag = install_tag(a, v) + print(" %-30s %-26s %s" % (a, v, tag)) + else: + print(" (full tree: --status --full)") + print(" state : %s" % _stack_state_file(path)) + return + + if not args.run: + printInfo("\nDry run -- nothing was built. Replay the plan with:\n" + " packagemanager stack %s --run" % args.name) + return + + print() + done = 0 + for e, state in rows: + kind, target = e["kind"], e["target"] + if state == "done (stack cache)": + done += 1 + continue + printHeader("stack (%d/%d): %s %s" % (done + 1, len(rows), kind, target)) + # A BOOK ENTRY CAN BECOME A CLONE. elogind is `book ... url=...`, + # and its kind only turns into "git" when the judging finds it absent + # from the book -- after the early --no-git filter has run. So the + # decision is taken here too, where the kind is final. + if getattr(args, "no_git", False) and kind in ("git", "tar"): + printInfo(" %s: skipped (--no-git)" % target) + continue + if kind == "book": + # NEVER drop a whole CLASS of dependencies. + # + # `required-only=1` (--ignore-recommended) was an attempt to keep + # texlive and LibreOffice out of a sway build. It removed far + # more: texlive's OWN dependencies -- Cairo, libpaper, the PATH + # page -- are all "recommended", so texlive was left with none, + # scheduled first, and built against a freetype that did not + # exist yet. A recommendation is still a dependency; the book + # knows why it is there. If a package is unwanted, NAME IT. + # deps=required -- the book's REQUIRED graph only. + # + # A first attempt at this (1.12.47) went badly and was reverted: + # texlive's own dependencies are all recommended, so dropping the + # class left it with none and it was scheduled FIRST. That was an + # ORDERING bug, fixed since (1.12.69 is a real topological sort), + # not an argument against choosing the graph you want. The + # recommendations worth keeping are named as entries of their own, + # which is what a stack file is for. + if not _install_via_self(target, args.yes, recursive=True, + required_only=(e["opts"].get("deps") == "required"), + ignore=",".join( + [x for x in [e["opts"].get("ignore")] if x] + + avoid) or None, + ignore_prereqs=getattr(args, "ignore_prereqs", + False)): + printError("stack stopped at %r (%d of %d done). Fix it and " + "re-run -- finished entries are skipped." + % (target, done, len(rows))) + sys.exit(1) + elif kind in ("git", "tar"): + if _git_installed(e) and not args.force: + printSuccess(" %s: already installed (--force rebuilds)" + % target) + else: + spath, _ = _stack_git_script(e, script_dir) + # THE SAME CHECKS AS A BOOK PACKAGE. + # + # A git entry never goes through the planner, so nothing had + # looked at what its commands need: elogind's clone died on + # `git: command not found` after the account was made, the + # source directory created and the phase started. The check + # is a dictionary lookup; run it here too. + # A GIT ENTRY CAN NAME ITS DEPENDENCIES. + # + # sway is built from git, so nothing in the book orders + # anything before it, and it stopped on + # ERROR: Dependency "json-c" not found + # json-c IS a BLFS package (JSON-C-0.18); it was simply never + # built, because a git entry had no way to say it needed one. + # `needs=` closes that: the named packages are installed -- + # through the same door as everything else -- before the entry + # runs. + for _need in [x.strip() for x in + (e["opts"].get("needs") or "").replace(",", " ").split() + if x.strip()]: + if gather_user(_need).state == "installed": + continue + printInfo(" %s needs %s -- installing it first" + % (target, _need)) + if not _install_via_self(_need, args.yes, recursive=True): + printError("stack stopped at %s %r: could not install" + " %s (%d of %d done)." + % (kind, target, _need, done, len(rows))) + sys.exit(1) + # MACHINE.CONF REACHES A GIT ENTRY TOO. + # + # apply_machine_opts was called only from resolve_script, the + # BOOK path. So the box could tell someone to put + # [webrtc-audio-processing] + # cpp_args = -include cstdint + # in machine.conf for a git stack entry, and nothing would + # read it -- the advice was correct and inert. A script is a + # script; the options belong to the package, not to where its + # commands came from. + apply_machine_opts(target, spath) + _MISSING_TOOLS.clear() + _check_script_prereqs(target, spath) + if _MISSING_TOOLS and not getattr(args, "ignore_prereqs", False): + # BUILD IT, DO NOT JUST NAME IT. + # + # Stopping with "packagemanager install git --run + # --recursive" asks the person to type what the stack is + # for. A plan does this itself (1.14.14: a missing + # provider joins the plan and it is rebuilt); a git entry + # is not a plan, so it stopped instead. If the book has + # the tool, install it and carry on -- that is what an + # unattended run is. + _fixable = sorted({_provider_of(t) for t in _MISSING_TOOLS + if _provider_of(t)}) + _stuck = sorted(t for t in _MISSING_TOOLS + if not _provider_of(t)) + for _prov in _fixable: + printInfo(" %s needs %s -- installing it first" + % (target, _prov)) + if not _install_via_self(_prov, args.yes, recursive=True): + printError("stack stopped at %s %r: could not" + " install %s (%d of %d done)." + % (kind, target, _prov, done, len(rows))) + sys.exit(1) + if _stuck: + _report_missing_tools() + printError("stack stopped at %s %r: it needs %s, which" + " no BLFS page provides (%d of %d done)." + % (kind, target, ", ".join(_stuck), + done, len(rows))) + printInfo(" Those come from LFS -- check they were" + " built and are on PATH.") + printInfo(" To start regardless: --ignore-prereqs") + sys.exit(1) + install_local(target, spath, True, args.yes, "install") + else: # check / cfg -- a root script through the one runner + sp = _stack_root_script(e, stack_dir) + rc = subprocess.run(["lfs-helper", "run-script", "--user", "root", + "--as-root", "--script", sp, + "--phase", "all", "--name", ""]).returncode + if rc != 0: + printError("stack stopped at %s %r (exit %d, %d of %d done)." + % (kind, target, rc, done, len(rows))) + sys.exit(1) + # DO NOT RECORD A GIT ENTRY THAT IS NOT THERE. + # + # The cache makes an entry disappear from every later run, so a false + # "done" is permanent. A git/tar entry declares what proves it + # installed (prog=/have=); if that is still absent after a build the + # tools said succeeded, say so and leave the entry unrecorded rather + # than hiding it forever. + if e["kind"] in ("git", "tar") and not _git_installed(e): + printWarning("%s finished, but %s is still missing -- NOT recording" + " it as done, so the next run tries again." + % (target, e["opts"].get("have") + or e["opts"].get("prog") or target)) + else: + _stack_mark_done(path, e["_key"]) + done += 1 + printSuccess("\nstack %s: all %d entries done." % (args.name, done)) # =========================================================================== # @@ -5128,6 +9501,139 @@ def cmd_list(args): # removing the files first, or an old manual userdel). Writes the paths to a # file so you can review and decide what to do (reassign, remove, or ignore). +def cmd_reclaim(args): + """Give a package back the files something else took from it. + + The ownership rule -- the installer owns what it installs -- is right for + packages that legitimately overlap, and wrong when one package installs a + BUNDLED COPY of another (1.14.116). By the time that is noticed, the + victim's files belong to the wrong account: /usr/bin/dvbv5-scan owned by + p_libdisplay-info, listed in p_v4l-utils's manifest. Only root can put + them back, and only the manifest knows which they are. + """ + if os.geteuid() != 0: + printError("reclaim changes ownership; run it as root.") + return 1 + rc = 0 + for name in args.packages: + acct = pkgusr_name(name) + manifest = os.path.join(pkgusr_home(acct), "pkg.lst") + try: + paths = [l.strip() for l in open(manifest) if l.strip()] + except OSError as e: + printError("%s: cannot read %s (%s)" % (acct, manifest, e)) + rc = 1 + continue + wrong = [] + for path in paths: + try: + st = os.lstat(path) + except OSError: + continue + try: + owner = pwd.getpwuid(st.st_uid).pw_name + except (KeyError, AttributeError): + owner = str(st.st_uid) + if owner != acct: + wrong.append((path, owner)) + if not wrong: + printSuccess("%s: every recorded file is already its own (%d checked)" + % (acct, len(paths))) + continue + printWarning("%s: %d recorded file(s) belong to someone else" + % (acct, len(wrong))) + for path, owner in wrong[:20]: + printInfo(" %-52s now %s" % (path, owner)) + if len(wrong) > 20: + printInfo(" ... and %d more" % (len(wrong) - 20)) + if not args.run: + printInfo(" Dry run -- re-run with --run to give them back.") + continue + back = 0 + for path, _owner in wrong: + try: + os.lchown(path, pwd.getpwnam(acct).pw_uid, + grp.getgrnam(acct).gr_gid) + back += 1 + except (OSError, KeyError) as e: + printError(" could not return %s (%s)" % (path, e)) + rc = 1 + printSuccess("%s: %d file(s) returned" % (acct, back)) + return rc + + +def cmd_which_package(args): + """Which package installed this path. + + The README has promised this command for as long as it has existed and + packagemanager never had it: + + packagemanager: error: argument <command>: invalid choice: 'which-package' + + `lfs-helper which-package` does exist, but it reads the BUILD manifests -- + the chapter 5-9 steps -- so for anything installed afterwards it answers + + Nothing recorded means it was installed before file tracking existed, + or outside these tools entirely. + + which is wrong twice over: these tools installed it, and they did record + it, in the package user's own pkg.lst. Two file registries, one question, + and each tool could only see its own. + """ + rc = 0 + for target in args.paths: + path = os.path.abspath(os.path.expanduser(target)) + printHeader(path) + + owner = None + try: + owner = pwd.getpwuid(os.stat(path).st_uid).pw_name + except (KeyError, OSError): + pass + + # The manifests: which package RECORDS having installed it. + hits = [] + for root in pkgusr_roots(): + try: + names = sorted(os.listdir(root)) + except OSError: + continue + for n in names: + lst = os.path.join(root, n, "pkg.lst") + if not os.path.isfile(lst): + continue + try: + with open(lst, errors="replace") as f: + for line in f: + if line.strip() == path: + hits.append(n) + break + except OSError: + continue + if hits: + for n in hits: + printSuccess(f" recorded by: {n} ({os.path.join(pkgusr_home(n), 'pkg.lst')})") + else: + printWarning(" no pkg.lst records this path") + rc = 1 + + # ...and who owns it now, which is the other half of the answer and + # can disagree with the manifest. + if owner: + printInfo(f" owned on disk by: {owner}") + if hits and owner not in hits: + printWarning(f" the owner is not among the packages that " + f"recorded it -- one of the two is wrong") + else: + printWarning(" not on disk, or its owner does not resolve") + + if not hits: + printInfo(" The build steps have their own manifests; for anything") + printInfo(" installed during chapters 5-9 ask:") + printInfo(f" lfs-helper which-package {path}") + return rc + + def cmd_files_with_broken_id(args): out_file = args.outfile or "/tmp/invalid_files.txt" printInfo("Scanning for orphaned files (uid/gid with no matching name); " @@ -5267,12 +9773,43 @@ def cmd_fix_install(args): mode = "g+w,o+t" if want_sticky else "g+w,o-t" def scan_wrong(): - printInfo("Scanning for directories owned by group 'install' " - "(runs 'find /', may take a while)...") - entries = owned_entries("install") - if entries is None: - printError("forall_direntries_from not found; cannot scan.") - return None + # EVERY DIRECTORY WITH GROUP install, FROM find(1). + # + # This asked forall_direntries_from -- the C helper from the + # package-users hint -- and on an older system that helper is an + # older build: it may stop at mount points or list by user only, and + # on a machine in daily use the install group is on hundreds of + # directories, not the twenty-two seed ones. "Wherever the install + # group is set, the directory needs g+w and the sticky bit" is a + # one-line find, so ask find, and prune only the virtual filesystems. + roots = [r for r in (getattr(args, "roots", None) or ["/"])] + # /snapshots holds btrfs/rsync copies of the whole tree: every + # install directory again, once per snapshot. Fixing those is + # pointless (a snapshot is read-only, or a record) and slow (it + # multiplies the scan by the snapshot count), so it is pruned by + # default; --exclude adds more, and --no-default-excludes puts + # /snapshots back if someone really wants it. + prune = ["/proc", "/sys", "/dev", "/run", "/tmp"] + if not getattr(args, "no_default_excludes", False): + prune += ["/snapshots", "/.snapshots"] + prune += [x.rstrip("/") or "/" for x in (getattr(args, "exclude", None) or [])] + printInfo("Scanning %s for directories owned by group 'install'" + " (pruning %s) ..." % (", ".join(roots), ", ".join(prune))) + cmd = ["find"] + roots + ["("] + \ + sum([["-path", x, "-o"] for x in prune], [])[:-1] + \ + [")", "-prune", "-o", "-type", "d", "-group", "install", "-print0"] + try: + r = subprocess.run(cmd, capture_output=True, timeout=1800) + entries = [e.decode("utf-8", "replace") + for e in r.stdout.split(b"\0") if e] + except (OSError, subprocess.SubprocessError) as e: + printError("find failed (%s); falling back to forall_direntries_from" % e) + entries = owned_entries("install") + if entries is None: + printError("forall_direntries_from not found either; cannot scan.") + return None + printInfo(" %d director%s group 'install'" + % (len(entries), "y carries" if len(entries) == 1 else "ies carry")) wrong = [] for path in entries: try: @@ -5480,39 +10017,69 @@ def ensure_wget_workaround(): " This is undone automatically as soon as certificates exist.\n") +def _current_blfs_book(): + """The book blfs would use right now ('' when unknown). + + Asks `blfs books --brief` -- default + cached list, NO network -- and + reads the Default line. The old parser took the first [bracketed] + filename from the full listing, which is the alphabetically-first cached + book: right only while the default happened to sort first, and 12.4 sorts + before 13.0. An older blfs without --brief gets the plain listing, whose + Default line reads the same.""" + rc, out, _e = run_blfs(["books", "--brief"]) + if rc != 0: + rc, out, _e = run_blfs(["books"]) + if rc != 0: + return "" + for ln in out.splitlines(): + if ln.startswith("Default:"): + return ln.split(":", 1)[1].strip() + return "" + + def _warn_init_mismatch(): """The BLFS book's init flavour should match the system's. A systemd book emits `systemctl` calls in its install scripts; on a SysV system those fail with "command not found" in the middle of an otherwise - successful install, and every affected package has to be edited by hand. - The two books differ in more than that, so it is worth saying once.""" + successful install. Since the generated scripts now shim systemctl into + a visible no-op where the binary is absent, USING the other flavour is a + reasonable choice -- the SysV book is no longer maintained -- so this is + asked ONCE and the answer is recorded. A different mismatch (the pair + changing direction, or a different book) asks again.""" have_systemd = (os.path.isdir("/usr/lib/systemd/system") or bool(shutil.which("systemctl"))) - rc, out, _e = run_blfs(["books"]) - if rc != 0: - return - book = "" - for ln in out.splitlines(): - if "[" in ln and "]" in ln: - book = ln.split("[", 1)[1].split("]", 1)[0] - break + book = _current_blfs_book() if not book: return book_systemd = "systemd" in book.lower() if book_systemd == have_systemd: return + pair = "%s-on-%s" % ("systemd" if book_systemd else "sysv", + "systemd" if have_systemd else "sysv") + if load_config().get("init_mismatch_ok") == pair: + return printWarning("\nThe BLFS book in use does not match this system's init:") printInfo(f" book : {book} ({'systemd' if book_systemd else 'SysV'})") printInfo(f" system : {'systemd' if have_systemd else 'SysV'}") printInfo("") printInfo("Install scripts from the wrong book carry commands this system") - printInfo("does not have -- `systemctl: command not found` mid-install is") - printInfo("the usual symptom. Switch with:") + printInfo("does not have. Generated scripts turn a missing systemctl into") + printInfo("a visible skip, so using this book anyway is workable. Or") + printInfo("switch with:") printInfo("") printInfo(" blfs fetch %s" % ("stable" if have_systemd is False else "stable-systemd")) printInfo("") + if sys.stdin.isatty(): + if ask_yes_no("Keep using this book, and stop this warning?"): + cfg = load_config() + cfg["init_mismatch_ok"] = pair + if save_config(cfg): + printInfo(" recorded -- undo with: packagemanager config " + "(remove init_mismatch_ok)") + else: + printWarning(" could not save the answer; it will ask again.") def _hint_generate_ca_bundle(): @@ -5665,6 +10232,9 @@ def main(): p.add_argument("--regenerate", action="store_true", help="rebuild scripts from the book, ignoring any edited " "script in the package user's home") + p.add_argument("--ignore-prereqs", action="store_true", + help="start even when a build tool or variable the plan's " + "scripts need is missing (it will fail there)") p.add_argument("--run", action="store_true", help="actually build/install (default is a dry-run plan)") p.add_argument("--yes", action="store_true", @@ -5681,6 +10251,9 @@ def main(): "skipped; use 'verify' to recheck all.") p.add_argument("packages", nargs="*", help="packages to update (with deps); default: all installed") + p.add_argument("--toolchain", action="store_true", + help="also rebuild toolchain packages (glibc, binutils, " + "gcc) from the LFS book -- risky on a running system") p.add_argument("--recursive", action="store_true", help="also update the named package(s)' dependencies " "(default: just the named package(s))") @@ -5697,6 +10270,9 @@ def main(): p.add_argument("--regenerate", action="store_true", help="rebuild scripts from the book, ignoring any edited " "script in the package user's home") + p.add_argument("--ignore-prereqs", action="store_true", + help="start even when a build tool or variable the plan's " + "scripts need is missing (it will fail there)") p.add_argument("--clean", action="store_true", help="before installing each new version, remove the old " "files (except configs under /etc,/var,... and the source " @@ -5744,13 +10320,27 @@ def main(): p = sub.add_parser("reload-pkg-list", help="regenerate /usr/src/<name>/pkg.lst") p.add_argument("packages", nargs="+") + p.add_argument("--background", action="store_true", + help="run detached and return at once (the old default)") p.add_argument("--foreground", action="store_true", help="wait for it instead of running as a disowned task") p.set_defaults(func=cmd_reload_pkg_list) + # `bootstrap`, not `setup`. + # + # Every other command here is a verb naming what it does to PACKAGES -- + # install, update, remove, verify. "setup" named neither the thing nor the + # action, and the vocabulary already used the word for something else: + # `packagemanager user setup` applies the shared-account treatment to an + # existing application user. Two unrelated commands, one word. + # + # `setup` stays as a hidden alias: an older installed `lfs` calls + # `packagemanager setup --sources` from get-sources, and a rename that + # breaks the tool that downloads the sources is a rename that costs a + # reboot to discover. p = sub.add_parser( - "setup", - help="bootstrap the tooling on a freshly booted system", + "bootstrap", aliases=["setup"], + help="get a freshly booted system to where blfs and packagemanager work", description=_SETUP_HELP, formatter_class=argparse.RawDescriptionHelpFormatter) p.add_argument("--sources", action="store_true", @@ -5766,10 +10356,68 @@ def main(): help="answer yes to the questions the wget build asks") p.set_defaults(func=cmd_setup) + p = sub.add_parser( + "env", + help="build-time variables the book expects to be exported") + p.add_argument("--set", action="append", metavar="NAME=VALUE", + help="set one (repeatable)") + p.add_argument("--ask", action="store_true", + help="ask for every variable a generated script needs") + p.add_argument("--refresh", action="store_true", + help="drop scripts from an older blfs so the current " + "questions can be seen") + p.set_defaults(func=cmd_env) + + p = sub.add_parser( + "stack", + help="replay a stack file: one command, one whole environment") + p.add_argument("name", help="stack name or path (e.g. sway, ./sway.stack)") + p.add_argument("--run", action="store_true", + help="actually build (default: print the plan)") + p.add_argument("--yes", "-y", action="store_true", + help="answer yes to every prompt (PM_YES for the installs)") + p.add_argument("--force", action="store_true", + help="rebuild git entries even when their program is on PATH") + p.add_argument("--status", action="store_true", + help="how far this stack has got, and what is next") + p.add_argument("--no-git", action="store_true", + help="skip git and tarball entries (the extras); useful " + "before git itself is built") + p.add_argument("--full", action="store_true", + help="with --status: every package in the tree and its state") + p.add_argument("--ignore-prereqs", action="store_true", + help="start entries even when a build tool or variable " + "their scripts need is missing") + p.add_argument("--only", action="append", metavar="ENTRY", + help="run just these entries (repeatable or comma-separated); " + "they are rebuilt even if the cache says done") + p.add_argument("--take-new", action="store_true", + help="replace this stack file with the shipped .new one " + "(yours is kept as .bak) and run that") + p.add_argument("--check-refs", action="store_true", + help="ask each git entry's remote whether its ref= still " + "exists, and stop -- builds nothing") + p.set_defaults(func=cmd_stack) + + p = sub.add_parser("reclaim", + help="give a package back files another package took " + "from it (ownership only; nothing is rebuilt)") + p.add_argument("packages", nargs="+") + p.add_argument("--run", action="store_true", + help="actually change ownership (default: dry run)") + p.set_defaults(func=cmd_reclaim) + + p = sub.add_parser("which-package", aliases=["which-pkg"], + help="which package installed a path (reads pkg.lst)") + p.add_argument("paths", nargs="+", metavar="PATH") + p.set_defaults(func=cmd_which_package) + p = sub.add_parser("pip", help="install or update a Python module as a " "package user (not as root)") - p.add_argument("action", choices=["install", "update", "upgrade"], - help="install or update") + p.add_argument("action", choices=["install", "update", "upgrade", "record"], + help="install, update, or record (write the manifest and " + "version for a module installed before 1.14.37, which " + "wrote no record at all)") p.add_argument("modules", nargs="*", help="module names, e.g. beautifulsoup4") p.add_argument("--find-links", metavar="DIR", help="install from local wheels in DIR instead of the network " @@ -5777,7 +10425,8 @@ def main(): p.add_argument("--no-deps", action="store_true", help="do not let pip pull dependencies -- name them yourself, " "in order (what `setup` does, offline)") - p.set_defaults(func=lambda a: cmd_pip( + p.set_defaults(func=lambda a: cmd_pip_record(a.modules) if a.action == "record" + else cmd_pip( argparse.Namespace(modules=a.modules, upgrade=(a.action in ("update", "upgrade")), no_deps=a.no_deps, @@ -5789,10 +10438,12 @@ def main(): p.set_defaults(func=cmd_add_user) p = sub.add_parser("script", help="run one phase of a package's install " - "script as the package user (unpack/build/install/...)") + "script as the package user (unpack/build/install/...), " + "or 'fetch' to just get the script for editing") p.add_argument("phase", choices=["all", "unpack", "build", "test", "install", - "configure", "update"], - help="which phase to run") + "configure", "update", "fetch"], + help="which phase to run ('fetch' gets the script, runs " + "nothing)") p.add_argument("package") p.add_argument("--regenerate", action="store_true", help="regenerate the script from the book before running") @@ -5820,6 +10471,20 @@ def main(): p.set_defaults(func=cmd_paths) p = sub.add_parser("config", help="show or change the prefixes / main user") + p.add_argument("--base-dir", + help="where account homes live (default /usr/src)") + p.add_argument("--pkgusr-subdir", + help="folder inside the base for PACKAGE users " + "(default 'pkgusr'; empty puts them in the base)") + p.add_argument("--cfguser-subdir", + help="folder inside the base for config users (default 'cfg')") + p.add_argument("--pkgusr-prefix", + help="prefix for PACKAGE users (empty string for none); " + "refuses if accounts already use the current one") + p.add_argument("--cfguser-prefix", + help="prefix for config users") + p.add_argument("--force", action="store_true", + help="change a prefix even though accounts already use it") p.add_argument("--collector-prefix", help="prefix for shared install-dir groups") p.add_argument("--user-prefix", help="prefix for shared users") p.add_argument("--main-user", help="your human login account (e.g. n76310)") @@ -5913,6 +10578,13 @@ def main(): help="check/fix permissions on all group-'install' dirs") p.add_argument("--run", action="store_true", help="apply the fixes (default lists what would change)") + p.add_argument("roots", nargs="*", + help="directories to scan (default: / with /proc, /sys, " + "/dev, /run, /tmp and /snapshots pruned)") + p.add_argument("--exclude", action="append", metavar="DIR", + help="prune this directory too (repeatable)") + p.add_argument("--no-default-excludes", action="store_true", + help="scan /snapshots and /.snapshots as well") p.add_argument("--sticky", action="store_true", help="also set the sticky bit (default: group-writable, no sticky)") p.add_argument("--yes", action="store_true", diff --git a/stacks/kernel-sway.sh b/stacks/kernel-sway.sh new file mode 100755 index 0000000..2c61ca4 --- /dev/null +++ b/stacks/kernel-sway.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# kernel-sway.sh -- VERIFY the running kernel carries what sway + webkit need. +# Builds nothing; a missing feature fails the stack with its name. +fail=0 +need() { # need <config-option> <why> + local cfg="" opt="$1" why="$2" + for c in /proc/config.gz "/boot/config-$(uname -r)" /boot/config; do + [ -r "$c" ] && cfg="$c" && break + done + if [ -z "$cfg" ]; then + echo "!! cannot read a kernel config (/proc/config.gz or /boot/config*)" + echo " enable CONFIG_IKCONFIG_PROC, or place the config in /boot" + exit 1 + fi + case "$cfg" in + *.gz) zgrep -qE "^$opt=(y|m)" "$cfg" ;; + *) grep -qE "^$opt=(y|m)" "$cfg" ;; + esac || { echo "!! $opt is not set -- $why"; fail=1; } +} +need CONFIG_DRM "no DRM: no wayland compositor" +need CONFIG_DRM_AMDGPU "the amdgpu KMS driver (radeonsi machines)" +need CONFIG_INPUT_EVDEV "libinput reads /dev/input/event*" +need CONFIG_SND "sound core" +need CONFIG_SECCOMP "webkit's sandbox" +need CONFIG_USER_NS "webkit's sandbox (bubblewrap)" +need CONFIG_TMPFS "XDG_RUNTIME_DIR lives on tmpfs" +# runtime, not config: the render node must exist +[ -e /dev/dri/renderD128 ] || { echo "!! no /dev/dri/renderD* render node"; fail=1; } +[ "$fail" = 0 ] && echo "# kernel: everything sway + webkit need is present" +exit $fail diff --git a/stacks/machine.conf b/stacks/machine.conf new file mode 100644 index 0000000..91052e4 --- /dev/null +++ b/stacks/machine.conf @@ -0,0 +1,33 @@ +# machine.conf -- THIS machine's build options, merged into generated +# scripts' `meson setup` lines as -Dkey=value the moment the script is born. +# The stack file stays portable; this file changes per computer. + +[mesa] +platforms = x11,wayland +gallium-drivers = radeonsi +vulkan-drivers = amd +video-codecs = all +glvnd = disabled +libunwind = disabled +llvm = enabled +valgrind = disabled +egl = enabled +opengl = true + +# Per-package build options. Same mechanism as [mesa]: every key is merged +# into that package's generated `meson setup` line as -Dkey=value. + +# iptables: the book configures --disable-nftables, which builds the legacy +# backend only. The nft backend needs this flag and libnftnl (a tar entry in +# network.stack, since BLFS has no page for it). +[iptables] +configure_args = --enable-nftables + +# libdisplay-info builds v4l-utils as a SUBPROJECT for its edid-decode +# comparison tests: it clones the repo and installs libdvbv5, +# libv4lconvert and a dozen locale files -- every one of them already owned +# by p_v4l-utils. The book passes --wrap-mode=nofallback, which only blocks +# fallbacks for dependency() lookups; an explicit subproject() still +# downloads. nodownload blocks that. +[libdisplay-info] +wrap_mode = nodownload diff --git a/stacks/mytools.sh b/stacks/mytools.sh new file mode 100755 index 0000000..1891337 --- /dev/null +++ b/stacks/mytools.sh @@ -0,0 +1,108 @@ +#!/bin/bash +# mytools.sh -- install everything of your own under ONE package user. +# +# The work is in stacks/mytools/install_mytools, an ordinary package-user +# script: p_mytools owns the files, pkg.lst records them, `verify` lists it, +# and `packagemanager remove mytools` takes it all away again. This step +# only does the two things a package user may not do itself: create the +# account, and set the setuid bit on the seatd wrapper. +set -u + +here="$(cd "$(dirname "$0")" && pwd)" +script="$here/mytools/install_mytools" +[ -f "$script" ] || { echo "!! missing $script"; exit 1; } + +u="${LFS_DESKTOP_USER:-}" +[ -z "$u" ] && [ -r /etc/pkgusr/desktop-user ] && u="$(head -n1 /etc/pkgusr/desktop-user)" +if [ -z "$u" ] || ! id "$u" >/dev/null 2>&1; then + echo "!! no desktop user. seatd-user is compiled for ONE username." + echo " useradd -m -G audio,video,input,wheel <name> && passwd <name>" + echo " LFS_DESKTOP_USER=<name> packagemanager stack services --run" + exit 1 +fi +printf '%s\n' "$u" > /etc/pkgusr/desktop-user +echo "# desktop user: $u" + +# 1. the account, named after the person: p_<user>. Their software, their +# name -- `verify`, pkg.lst and `packagemanager remove` all say it. +pkg="${MYTOOLS_NAME:-$u}" +acct="p_$pkg" +if ! id "$acct" >/dev/null 2>&1; then + packagemanager add-user "$pkg" || { + echo "!! could not create the package user '$pkg'"; exit 1; } +fi + +# 2. CLEAR THE WAY FOR THE WRAPPER. +# +# /usr/bin is sticky (drwxrwxr-t), so only a file's owner may replace it -- +# and once step 3 below has made seatd-<user> root-owned and setuid, the +# package user can no longer overwrite it: +# install: cannot remove '/usr/bin/seatd-n76310': Operation not permitted +# Root can, and root is running this. Remove it before the build so the +# package user writes a fresh one, and step 3 puts the bit back. +w="/usr/bin/seatd-$u" +if [ -e "$w" ] && [ "$(stat -c %U "$w")" != "$acct" ]; then + rm -f "$w" && echo "# removed the old $w (root-owned; the build replaces it)" +fi + +# 3. THE HOME MUST BE WHERE EVERY TOOL LOOKS. +# +# add_package_user (the hint's helper) puts a new account's home at +# /usr/src/<acct>, without the pkgusr/ component, and pm-install then works +# there while `verify` scans /usr/src/pkgusr -- so the package installs and +# is reported "not installed", with its pkg.lst in a directory nothing reads. +# fix-home is the repair; run it before anything else touches the account. +lfs-helper fix-home "$acct" --run >/dev/null 2>&1 || true +canon="/usr/src/pkgusr/$acct" +real="$(getent passwd "$acct" | cut -d: -f6)" +if [ "$real" != "$canon" ]; then + echo "!! $acct's home is $real, but every tool looks in $canon." + echo " Repair it, then re-run this step:" + echo " lfs-helper fix-home $acct --run" + exit 1 +fi + +# 4. REFRESH THE HOME COPY. +# +# pm-install prefers the script already in the package home (1.14.21): an +# edit there survives, which is right for a book page someone tweaked. But +# THIS script is shipped and updated by releases, and the home copy is a +# stale duplicate -- the run kept using an old one and the fixes never took +# effect. Root owns this decision: copy when they differ. +home_script="$(getent passwd "$acct" | cut -d: -f6)/$(basename "$script")" +if [ -f "$home_script" ] && ! cmp -s "$script" "$home_script"; then + install -m 755 -o "$acct" -g "$acct" "$script" "$home_script" \ + && echo "# refreshed $home_script from the shipped copy" +fi + +# 5. build and install as that user, through the one runner +DESKTOP_USER="$u" MYTOOLS_NAME="$pkg" lfs-helper pm-install "$acct" "$script" || exit 1 + +# 6. the setuid bit: root's job, refused to the package user on purpose +if [ -f "$w" ]; then + chown root:"$u" "$w" && chmod 4750 "$w" \ + && echo "# $w -> $(stat -c '%a %U:%G' "$w")" +else + echo "!! $w was not installed -- see the build output above" + exit 1 +fi + +# 7. `sway` should start the wrapper +home="$(getent passwd "$u" | cut -d: -f6)" +if ! grep -qs "alias sway=" "$home/.bashrc" 2>/dev/null; then + printf "\n# start sway through the wrapper\nalias sway='sway_start'\n" >> "$home/.bashrc" + chown "$u:$u" "$home/.bashrc" + echo "# added alias sway='sway_start' to $home/.bashrc" +fi +# 8. the manifest must exist where verify reads it, or the package is +# installed and invisible +if [ ! -s "$canon/pkg.lst" ]; then + echo "!! $canon/pkg.lst is missing or empty -- the files were installed" + echo " but nothing records them, so verify will say 'not installed'." + echo " packagemanager reload-pkg-list $pkg" + exit 1 +fi +echo "# $(grep -c . "$canon/pkg.lst") file(s) recorded in $canon/pkg.lst" +echo "# done -- everything of yours is owned by $acct:" +echo "# packagemanager verify | grep $pkg" +echo "# cat /usr/src/pkgusr/$acct/pkg.lst" diff --git a/stacks/mytools/install_mytools b/stacks/mytools/install_mytools new file mode 100755 index 0000000..420e9f2 --- /dev/null +++ b/stacks/mytools/install_mytools @@ -0,0 +1,116 @@ +#!/bin/bash +############################################ +### mytools -- everything of your own, under one package user +### +### One account (p_mytools) owns every script and program you wrote or +### packaged yourself, so `packagemanager verify` lists them, pkg.lst records +### every file, and removing them is one command. Add a line to SOURCES and +### re-run; nothing else changes. +### +### phases: all | unpack | build | install (see lfs-phases) + +# Named after the person whose software this is: the account is p_<user>, +# so `verify`, pkg.lst and `packagemanager remove` all speak their name. +name="${MYTOOLS_NAME:-${DESKTOP_USER:-$(cat /etc/pkgusr/desktop-user 2>/dev/null || echo mytools)}}" +name_version="$name-1" +url='' +link='' +pkg='' +md5_sum='' +info='Personal scripts and programs: the seatd-user wrapper, sway_start, the wifi script, and the sway helper programs.' +installed_content=('installed_programs') +installed_programs='wifi, sway_start' + +# --------------------------------------------------------------------------- +# SOURCES -- one line per thing. Fields, separated by '|': +# kind raw = a single file, installed as-is +# tar = a tarball; unpacked, then BUILD runs in its directory +# git = a repository; cloned, then BUILD runs in it +# name what it is called once installed +# url where it comes from (a file:// path works offline) +# dest where it goes (a directory, or empty for /usr/bin) +# build shell run in the source dir; empty for raw files +# --------------------------------------------------------------------------- +# SINGLE-quoted, always: a double-quoted element collapses the escapes before +# eval ever sees them. The build command is the author's own, verbatim: +# u=n76310; gcc seatd-user.c -DSEATD_USER="$u" -o seatd-$u +# which reaches gcc as -DSEATD_USER=n76310 -- the program stringifies it +# itself. (An earlier version passed \"n76310\"; that is a different macro +# and only one of the two can be right, so copy what works by hand.) +SOURCES=( +'raw|wifi|https://git.christianimmanuel.de/scripts/raw/Administration/wifi|/usr/bin|' +'raw|sway_start|https://git.christianimmanuel.de/scripts/raw/Administration/sway_start|/usr/bin|' +'tar|seatd-user|https://git.christianimmanuel.de/sway/seatd-user/archive/seatd-user.tar.gz|/usr/bin|gcc seatd-user.c -DSEATD_USER=$DESKTOP_USER -o $OUT/seatd-$DESKTOP_USER' +# The sway helpers. They are separate package users today; move them here by +# uncommenting, then `packagemanager remove swas swov swbr minibrowser`. +#'git|swas|https://git.christianimmanuel.de/sway/SwAS|/usr/bin|make' +#'git|swov|https://git.christianimmanuel.de/sway/SWOv|/usr/bin|make' +#'git|swbr|https://git.christianimmanuel.de/sway/SwBr|/usr/bin|make' +#'git|browser|https://git.christianimmanuel.de/web-tools/Browser|/usr/bin|make' +) + +# the user seatd-user is compiled for; the desktop-user step records it +DESKTOP_USER="${DESKTOP_USER:-$(cat /etc/pkgusr/desktop-user 2>/dev/null || true)}" + +_srcdir() { printf '%s/src/%s' "$HOME" "$1"; } + +build_pkg() { + [ -n "$DESKTOP_USER" ] || { + echo "!! DESKTOP_USER is not set and /etc/pkgusr/desktop-user is empty." + echo " seatd-user is compiled for ONE username; name it:" + echo " echo <user> > /etc/pkgusr/desktop-user" + return 1 + } + mkdir -p "$HOME/src" "$HOME/out" + local line kind nm url dest build d OUT + OUT="$HOME/out" + for line in "${SOURCES[@]}"; do + IFS='|' read -r kind nm url dest build <<< "$line" + [ -n "$nm" ] || continue + d="$(_srcdir "$nm")" + echo "== $nm ($kind)" + case "$kind" in + raw) + wget -q -O "$OUT/$nm" "$url" || { echo "!! fetch failed: $url"; return 1; } + chmod 755 "$OUT/$nm" ;; + tar) + rm -rf "$d"; mkdir -p "$d" + wget -q -O "$d/pkg.tar.gz" "$url" || { echo "!! fetch failed: $url"; return 1; } + ( cd "$d" && tar xf pkg.tar.gz ) || return 1 + ( cd "$(dirname "$(find "$d" -name '*.c' -print -quit)")" \ + && eval "$build" ) || return 1 ;; + git) + if [ -d "$d/.git" ]; then ( cd "$d" && git pull --ff-only ) || return 1 + else git clone --depth 1 "$url" "$d" || return 1; fi + ( cd "$d" && eval "$build" ) || return 1 + [ -x "$d/$nm" ] && cp -f "$d/$nm" "$OUT/$nm" ;; + esac + done +} + +install_pkg() { + local line kind nm url dest build f + for line in "${SOURCES[@]}"; do + IFS='|' read -r kind nm url dest build <<< "$line" + [ -n "$nm" ] || continue + dest="${dest:-/usr/bin}" + install -v -d -m755 "$dest" + # ONE ENTRY, ITS OWN ARTIFACT. This installed both $nm AND the + # seatd wrapper on every pass, so the wrapper was written three + # times -- harmless until the third try hit a file it no longer + # owned. seatd-user's product is named after the user, not the + # entry, which is why it needs the special case. + case "$nm" in + seatd-user) f="$HOME/out/seatd-$DESKTOP_USER" ;; + *) f="$HOME/out/$nm" ;; + esac + [ -f "$f" ] || { echo "!! $nm: nothing was built at $f"; return 1; } + install -v -m755 "$f" "$dest/$(basename "$f")" + done + echo + echo "## /usr/bin/seatd-$DESKTOP_USER must be setuid root, group $DESKTOP_USER." + echo "## A package user cannot set that bit -- the wrapper records the" + echo "## request and the build reports it; the mytools step applies it." +} + +. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@" diff --git a/stacks/network.stack b/stacks/network.stack new file mode 100644 index 0000000..7666a55 --- /dev/null +++ b/stacks/network.stack @@ -0,0 +1,42 @@ +# network.stack -- wired and wireless networking, for a SysV system. +# +# The sway stack builds a desktop; nothing in it brings up a network. This is +# what `wifi` (wpa_cli / wpa_supplicant / ifup) needs, plus a firewall. +# +# packagemanager stack network --check-refs # verify before building +# packagemanager stack network --yes --run +# +# Every entry here is a BLFS package, checked against the book. + +book libnl # netlink, what wpa_supplicant talks to the kernel with +book wpa_supplicant # wpa_supplicant, wpa_cli, wpa_passphrase +book wireless_tools # iwconfig/iwlist, if you want them + +book libmnl # minimal netlink library; libnftnl needs it + +# libnftnl is NOT a BLFS package -- the nf_tables netlink API that the nft +# backend of iptables is built on. Version 1.3.2 is what netfilter.org's +# own news page lists as current (released 2026-08-31); check the tarball +# against the .sha256 beside it before trusting this line, and bump it when +# upstream moves -- `--check-refs` asks git remotes and cannot see a tar url. +tar libnftnl url=https://www.netfilter.org/pub/libnftnl/libnftnl-1.3.2.tar.xz pkg=libnftnl-1.3.2.tar.xz have=/usr/lib/libnftnl.so build="./configure --prefix=/usr --disable-static; make" install="make install" + +book iptables # nft backend, via machine.conf (see below) + +book net-tools # ifconfig, route, netstat + +# --------------------------------------------------------------------------- +# iptables-nft +# +# The book's iptables page configures --disable-nftables, so the flag has to +# come from machine.conf. The shipped stacks/machine.conf carries it: +# +# [iptables] +# configure_args = --enable-nftables +# +# Your /etc/pkgusr/stacks/machine.conf is never overwritten (1.14.58), so if +# it predates this, take that section from machine.conf.new beside it. +# +# The legacy backend is being retired upstream; iptables-nft keeps the +# familiar rule syntax on top of nf_tables. +# --------------------------------------------------------------------------- diff --git a/stacks/package-fixes.conf b/stacks/package-fixes.conf new file mode 100644 index 0000000..fd741cb --- /dev/null +++ b/stacks/package-fixes.conf @@ -0,0 +1,60 @@ +# Workarounds the TOOLS ship, refreshed with every release. +# +# This file is REPLACED by `lfs build-system run` / `sync-tools`; put your own +# choices in machine.conf instead, which is never overwritten and wins where +# both mention the same key. + +[cairo] +# Cairo 1.18.4 defines its own static ctime_r when it believes glibc lacks +# it, then collides with the real one. +c_args = -DHAVE_CTIME_R + +# NOTE: libassuan and libgpg-error used to carry --disable-doc here. That +# flag does not exist in those packages (autotools accepts unknown +# --disable-* silently), and the real cause was the book's own +# `make -C doc pdf ps` line, which the generator now comments out. + + + +[gpgme] +configure_args = --disable-gpg-test + +[vala] +# valadoc needs Graphviz, which this desktop avoids. The book says so +# itself: "--disable-valadoc: This option is required if Graphviz is not +# installed." Without it: +# configure: error: Package requirements (libgvc >= 2.16) were not met +configure_args = --disable-valadoc + +# Dependencies the BOOK does not record but the build needs. `needs` is +# added to that package's REQUIRED list, so the ordering places it. +[libass] +needs = harfbuzz + +[ffmpeg] +# ffmpeg only RECOMMENDS libass, and both sit inside one big cycle in the +# book's graph -- so the cycle-breaking was free to drop that edge and put +# ffmpeg first: ERROR: libass >= 0.11.0 not found using pkg-config +# ffmpeg's configure hard-requires it, so say so: a required edge is never +# dropped. +needs = libass + +[glycin] +# glycin only RECOMMENDS librsvg and its meson hard-requires it: +# ERROR: Dependency "librsvg-2.0" not found +# Same shape as ffmpeg/libass -- both inside one cycle, so the recommended +# edge was free to be dropped (glycin 197, librsvg 206). +needs = librsvg + +[vulkan-loader] +# its CMake scripts shell out to git to stamp a version: +# FileNotFoundError: [Errno 2] No such file or directory: 'git' +needs = git + +[mesa] +# glslang: the Vulkan drivers machine.conf turns on need glslangValidator. +# rust-bindgen: mesa runs it as soon as it FINDS a rust compiler, and since +# rust is built here (for librsvg) that is always. The book's mesa page +# lists neither. +# libclc: radeonsi (machine.conf) builds the OpenCL bits, which need it. +needs = glslang rust-bindgen libclc diff --git a/stacks/services.stack b/stacks/services.stack new file mode 100644 index 0000000..c2ba73c --- /dev/null +++ b/stacks/services.stack @@ -0,0 +1,63 @@ +# services.stack -- the boot-time services a SysV desktop needs. +# +# /etc/init.d after an LFS build holds only LFS's own scripts; nothing starts +# dbus, alsa, bluetooth or a cron daemon. Those scripts live in +# blfs-bootscripts, which is a separate tarball, not a BLFS package -- which +# is why a package's own page says "make install-<service>" and the generated +# script comments it out (1.14.20): the target is in THIS package, not that +# one. +# +# packagemanager stack services --yes --run +# +# Run it after sway.stack and network.stack: a boot script for a service that +# is not installed is harmless but pointless. + +# Everything of your own, under ONE package user (p_mytools): the seatd-user +# wrapper, sway_start, the wifi script, and -- when you uncomment them in +# stacks/mytools/install_mytools -- swas, swov, swbr and browser. Add a line +# to SOURCES there and re-run; the package user owns the files, pkg.lst +# records them, and `packagemanager remove mytools` removes the lot. +# +# seatd-user replaces a seatd boot service: a setuid program compiled for ONE +# username, so there is no daemon and no `seat` group. +cfg mytools + +# (The kernel is NOT a stack entry: it is `lfs-kernel`, its own tool, so the +# config can be edited before anything is built. See /etc/pkgusr/kernel.conf +# and `lfs-kernel --help`.) + +# bash-completion is NOT a BLFS package -- it comes from upstream. 2.18.0 is +# what the remote reports as newest (checked with git ls-remote); `--check-refs` +# will confirm it before any build. +git bash-completion url=https://github.com/scop/bash-completion ref=2.18.0 \ + have=/usr/share/bash-completion/bash_completion \ + build="autoreconf -i && ./configure --prefix=/usr --sysconfdir=/etc" \ + install="make install" + +# coloured ls, the BLFS /etc/bashrc, and `su <tab>` over the package users. +# The Bash Shell Startup Files page is CONFIG, not a package, so no package +# stack ever ran it. +cfg shell-env + +book acpid # power/lid/button events -- what a laptop needs +book fcron # the cron daemon; make-ca's weekly update needs one + +# blfs-bootscripts: 20250225 is the FINAL release (BLFS has stopped shipping +# the SysV edition). Only the scripts for services this system actually has +# are installed -- adding one for something uninstalled just leaves a symlink +# in rc?.d that fails at boot. +# +# dbus needed by sway, elogind and most of the desktop +# alsa saves/restores the mixer state across reboots +# bluetooth bluetoothd, from BlueZ +# acpid lid and power button +# fcron cron +# iptables restores the firewall ruleset at boot +# random seeds /dev/urandom from the saved pool +# service-wpa the `wpa` helper `ifup` uses for a wireless interface +tar blfs-bootscripts \ + url=https://anduin.linuxfromscratch.org/BLFS/blfs-bootscripts/blfs-bootscripts-20250225.tar.xz \ + pkg=blfs-bootscripts-20250225.tar.xz \ + have=/etc/rc.d/init.d/dbus \ + build="true" \ + install="make install-dbus install-alsa install-bluetooth install-acpid install-fcron install-iptables install-random install-service-wpa" diff --git a/stacks/shell-env.sh b/stacks/shell-env.sh new file mode 100755 index 0000000..c254b33 --- /dev/null +++ b/stacks/shell-env.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# shell-env.sh -- interactive shell environment: coloured ls, bash +# completion, and username completion for su (root step, idempotent). +# +# BLFS's "The Bash Shell Startup Files" page writes /etc/profile and +# /etc/bashrc, and /etc/bashrc is what makes ls colourful -- it evals +# dircolors and aliases ls --color=auto. It is a config page, not a +# package, so nothing in a package stack ever ran it. +# +# bash-completion is not a BLFS package at all; it comes from upstream and +# is a `tar` entry in the stack beside this script. +set -u + +# 1. /etc/dircolors -- the database ls colours come from +if [ ! -f /etc/dircolors ]; then + if command -v dircolors >/dev/null 2>&1; then + dircolors -p > /etc/dircolors + echo "# wrote /etc/dircolors (from dircolors -p)" + fi +else + echo "# /etc/dircolors exists -- left alone" +fi + +# 2. /etc/bashrc -- colours, prompt, and the completion loader. +# Written only if absent: this is a file people edit. +if [ -f /etc/bashrc ]; then + echo "# /etc/bashrc exists -- left alone (see /etc/bashrc.new)" + _dest=/etc/bashrc.new +else + _dest=/etc/bashrc +fi +cat > "$_dest" <<'BASHRC' +# Begin /etc/bashrc +# Based on the BLFS "Bash Shell Startup Files" page. + +# Coloured ls and grep. +if [ -f /etc/dircolors ] ; then + eval "$(dircolors -b /etc/dircolors)" +fi +if [ -f "$HOME/.dircolors" ] ; then + eval "$(dircolors -b "$HOME/.dircolors")" +fi +alias ls='ls --color=auto' +alias grep='grep --color=auto' + +# Prompt: red for root, green for everyone else. +NORMAL="\[\e[0m\]" +RED="\[\e[1;31m\]" +GREEN="\[\e[1;32m\]" +if [[ $EUID == 0 ]] ; then + PS1="$RED\u [ $NORMAL\w$RED ]# $NORMAL" +else + PS1="$GREEN\u [ $NORMAL\w$GREEN ]\$ $NORMAL" +fi +unset RED GREEN NORMAL + +# bash-completion, if it is installed. Loads every completion in +# /usr/share/bash-completion/completions, including the package-user +# accounts for `su`. +if [ -r /usr/share/bash-completion/bash_completion ] ; then + . /usr/share/bash-completion/bash_completion +fi + +# Complete USER NAMES after su and friends. bash-completion supplies this +# too, but these three cost nothing and work without it -- which matters on +# a package-user system, where `su - p_<tab>` is a daily action. +complete -A user su sudo passwd chage groups id + +# End /etc/bashrc +BASHRC +chmod 644 "$_dest" +echo "# wrote $_dest" + +# 3. make sure /etc/profile actually sources it, and profile.d +if [ -f /etc/profile ] && ! grep -q '/etc/bashrc' /etc/profile 2>/dev/null; then + echo "# NOTE: /etc/profile does not source /etc/bashrc." + echo "# Interactive non-login shells read ~/.bashrc, which on this" + echo "# system should contain: . /etc/bashrc" +fi + +# 4. the package users' own bashrc: they share /etc/pkgusr/bashrc +if [ -f /etc/pkgusr/bashrc ] && ! grep -q '/etc/bashrc' /etc/pkgusr/bashrc; then + printf '\n# colours, completion and the shared aliases\n[ -r /etc/bashrc ] && . /etc/bashrc\n' \ + >> /etc/pkgusr/bashrc + echo "# package users now source /etc/bashrc too" +fi + +echo "# done -- open a new shell, or: . /etc/bashrc" diff --git a/stacks/sway-session.sh b/stacks/sway-session.sh new file mode 100755 index 0000000..189420e --- /dev/null +++ b/stacks/sway-session.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# sway-session.sh -- wire the session together (root step, idempotent). +set -u +# The desktop user: LFS_DESKTOP_USER, or what mytools.sh recorded in +# /etc/pkgusr/desktop-user -- one answer for both scripts. (LFS_HOME_USER +# is still honoured for old invocations.) Unknown = skip the config +# skeleton below, nothing else here needs it. +HOME_USER="${LFS_DESKTOP_USER:-${LFS_HOME_USER:-}}" +[ -z "$HOME_USER" ] && [ -r /etc/pkgusr/desktop-user ] \ + && HOME_USER="$(head -n1 /etc/pkgusr/desktop-user)" +[ -n "$HOME_USER" ] && printf '%s\n' "$HOME_USER" > /etc/pkgusr/desktop-user +HOME_DIR="/home/${HOME_USER:-nobody-set}" + +# 1. one launcher: a dbus session around sway. +# +# The seat comes from seatd, not elogind: elogind only registers a session +# through pam_elogind, and Linux-PAM is not on this system, so a tty login +# has no logind session for libseat to find. Saying the backend explicitly +# means libseat does not try logind first and fail with a puzzling message. +cat > /usr/bin/sway-session <<'LAUNCH' +#!/bin/bash +# start sway inside a dbus session; run from a plain tty login +export XDG_CURRENT_DESKTOP=sway +export LIBSEAT_BACKEND=seatd + +# The seat comes from seatd-user: a setuid wrapper compiled for one username, +# installed as /usr/bin/seatd-<user> by the mytools step. No daemon runs at +# boot and there is no `seat` group, so check for the wrapper instead. +# (It said /bin here while mytools installs to /usr/bin -- on a system where +# /bin is a symlink to /usr/bin that works by accident, and on one where it +# is not it reports a missing file that exists.) +if ! command -v "seatd-$(id -un)" >/dev/null 2>&1; then + echo "sway-session: seatd-$(id -un) is not on PATH -- nothing will give" >&2 + echo " sway a seat. As root: packagemanager stack services --run --only mytools" >&2 + echo " (or use your own launcher: sway_start)" >&2 + exit 1 +fi +# INPUT DEVICES COME FROM UDEV, NOT FROM /dev. +# +# libinput enumerates through libudev, which reads /run/udev -- so in a +# chroot without the host's /run bind-mounted there are no input devices at +# all, and the failure names libinput rather than the chroot: +# libinput initialization failed, no input devices +# Failed to start backend +# The compositor is fine; it has nothing to listen to. +if [ ! -d /run/udev ]; then + echo "sway-session: /run/udev is missing, so libinput will find no input" >&2 + echo " devices and the backend will fail. Two causes:" >&2 + echo " * a chroot -- udev's database lives in the host's /run." >&2 + echo " Boot the system, or bind-mount /run and /dev into the chroot." >&2 + echo " * udev is not running -- check /etc/rc.d/init.d/udev status" >&2 + echo " To prove the REST works anyway, with no keyboard or mouse:" >&2 + echo " WLR_LIBINPUT_NO_DEVICES=1 sway_start" >&2 + exit 1 +fi +# ...and a compositor needs the VT it was started from. Started from a +# different tty than the one seatd bound, it opens nothing. +if [ -n "${XDG_VTNR:-}" ] && [ -r /sys/class/tty/tty0/active ]; then + _active="$(cat /sys/class/tty/tty0/active 2>/dev/null)" + if [ -n "$_active" ] && [ "$_active" != "tty$XDG_VTNR" ]; then + echo "sway-session: you are on tty$XDG_VTNR but $_active is the active" >&2 + echo " console. Switch to it and start sway there." >&2 + exit 1 + fi +fi + +if [ -z "${XDG_RUNTIME_DIR:-}" ]; then + export XDG_RUNTIME_DIR="/run/user/$(id -u)" + mkdir -p "$XDG_RUNTIME_DIR" 2>/dev/null && chmod 700 "$XDG_RUNTIME_DIR" +fi +exec dbus-run-session -- sway "$@" +LAUNCH +chmod 755 /usr/bin/sway-session + +# 2. pipewire on demand, not at boot +cat > /usr/bin/pipewire-start <<'PW' +#!/bin/bash +# start pipewire + wireplumber for this session, if not already up +pgrep -x pipewire >/dev/null || { pipewire & disown; } +pgrep -x wireplumber >/dev/null || { wireplumber & disown; } +echo "# pipewire is up (pipewire-pulse socket in $XDG_RUNTIME_DIR)" +PW +chmod 755 /usr/bin/pipewire-start + +# 3. a sway config skeleton for the desktop user, only if none exists +if [ -d "$HOME_DIR" ] && [ ! -f "$HOME_DIR/.config/sway/config" ]; then + mkdir -p "$HOME_DIR/.config/sway" + cp /etc/sway/config "$HOME_DIR/.config/sway/config" 2>/dev/null || true + { + echo "" + echo "# --- added by the sway stack ---" + echo "exec swbr" + echo "# audio when needed: exec pipewire-start" + echo 'bindsym Mod4+b exec minibrowser' + echo 'bindsym Mod4+a exec swas' + echo 'bindsym Mod4+Tab exec swov' + } >> "$HOME_DIR/.config/sway/config" + chown -R "$HOME_USER:" "$HOME_DIR/.config/sway" 2>/dev/null || true + echo "# wrote $HOME_DIR/.config/sway/config" +fi +[ -n "$HOME_USER" ] || echo "# no desktop user known (LFS_DESKTOP_USER unset, no /etc/pkgusr/desktop-user): sway config skeleton skipped" +echo "# session wired: log in on a tty and run: sway (alias for sway_start)" diff --git a/stacks/sway.stack b/stacks/sway.stack new file mode 100644 index 0000000..2e4a652 --- /dev/null +++ b/stacks/sway.stack @@ -0,0 +1,302 @@ +# sway.stack -- a working Sway desktop with a minimal WebKit browser. +# stack-version: 1.13.67 +# Replay with: packagemanager stack sway --run +# Entries run top to bottom; book deps are resolved from the BLFS book, +# installed entries are skipped, the first failure stops the run. +# git refs are pinned to releases -- true them up if a clone fails. + +# Never built by this stack, however they are reached. harfbuzz RECOMMENDS +# texlive and LibreOffice; a sway desktop with a webkit browser wants +# neither, and texlive alone is hours of build time and a /opt tree. +# NOT BUILT BY THIS STACK. The book's recommendations are followed -- +# a recommendation the build actually needs (curl for cmake, icu for +# libxml2) must not be dropped by a blanket rule. These are the branches +# this desktop does not want, named one at a time, and everything they alone +# pull in goes with them. +avoid libreoffice +avoid asymptote +avoid biber +avoid qt6 +avoid okular +avoid phonon +avoid phonon-backend-vlc +avoid vlc +avoid cups +avoid openjdk +avoid java +avoid apache-ant +avoid postgresql +avoid openldap +avoid unixodbc +avoid graphviz +avoid gegl +# the KDE leaves okular pulls in: named individually, because a package can +# have several parents and pruning by one of them drops shared dependencies +# (graphite2 was lost that way) +avoid plasma-activities +avoid libkexiv2 +avoid kwindowsystem +avoid libfm-qt +avoid xdg-desktop-portal-lxqt +avoid xdg-desktop-portal-gnome +avoid gnome-desktop + +# The desktop-portal tail. The book makes xdg-desktop-portal REQUIRE all +# four backends -- gtk, wlr, gnome AND lxqt -- so one recommendation from +# gtk4 drags in the whole of xfce and lxqt, gnome-desktop and libfm-qt. +# Sway needs none of it to run; screen sharing wants xdg-desktop-portal-wlr, +# which can be built on its own later: +# packagemanager install xdg-desktop-portal-wlr --run +avoid xdg-desktop-portal +avoid xdg-desktop-portal-gtk +avoid xdg-desktop-portal-wlr +avoid xdg-desktop-portal-gnome +avoid xdg-desktop-portal-lxqt +avoid xfce4-panel +avoid xfce4-notifyd +avoid xfce4-dev-tools +avoid xfconf +avoid libxfce4util +avoid libxfce4ui +avoid libxfce4windowing +avoid exo +avoid garcon +avoid liblxqt +avoid lxqt-build-tools +avoid lxqt-notificationd +avoid lxqt-kwindowsystem +avoid libqtxdg +avoid polkit-qt +avoid libfm-qt +avoid menu-cache +avoid libwnck +avoid notification-daemon + +# Not this desktop either: a systemd package on a SysV system, phone modem +# stacks, a mail/ssh server, spell-checking dictionaries. +avoid systemd +avoid ModemManager +avoid libmbim +avoid libqmi +avoid geoclue2 +avoid openssh +avoid ruby +avoid aspell +avoid enchant +avoid gnome-keyring +avoid gcr +avoid startup-notification + +# Tails of things already avoided. Each of these is in the plan ONLY to +# serve a package this stack does not build, and the tools cannot yet work +# that out for themselves (see 1.12.83: pruning by the single parent that +# `blfs order` reports dropped shared dependencies, and broke harfbuzz). +# Checked one by one against the plan's parent column: +avoid libcupsfilters # cups (avoided) +avoid libppd # cups +avoid cups-filters # cups +avoid libkexiv2 # okular (avoided) +avoid exiv2 # libkexiv2 +avoid clucene # libreoffice (avoided) +avoid redland # libreoffice +avoid rasqal # redland +avoid raptor # rasqal +avoid jasper # qt6 (avoided) +avoid libmng # qt6 +avoid double-conversion # qt6 +avoid poppler # libcupsfilters -- its ONLY parent, avoided +avoid mupdf # libcupsfilters +avoid gs # libcupsfilters +avoid qpdf # libcupsfilters +avoid nss # poppler +avoid boost # poppler +avoid pdfio # libcupsfilters +avoid gpgmepp # poppler +# Checked against EVERY parent, not just the one the plan prints: these have +# no parent left in the plan at all. +avoid colord # cups (avoided) -- no live parent +avoid linux-pam # systemd (avoided) -- no live parent +avoid xdg-utils # cups -- no live parent +avoid freeglut # mupdf (avoided) -- no live parent +avoid polkit # colord + systemd (both avoided) -- no live parent +avoid libgusb # colord +# NOT avoided, though the plan labels them with an avoided parent: the label +# shows only the FIRST parent. gtk3 is really wanted by adwaita-icon-theme, +# ffmpeg by mpv, polkit by colord... and pulseaudio by pipewire/sdl3/gst. +# (rust and llvm STAY: glycin genuinely requires rust, and llvm is a stack +# entry in its own right for mesa.) + +# 0. verify the kernel (DRM/amdgpu, evdev, snd, seccomp, user ns, render node) +check kernel-sway + +# 1. seat and session (SysV: elogind is the logind; seatd is not in the book) +book dbus +book elogind url=https://github.com/elogind/elogind ref=v255.17 prog=loginctl build="meson setup build --prefix=/usr --buildtype=release -Dcgroup-controller=elogind -Ddbuspolicydir=/etc/dbus-1/system.d; ninja -C build" install="ninja -C build install" + +# 2. graphics -- llvm feeds mesa; mesa options come from the machine +# (machine.conf wiring pending; book defaults until then) +# cmake (pulled in by llvm and others) has NO required dependencies and four +# recommended ones -- and its own build turns on the system libraries, so a +# missing curl is a hard failure: +# CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! +# This is the trade-off, and the answer is the same one: name +# what the build actually needs. +book nghttp2 +book curl +book libarchive +book libuv +book llvm +# glslang provides glslangValidator, which mesa needs ONLY because +# machine.conf turns on the Vulkan drivers -- the book does not list it as a +# mesa dependency, so no dependency order can place it. A machine option +# that creates a dependency has to name it here. +book glslang +# The font stack goes BEFORE mesa. +# +# libass (built inside mesa's plan, via ffmpeg) hard-requires harfbuzz in its +# configure, and the book's libass page does not list it -- so no dependency +# order can place it. What CAN place it is entry order: anything an earlier +# entry installed is simply there. +book harfbuzz # text shaping: freetype recommends it, +book graphite2 # and webkit/pango need both +book fontconfig + +book mesa +book xwayland +book libinput +book libxkbcommon + +# 3. wayland core, then the compositor from git (not in the book) +book wayland +book wayland-protocols + +# 4. a terminal (foot, wayland-native) and its two small deps + +# 5. fonts and 2D +# +# From here on the entries are the RECOMMENDATIONS worth having, named +# instead of inherited: with the book's suggestions no longer +# arrive by themselves, which is the point -- LibreOffice, OpenJDK, CUPS, +# Qt6/okular/VLC, texlive and the xfce/lxqt portals are not part of this +# desktop. Anything missed shows up as a clean "not found" at configure +# time and becomes one more line here. + +# 6. GTK4 + WebKitGTK (6.0 API), gstreamer (bad included -- wanted), mpv +book pango # GTK4 text layout +book gdk-pixbuf # GTK4 image loading +book hicolor-icon-theme +book adwaita-icon-theme # cursors and the default icon set +book gtk4 +book glib-networking # TLS for libsoup, or the browser has no https +book libsoup3 +book gstreamer10 +book gst10-plugins-base +book gst10-plugins-good +# --------------------------------------------------------------------- +# WebRTC and camera streaming. +# +# BLFS 13.0 carries NONE of these, so they are git entries. They must be +# built BEFORE gst10-plugins-bad and webkitgtk, because webrtcbin -- the +# element a browser uses for WebRTC -- is compiled into gst-plugins-bad only +# if libnice is present, and webkitgtk links the result. Adding them later +# means recompiling both (hours). +# +# What each is for: +# libnice ICE: the NAT traversal WebRTC cannot work without +# libsrtp the encrypted media transport +# usrsctp WebRTC data channels +# webrtc-audio-processing echo cancellation for a microphone (the log's +# "webrtc-audio-processing-2 found: NO") +# +# The camera side is already in this stack: v4l-utils for the device and +# pipewire for routing it. +# +# UNTESTED -- the build lines follow each project's README, and none of these +# has been built here. Expect to adjust them; that is what the stack file is +# for. +git libnice url=https://gitlab.freedesktop.org/libnice/libnice.git ref=0.1.22 have=/usr/lib/pkgconfig/nice.pc build="meson setup build --prefix=/usr --buildtype=release -Dgtk_doc=disabled; ninja -C build" install="ninja -C build install" +git libsrtp url=https://github.com/cisco/libsrtp.git ref=v2.6.0 have=/usr/lib/pkgconfig/libsrtp2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +# ref: 1.0.0 does not exist upstream (checked: the tags are 0.9.3.0, 0.9.4.0, +# 0.9.5.0). 0.9.5.0 has meson.build, so the build line below still applies. +git usrsctp url=https://github.com/sctplab/usrsctp.git ref=0.9.5.0 have=/usr/lib/pkgconfig/usrsctp.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +git webrtc-audio-processing url=https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing.git ref=v2.1 have=/usr/lib/pkgconfig/webrtc-audio-processing-2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" + +book gst10-plugins-bad +# No exclusions. The book's dependency graph is built as the book states +# it, recommendations included: a recommendation is there for a reason, and +# skipping one is how texlive came to build before FreeType. This costs +# build time (texlive, LibreOffice and their tails are hours) and buys a +# system that matches the book. +book webkitgtk +book mpv + +# 7. audio: ALSA is the floor; pipewire installed, started on demand +book alsa-lib +book alsa-utils +book pipewire +book wireplumber + +# 8. SDL3 for the Sw* tools +book sdl3 + +# 9. the nimbin2 family: make all && make PREFIX=/usr install + +# 10. wire the session together (root step) + + +# TeX is NOT built by this stack. +# +# Putting `book texlive` last only moved the ENTRY: texlive is harfbuzz's +# recommendation, so it was still pulled into an early entry's plan and +# failed there (its configure wants freetype-config, which FreeType removed +# years ago -- an upstream problem, not this desktop's). Nothing here needs +# TeX: the book's `make ... pdf ps` manual builds are commented out (1.12.87) +# and asymptote/biber are avoided. +# +# To build it later, on its own: packagemanager install texlive --run +avoid texlive + + +# --------------------------------------------------------------------- +# Everything from here needs GIT, or is a plain tarball: kept to the END. +# +# A git entry failed with "git: command not found" -- git is itself a BLFS +# package, so anything cloned has to come after it is built. Grouping them +# here also means the desktop is finished before the extras start, and +# `--no-git` can skip the lot. +# --------------------------------------------------------------------- +book git + +# -Dserver=enabled: seatd-user execs /usr/bin/seatd, so the BINARY must +# exist even though no daemon runs at boot. -Dlibseat-seatd=enabled is what +# libseat needs for LIBSEAT_BACKEND=seatd; logind stays enabled as a fallback +# but is unusable here (no pam_elogind). +git seatd url=https://git.sr.ht/~kennylevinsen/seatd ref=0.9.1 prog=seatd build="meson setup build --prefix=/usr --buildtype=release -Dserver=enabled -Dlibseat-seatd=enabled -Dlibseat-logind=elogind; ninja -C build" install="ninja -C build install" +# wlroots' DRM backend needs libdisplay-info; without it meson DISABLES drm +# at configure time and says so only in a summary line, and sway then dies at +# runtime with "Cannot create DRM backend: disabled at compile-time". +book libdisplay-info + +# -Dbackends=drm,libinput,... is explicit on purpose: an auto-detected +# backend list is a silent dependency, and this is the dependency that was +# silently missing. +git wlroots url=https://gitlab.freedesktop.org/wlroots/wlroots.git ref=0.19.2 have=/usr/lib/pkgconfig/wlroots-0.19.pc build="meson setup build --prefix=/usr --buildtype=release -Dbackends=drm,libinput,x11 -Dsession=enabled; ninja -C build" install="ninja -C build install" +# needs=: book packages a git entry depends on -- nothing in the book orders +# anything before a git build, so its dependencies must be named here. +# the author's own options: man pages, swaybar, swaynag, bash completions +git sway url=https://github.com/swaywm/sway ref=1.11 prog=sway needs=json-c,cairo,pango build="meson setup build --prefix=/usr --buildtype=release -Dman-pages=enabled -Dswaybar=true -Dswaynag=true -Dbash-completions=true; ninja -C build" install="ninja -C build install" +git swaybg url=https://github.com/swaywm/swaybg ref=v1.2.1 prog=swaybg build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +git tllist url=https://codeberg.org/dnkl/tllist ref=1.1.0 have=/usr/include/tllist.h build="meson setup build --prefix=/usr" install="ninja -C build install" +git fcft url=https://codeberg.org/dnkl/fcft ref=3.3.2 have=/usr/include/fcft build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +git foot url=https://codeberg.org/dnkl/foot ref=1.28.0 prog=foot build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +tar dejavu-fonts url=https://downloads.sourceforge.net/dejavu/dejavu-fonts-ttf-2.37.tar.bz2 pkg=dejavu-fonts-ttf-2.37.tar.bz2 have=/usr/share/fonts/dejavu build="true" install="install -v -d -m755 /usr/share/fonts/dejavu; install -v -m644 ttf/*.ttf /usr/share/fonts/dejavu; fc-cache /usr/share/fonts/dejavu" +git sdl3-image url=https://github.com/libsdl-org/SDL_image ref=release-3.2.4 have=/usr/lib/libSDL3_image.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" +git sdl3-ttf url=https://github.com/libsdl-org/SDL_ttf ref=release-3.2.2 have=/usr/lib/libSDL3_ttf.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" +# Browser builds two programs, browser-mini and browser-big; the entry keeps +# its old name because p_minibrowser is the account already on disk. +git minibrowser url=https://git.christianimmanuel.de/web-tools/Browser prog=browser-mini +git swbr url=https://git.christianimmanuel.de/sway/SwBr prog=swbr +git swov url=https://git.christianimmanuel.de/sway/SWOv prog=swov +git swas url=https://git.christianimmanuel.de/sway/SwAS prog=swas + +cfg sway-session \ No newline at end of file diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh index d13d604..2a61229 100644 --- a/test_lfs_crosschain.sh +++ b/test_lfs_crosschain.sh @@ -10,7 +10,41 @@ set -u LFS_TOOL="${1:-./lfs}" -LFS_BOOK="${2:-/mnt/user-data/uploads/LFS-BOOK-12_4-NOCHUNKS.html}" +# A bare name is resolved off $PATH; anything unreadable stops NOW with one +# message. `bash test_lfs_crosschain.sh lfs` used to produce hundreds of +# python tracebacks -- every heredoc failing to open 'lfs' -- burying the one +# fact that mattered: there is no ./lfs in this directory. +case "$LFS_TOOL" in + */*) : ;; + *) if [ ! -f "$LFS_TOOL" ] && command -v "$LFS_TOOL" >/dev/null 2>&1; then + LFS_TOOL="$(command -v "$LFS_TOOL")" + fi ;; +esac +if [ ! -r "$LFS_TOOL" ]; then + echo "!! no such tool: $LFS_TOOL" >&2 + echo " run from the checkout (bash test_lfs_crosschain.sh ./lfs)" >&2 + echo " or give a path (bash test_lfs_crosschain.sh /usr/bin/lfs)" >&2 + exit 1 +fi + +# The book is found, not baked in: a path from whoever ran the suite last is +# wrong on every other machine. Missing entirely is fine -- the handful of +# tests that need it already fail with a clear FileNotFoundError naming it. +_find_book() { + local d f + for d in "$(dirname "$LFS_TOOL")" "${LFS_STORE:-/usr/share/lfs}/books"; do + for f in "$d"/LFS-BOOK-*-NOCHUNKS.html "$d"/LFS-BOOK-*-nochunks.html; do + [ -r "$f" ] && { printf '%s' "$f"; return 0; } + done + done + printf '%s' "LFS-BOOK-NOT-FOUND.html" +} +LFS_BOOK="${2:-$(_find_book)}" +# The phase runner every generated script sources (format v18). Tests that +# used to grep a generated script for what it DID now grep this file; tests +# that RUN a script find it through PKGUSR_LIB, as the real runners do. +PHASES_LIB="$(cd "$(dirname "$LFS_TOOL")" && pwd)/lfs-phases" +export PKGUSR_LIB="$PHASES_LIB" T=$(mktemp -d /tmp/lfstest.XXXXXX) PASS=0; FAIL=0 # Count in FILES, not shell variables. @@ -115,9 +149,9 @@ bash -n "$T/demo.sh" && ok "generated script is valid bash" || bad "syntax error # phases must be split the way the book reads grep -q "^\.\./configure" "$T/demo.sh" && ok "configure landed in build phase" \ || bad "configure not in build phase" -sed -n '/^install_pkg/,/INSTALL DONE/p' "$T/demo.sh" | grep -q "make install" \ +sed -n '/^install_pkg/,/^}/p' "$T/demo.sh" | grep -q "make install" \ && ok "make install landed in install phase" || bad "install phase wrong" -sed -n '/^configure_pkg/,/CONFIGURE DONE/p' "$T/demo.sh" | grep -q "post-install-fixup" \ +sed -n '/^configure_pkg/,/^}/p' "$T/demo.sh" | grep -q "post-install-fixup" \ && ok "post-install cmd landed in configure phase" || bad "configure phase wrong" # --------------------------------------------------------------- run it fully @@ -135,7 +169,7 @@ fi || bad "install produced no file" [ -f "$LFS/tools/fixup.txt" ] \ && ok "configure (post-install) phase ran" || bad "configure phase did not run" -grep -q "^/.*build$" "$LFS/build/.cc-build-demo" 2>/dev/null \ +grep -q "^/.*build$" "$LFS/build/.pkgusr-demo-1.0.cwd" 2>/dev/null \ && ok "recorded build dir for resume" || bad "build dir not recorded" # ------------------------------------------------- single-phase re-run works @@ -544,8 +578,8 @@ fi python3 - "$LFS_TOOL" <<'PY6' import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -body = lfs._crosschain_script_body("demo", "ch-system-demo", "demo-1.0", - "demo-1.0.tar.*", ["make", "make install"]) +import os +body = open(os.path.join(os.path.dirname(sys.argv[1]), "lfs-phases")).read() if "--no-same-owner" not in body: print(" FAIL unpack does not use tar --no-same-owner"); sys.exit(1) print(" PASS unpack never restores the archive's UIDs") @@ -555,7 +589,7 @@ _count_rc $? helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then # markers live in the build scratch now, not in the tarball store - grep -q 'rm -f "\$blddir/.cc-build-\$name"' "$helper_src" \ + grep -q 'rm -f "\$blddir"/.pkgusr-\*.dir' "$helper_src" \ && ok "stale build markers are cleared before a build" \ || bad "stale build markers are not cleared" grep -q "^cmd_clean_sources()" "$helper_src" \ @@ -1636,12 +1670,29 @@ if "--no-index" not in b: problems.append("pip is still allowed to reach for an index") # wget comes from the ordinary install path, not a private copy of it -if "install" not in sb or "wget" not in sb: - problems.append("setup no longer installs wget") -if re.search(r"^def _setup_install_wget\(", src, re.M): - wb = body("_setup_install_wget") - if '"install"' not in wb: - problems.append("wget is installed by some path other than `install`") +if "wget" not in sb: + problems.append("bootstrap no longer installs wget") +# it may delegate -- follow one hop rather than demanding the literal here +_wb = body("_setup_install_wget") + body("_install_via_self") +if '"install"' not in _wb: + problems.append("wget is installed by some path other than `install`") + +# THE FIVE STAGES. What was missing was never the individual checks -- the +# tool could already tell you about wget, the book, make-ca and the bundle +# separately. It was the ORDER, on a system where every one of them is +# unsatisfied at once and nothing says which to do first. +if not re.search(r"^_BOOTSTRAP_STAGES = \[", src, re.M): + problems.append("the bootstrap stages are not a list anything can read") +for key in ("wget", "wheels", "book", "make_ca", "ca"): + if '"%s"' % key not in body("_bootstrap_state"): + problems.append("bootstrap does not check for %s" % key) +# stages 3-5 need a working download, so they are printed, never run +nb = body("_bootstrap_next_steps") +for cmd in ("blfs fetch", "make-ca", "make-ca -g --force"): + if cmd not in nb: + problems.append("the remaining steps do not mention %s" % cmd) +if "subprocess" in nb or "_run(" in nb: + problems.append("the remaining steps are executed instead of printed") # IDEMPOTENT. The old final step ran exactly once, at the end of a six-hour # build; this one has to be safe to run again after fixing whatever failed. @@ -1662,10 +1713,38 @@ print(" PASS setup installs wget and the wheels, offline, as package users") PYSETUP _count_rc $? +# The old name must keep working. `lfs build-system get-sources` on an +# already-installed system calls `packagemanager setup --sources`, and a rename +# that breaks the tool which downloads the sources costs a reboot to find out. +_o="$(python3 "$pm_src" setup --sources 2>/dev/null | grep -c '^https\?://')" +[ "${_o:-0}" -ge 7 ] \ + && ok "the old 'setup' name still answers --sources" \ + || bad "renaming setup broke the call get-sources makes" +_o="$(python3 "$pm_src" bootstrap --sources 2>/dev/null | grep -c '^https\?://')" +[ "${_o:-0}" -ge 7 ] \ + && ok "bootstrap --sources lists the wheels" \ + || bad "bootstrap --sources lists nothing" +# and lfs must ask for the new name, with the old one as the fallback +if grep -q '"packagemanager", "bootstrap", "--sources"' "$LFS_TOOL"; then + grep -q '"packagemanager", "setup", "--sources"' "$LFS_TOOL" \ + && ok "get-sources asks for bootstrap, and falls back to setup" \ + || bad "get-sources cannot talk to an older packagemanager" +else + bad "get-sources still asks for the old name only" +fi + # It must REFUSE rather than half-run when the wheels were never downloaded. # There is no network on a fresh system, so "try it and see" is a hang. +# +# On a system that is ALREADY bootstrapped there is nothing left for setup to +# install, so exit 0 with an empty sources dir is the truth, not the bug this +# guards against. That state is skipped, not failed -- the refusal path can +# only be exercised where something is still missing. _sbx="$T/setup-empty"; mkdir -p "$_sbx" _o="$(LFS_SOURCES_DIR="$_sbx" python3 "$pm_src" setup --run --yes 2>&1)"; _rc=$? +if [ "$_rc" -eq 0 ] && ! printf '%s\n' "$_o" | grep -q '^ -- '; then + echo " SKIP setup has nothing left to install on this system" +else _p="" [ "$_rc" -ne 0 ] || _p="$_p;setup --run succeeded with no wheels on disk" printf '%s\n' "$_o" | grep -q 'get-sources' \ @@ -1680,6 +1759,7 @@ else ok "setup stops when the wheels are missing, and says which" fi fi +fi @@ -1744,9 +1824,19 @@ if [ -f "$helper_src" ]; then grep -q "A ROOT step must not get" "$helper_src" \ && ok "root steps run without the package-user wrappers" \ || bad "root steps still get the wrappers and cannot chown" - grep -q "envpass=\"\$envpass PATH='/usr/local/bin" "$helper_src" \ - && ok "the root PATH has no wrapper directory" \ - || bad "root's PATH still starts with the wrappers" + # the PATH is built from a variable now (1.13.19 appends the /opt + # directories the packages' own profile snippets add), so check the + # PROPERTY: root gets the base, a package user gets the wrappers first + _rootpath="$(_slice_fn "$helper_src" cmd_build | grep -A2 'as_root" = 1' | grep "PATH='")" + case "$_rootpath" in + *'$WRAPPERS'*) bad "root's PATH still starts with the wrappers" ;; + *"PATH='\$_base'"*) ok "the root PATH has no wrapper directory" ;; + *) bad "root's PATH is not built from the shared base list" ;; + esac + _slice_fn "$helper_src" cmd_build | grep -q "PATH='\$WRAPPERS:\$_base'" \ + || bad "a package user's PATH no longer starts with the wrappers" + _slice_fn "$helper_src" cmd_build | grep -q '/opt/\*) \[ -d' \ + || bad "packages installed under /opt are invisible to later builds" # /usr/src is shared install space; /sources and /build are scratch. Both # kinds must be beyond a package's reach, but by DIFFERENT mechanisms -- # conflating them is what once made /root root:install drwxrwx---, and what @@ -1913,12 +2003,6 @@ if [ -f "$helper_src" ]; then || bad "shared drop-in directories are misclassified" fi -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - grep -q "^diagnose_install_failure()" "$pmi" \ - && ok "a failed install explains itself instead of dumping a traceback" \ - || bad "install failures are reported as raw build output" -fi # ---- packagemanager must recover like lfs-helper does ----------------------- # # The same permission problem happens after the base build as during it, and it @@ -1981,7 +2065,7 @@ print(" PASS a large log is scanned in %.0f ms and the error is still found" PY32 _count_rc $? fi -for f in lfs-helper packagemanager_install; do +for f in lfs-helper; do t="$(dirname "$LFS_TOOL")/$f" [ -f "$t" ] || continue grep -q "tail -c 524288" "$t" \ @@ -1994,22 +2078,6 @@ done # affairs under the package-user model, not a mistake to report. The answer is # always the same -- put both packages in a collector group -- so it should be # done automatically, not printed as a command for the user to copy. -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - grep -q "^auto_grant_from_log()" "$pmi" \ - && ok "packagemanager grants collector access on a permission failure" \ - || bad "packagemanager only reports permission failures" - grep -qE '_round\" -lt (6|20)' "$pmi" \ - && ok "it retries until nothing new needs granting" \ - || bad "one grant round is not enough for a multi-directory install" - # an EXISTING collector group must be joined, not duplicated - grep -q '"\${prefix}"\*)' "$pmi" \ - && ok "an existing collector group is joined rather than replaced" \ - || bad "a second group would be made for an already-shared directory" - grep -q "PM_NO_AUTO_FIX" "$pmi" \ - && ok "the automatic repair can be turned off" \ - || bad "no way to opt out of automatic permission changes" -fi # ---- a retry wrapper must not call itself ----------------------------------- # # run_pm_install() gained a retry loop and, in wrapping the old call, ended up @@ -2059,15 +2127,6 @@ fi # Treating those as install targets granted a package write access to # /usr/lib/python3.13 -- and invented a "<prefix>_root" group, which would mean # "may write anywhere root owns" and undoes the entire scheme. -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - grep -q 'owned by root -- NOT shared' "$pmi" \ - && ok "a collector group is never created for root" \ - || bad "a root collector group could still be created" - grep -q 'grep -vE "\^\[\[:space:\]\]\*(File|Traceback' "$pmi" \ - && ok "traceback frames are excluded when looking for failed paths" \ - || bad "paths from tracebacks could be granted" -fi helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then grep -q "not creating a collector group for it" "$helper_src" \ @@ -2174,8 +2233,11 @@ if [ -f "$bt" ]; then grep -q 'inst.find_all(\["pre", "h3", "h4", "div"\])' "$bt" \ && ok "commands are collected from nested sub-sections too" \ || bad "only top-level command blocks would be found" - # a stale index must not survive the fix - grep -q "CACHE_VERSION = 4" "$bt" \ + # a stale index must not survive an extraction change: the version must + # be AT LEAST the one each fix shipped with (pinning an exact number made + # this test itself go stale on the next bump) + _cv="$(grep -oE '^CACHE_VERSION = [0-9]+' "$bt" | grep -oE '[0-9]+')" + [ "${_cv:-0}" -ge 5 ] \ && ok "the cache version was bumped so stale indexes rebuild" \ || bad "cached books would keep serving the old empty commands" fi @@ -2948,12 +3010,6 @@ if [ -f "$helper_src" ]; then && ok "no collector group is created outside the allocator" \ || bad "a bare groupadd bypasses the collector range (line $outside)" fi -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - grep -q "_cgid=90000" "$pmi" \ - && ok "packagemanager uses the same collector range" \ - || bad "packagemanager still allocates collector gids from the user range" -fi # ---- the two tools spell the same thing the same way ------------------------ # python3 - "$LFS_TOOL" <<'PY47' @@ -3501,45 +3557,6 @@ fi # install: cannot change permissions of '/usr/sbin': Operation not permitted # lfs-helper wraps install/chmod/chown during the chroot build for exactly # this; packagemanager_install did not, so the same package failed after boot. -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - grep -q "^_make_build_wrappers()" "$pmi" \ - && ok "packagemanager wraps install/chmod for the build" \ - || bad "a package re-moding an existing directory still fails" - grep -q "PATH='\$_wrapdir'" "$pmi" \ - && ok "the wrappers are ahead of the real tools during the build" \ - || bad "the wrappers are created but never used" - - # behaviour, not just presence - wd="$T/wrapmk"; mkdir -p "$wd/wrap" "$wd/exists" - ( eval "$(sed -n '/^_make_build_wrappers() {/,/^}/p' "$pmi")" - _make_build_wrappers "$wd/wrap" ) >/dev/null 2>&1 - if PATH="$wd/wrap:$PATH" install -vdm755 "$wd/exists" >/dev/null 2>&1; then - ok "install -d on an existing directory succeeds" - else - bad "install -d on an existing directory still errors" - fi - if PATH="$wd/wrap:$PATH" install -vdm755 "$wd/fresh" >/dev/null 2>&1 \ - && [ -d "$wd/fresh" ]; then - ok "install -d still creates directories that are missing" - else - bad "the wrapper broke directory creation" - fi - echo data > "$wd/src" - if PATH="$wd/wrap:$PATH" install -m644 "$wd/src" "$wd/exists/f" >/dev/null 2>&1 \ - && [ -f "$wd/exists/f" ]; then - ok "installing a file is passed through untouched" - else - bad "the wrapper broke ordinary file installs" - fi - # a real failure must still be a failure - if PATH="$wd/wrap:$PATH" install -d "$wd/nonexistent-parent/sub/deeper/x" \ - 2>/dev/null && [ ! -d "$wd/nonexistent-parent/sub/deeper/x" ]; then - bad "the wrapper reports success for a directory it did not create" - else - ok "a genuine failure is not masked" - fi -fi # ---- a step's manifest lists what it owns, not what happened while it ran --- # # last-step builds wget as the package user 'wget'. Everything written during @@ -3629,5396 +3646,17508 @@ fi # bash parses top to bottom: a function called at line 437 but defined at 466 # does not exist yet, and the build ran without its wrappers: # packagemanager_install: line 437: _make_build_wrappers: command not found -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - python3 - "$pmi" <<'PY60' -import re, sys -lines = open(sys.argv[1]).read().split("\n") -defs, bad = {}, [] -for i, l in enumerate(lines): - m = re.match(r"^([a-zA-Z_][a-zA-Z0-9_]*)\(\) \{", l) - if m: - defs.setdefault(m.group(1), i) -for name, dline in defs.items(): - for i, l in enumerate(lines[:dline]): - if re.search(r"^\s+%s\b" % re.escape(name), l) and not l.strip().startswith("#"): - enclosing = max((d for d in defs.values() if d < i), default=-1) - if enclosing == -1: - bad.append("%s used at line %d, defined at %d" % (name, i + 1, dline + 1)) - break -if bad: - for b in bad: print(" FAIL %s" % b) - sys.exit(1) -print(" PASS every helper is defined before it is used") -PY60 - _count_rc $? - # ---- one implementation of the collector-group rules --------------------- # - # A package installed after boot must behave exactly like one built during - # the chroot stage. Two copies of the rules drift; lfs-helper owns them. - gd="$(sed -n '/^grant_dir_to_user() {/,/^}/p' "$pmi")" - case "$gd" in - *"lfs-helper grant-dir"*) - ok "packagemanager uses lfs-helper's collector-group rules" ;; - *) bad "packagemanager keeps its own copy of the rules" ;; +# ---- one colour vocabulary across the toolchain (1.11.26) ----------------- # +# Red states a failure, orange warns, green states success -- in EVERY tool. +# The audit found: lfs had the painters and 20 bare uncoloured "!" errors +# beside them; blfs had green for commands and NO red at all, so its errors +# were the only uncoloured errors in the set; lfs-sanity printed "!!" plain; +# the engine printed a warning in red. An error that does not look like the +# other tools' errors reads as less serious, which is backwards. +_p="" +# lfs: painters exist and no bare "!" error writes remain +grep -q '^def fail(' "$LFS_TOOL" || _p="$_p;lfs has no fail()" +grep -qE 'sys\.stderr\.write\("!' "$LFS_TOOL" \ + && _p="$_p;lfs still writes uncoloured errors" +# blfs: the same three painters, and -V like every other tool +_b="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_b" ]; then + for fn in fail warn ok; do + grep -q "^def $fn(" "$_b" || _p="$_p;blfs has no $fn()" + done + grep -q '"-V", "--version"' "$_b" || _p="$_p;blfs lacks the -V every other tool has" +fi +# lfs-sanity: a problem is red, a clean verdict green (guarded for pipes) +_ss="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$_ss" ]; then + grep -q '_C_ERR' "$_ss" || _p="$_p;lfs-sanity prints problems without colour" + grep -q 'if \[ -t 1 \]' "$_ss" || _p="$_p;lfs-sanity colours a piped report" +fi +# warnings go to stderr in EVERY tool, not just the bash ones: a note printed +# to stdout lands inside whatever a caller is capturing. The config-conflict +# note was read back as part of the value it stood next to. (1.11.28) +if [ -f "$pmt" ]; then + _pw="$(grep 'def printWarning' "$pmt")" + case "$_pw" in + *sys.stderr*) : ;; + *) _p="$_p;packagemanager warns onto stdout, polluting captured output" ;; esac - case "$gd" in - *"fallback for a system without it"*) - ok "it still works where lfs-helper is not installed" ;; - *) bad "no fallback when lfs-helper is absent" ;; - esac - # the delegation must come FIRST, or the local copy decides - i_del="$(printf '%s' "$gd" | grep -n "lfs-helper grant-dir" | head -1 | cut -d: -f1)" - i_own="$(printf '%s' "$gd" | grep -n 'prefix="\$(collector_prefix_of)"' | head -1 | cut -d: -f1)" - if [ -n "$i_del" ] && [ -n "$i_own" ] && [ "$i_del" -lt "$i_own" ]; then - ok "the shared rules are consulted before the local copy" - else - bad "the local copy runs even when lfs-helper is available" - fi +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "red is failure, orange is warning, green is success -- in every tool" +fi - # ---- one set of wrappers, not two (1.11.3) ------------------------------ # - # The wrappers ARE the rule about what a package user may do: chown skipped, - # chgrp skipped, `install -d` on an existing directory allowed to succeed. - # lfs-helper writes them to /usr/lib/pkgusr, where they outlive the build and - # where the package-user profile already points. This script wrote its own - # 480-line copy into a fresh /tmp directory on EVERY run: a second answer to - # the same question, and the profile pointed at neither of them. - _p="" - grep -q '^_wrapper_dir_from_lfs_helper()' "$pmi" \ - || _p="$_p;there is no way to ask lfs-helper for the wrappers" - grep -q 'lfs-helper wrapper-dir' "$pmi" \ - || _p="$_p;the wrapper directory is assumed rather than asked for" - # every call site must go through the resolver first - _n_ask="$(grep -c '_wrapper_dir_from_lfs_helper' "$pmi")" - _n_own="$(grep -c '_make_build_wrappers "\$_wrapdir"' "$pmi")" - [ "${_n_ask:-0}" -gt "${_n_own:-0}" ] \ - || _p="$_p;a call site still builds its own wrappers without asking" - # and the SHARED directory must never be deleted afterwards -- that would - # take the wrappers away from every package user on the system - grep -q '_wraptmp' "$pmi" \ - || _p="$_p;the cleanup does not distinguish the shared dir from a temporary one" - if grep -q 'rm -rf "$_wrapdir"' "$pmi" && ! grep -q '_wraptmp.*=.*1.*rm -rf "$_wrapdir"' "$pmi"; then - _p="$_p;the wrapper directory is removed unconditionally" - fi - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done +# ---- a collector group belongs in its own range, under one prefix (1.11.25) # +# A real tree ended with +# !! group nimgnu_p_openssl (gid 10093) is outside every convention +# Two faults. `groupadd <name>` with no -g takes the next free system gid, so +# the group landed at 10093, in the middle of the package-user accounts. And +# the name stacked two prefixes: the owner is `p_openssl`, so nimgnu_name gave +# `nimgnu_p_openssl` -- which is not even what the prompt offered. +python3 - "$pm_src" <<'PYGRP' +import sys, re, importlib.machinery as m +pm = m.SourceFileLoader('pm_grp', sys.argv[1]).load_module() +problems = [] +# one prefix: the owner's is stripped before the collector's is added +for given, want_suffix in (("p_openssl", "openssl"), ("cfg_bootscripts", "bootscripts")): + got = pm.nimgnu_name(given) + if not got.endswith(want_suffix) or "_p_" in got or "_cfg_" in got: + problems.append("nimgnu_name(%r) = %r keeps the owner's prefix" % (given, got)) +# an already-collector name is left alone, and audio is special +if pm.nimgnu_name("audio") != "audio": + problems.append("the audio group is renamed") +# the gid comes from the collector range +gid = pm._next_collector_gid() +if gid is None or not (90000 <= gid < 99000): + problems.append("the next collector gid is %r, outside 90000-99000" % gid) +# and every creation path uses it -- a bare groupadd takes the next system gid +src = open(sys.argv[1]).read() +# every CALLER must go through the helper. The helper itself builds the +# command with an optional -g, so exclude its own body from the scan. +_helper = re.search("def _create_collector_group.*?(?=\n\ndef )", src, re.S) +_body = src.replace(_helper.group(0), "") if _helper else src +for m2 in re.finditer("_cmd_path..groupadd..[^]]*]", _body): + frag = m2.group(0) + # the install group is created with an explicit 9999, which is fine + if '"-g"' not in frag: + problems.append("a groupadd without -g remains: %s" % frag[:60]) +if not _helper: + problems.append("there is no _create_collector_group") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a collector group gets one prefix and a gid in its own range") +PYGRP +_count_rc $? + +# the sanity report's summary must agree with its own body +_ss="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$_ss" ]; then + # every "!!" must come from hit(), which counts; an awk that prints its own + # made a report flag a bad group and still say "no problems found". + if grep -qE "awk[^|]*\{print \\?\"!!" "$_ss"; then + bad "lfs-sanity prints a problem that its summary never counts" else - ok "both tools use one set of wrappers, and only a temporary one is deleted" + ok "every problem lfs-sanity prints is counted in its summary" fi +fi + +# ---- the file to edit is named last, and it is the one that survives (1.11.24) # +# The failure said +# install file: /tmp/packagemanager/install_files/install_make-ca-1.16.1 +# three messages before the end. Two problems. That path is where a script +# freshly generated from the book lands; the engine copies it into the package +# user's home and the next run REGENERATES the /tmp one, so an edit there is +# silently discarded -- the exact failure the message exists to prevent. And +# what you do next should not be something you scroll back for. +python3 - "$pm_src" <<'PYEDIT' +import sys, re +src = open(sys.argv[1]).read() +problems = [] +for fn in ("cmd_install", "cmd_update"): + m = re.search(r"def %s\(.*?(?=\n\ndef )" % fn, src, re.S) + if not m: + continue + body = m.group(0) + i_fail = body.find("sys.exit(result.returncode)") + if i_fail < 0: + continue + seg = body[:i_fail] + i_edit = seg.rfind("To change what it does, edit") + if i_edit < 0: + problems.append("%s never says which file to edit" % fn) + continue + # nothing but the exit may follow it + tail = seg[i_edit:] + for later in ("logs:", "Then re-run", "The plan is saved"): + if later in tail: + problems.append("%s prints '%s' after the file to edit" % (fn, later)) + # and it must prefer the copy in the package user's home. The resolution + # happens just BEFORE the message, so look at the whole failure block. + if "pkgusr_home(" not in seg: + problems.append("%s still names the temporary copy" % fn) + if "_canon" not in seg: + problems.append("%s does not resolve the surviving script" % fn) +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a failed install ends by naming the script that survives a re-run") +PYEDIT +_count_rc $? + +# a collector group grants a DIRECTORY, never a package +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + if grep -q "everything owned by '\$owner'" "$_h"; then + bad "the group prompt still claims it grants everything a package owns" + else + ok "the group prompt says what a collector group actually grants" + fi +fi - # the account home is asked for, not re-derived from the layout - _hf="$(sed -n '/^pkgusr_home_for() {/,/^}/p' "$pmi")" - case "$_hf" in - *"lfs-helper pkgusr-home"*) ok "the account home comes from the chokepoint" ;; - *) bad "packagemanager_install re-derives where an account lives" ;; +# ---- two registries, one question (1.11.23) ------------------------------- # +# `packagemanager which-package` is in the README and never existed: +# error: argument <command>: invalid choice: 'which-package' +# And lfs-helper's version reads the BUILD manifests only, so a package +# installed ten minutes earlier by these tools was reported as +# Nothing recorded means it was installed ... outside these tools entirely. +# Wrong twice: they installed it, and they recorded it -- in the package user's +# own pkg.lst, which neither tool consulted. +_wp="$T/which"; mkdir -p "$_wp/pkgusr/p_wget" "$_wp/cfg" +echo "/usr/bin/wget" > "$_wp/pkgusr/p_wget/pkg.lst" +_p="" +_o="$(PKGUSR_BASE="$_wp" python3 "$pm_src" which-package /usr/bin/wget 2>&1)" +case "$_o" in + *"invalid choice"*) _p="$_p;packagemanager still has no which-package" ;; + *p_wget*) ;; + *) _p="$_p;which-package does not find the package that recorded the path" ;; +esac +# the README documents it, so it has to exist +_rm="$(dirname "$LFS_TOOL")/README.md" +if [ -f "$_rm" ] && grep -q 'packagemanager which-package' "$_rm"; then + case "$_o" in *"invalid choice"*) _p="$_p;the README documents a command that does not exist" ;; esac +fi +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _o="$(LFS_PKGUSR_ROOT="$_wp/pkgusr" LFS_CFGUSR_ROOT="$_wp/cfg" \ + bash "$_h" which-package /usr/bin/wget 2>&1)" + case "$_o" in + *"outside these tools entirely"*) + _p="$_p;lfs-helper still calls a tracked package untracked" ;; esac + case "$_o" in + *p_wget*) ;; + *) _p="$_p;lfs-helper does not look in pkg.lst" ;; + esac +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "both tools answer 'which package installed this', from either registry" fi -# ---- the kernel and the bootloader are yours (1.11.6) --------------------- # -# Both are decisions about the whole MACHINE, and getting either wrong costs -# the system you are building on. The build must not do them by default, and -# must not let you find that out after the reboot. +# ---- a wrapper that exists can still be the wrong wrapper (1.11.21) ------- # +# 1.11.18 fixed the install wrapper; 1.11.20 hit the identical failure on the +# identical line. A snapshot had restored the OLD wrapper: it existed and was +# executable, and the check said "missing or incomplete", so the fixed one +# never reached the tree. The comment on that check claimed it repaired a +# tree whose wrappers predate a fix. It could not see "predates". _h="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$_h" ]; then _p="" - grep -q '_say_what_is_yours_to_finish' "$_h" \ - || _p="$_p;the build never says the kernel and bootloader are yours" - _sf="$(sed -n '/^_say_what_is_yours_to_finish() {/,/^}/p' "$_h")" - printf '%s' "$_sf" | grep -qi 'kernel' \ - || _p="$_p;the closing note does not mention the kernel" - printf '%s' "$_sf" | grep -qi 'refind' \ - || _p="$_p;the closing note does not say how to add a bootloader entry" - # it is a note, not an action: nothing in it may write anything - printf '%s' "$_sf" | grep -qE '^\s*(cp|mv|rm|dd|mkfs|chmod|chown|install) ' \ - && _p="$_p;the closing note runs commands instead of printing them" + # the writer stamps what it wrote, with the same fingerprint --version prints + _mw="$(sed -n '/^make_wrappers() {/,/^}$/p' "$_h" | head -30)" + printf '%s' "$_mw" | grep -q '.written-by' \ + || _p="$_p;the wrappers carry no stamp saying who wrote them" + printf '%s' "$_mw" | grep -q '_build_id' \ + || _p="$_p;the stamp does not change when the code does" + # and for real + _wl="$T/wrap-stale"; mkdir -p "$_wl" + LFS_WRAPPERS="$_wl" bash "$_h" make-wrappers --run >/dev/null 2>&1 + [ -f "$_wl/.written-by" ] || _p="$_p;make-wrappers wrote no stamp" + # (the stale-stamp CONSUMER retired with the fifth tool: the one loop + # rewrites its wrappers through make-wrappers itself) if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done else - ok "the build ends by saying the kernel and bootloader are yours" + ok "the wrapper writer stamps what it wrote" fi fi -# and no bootloader is set up unless it was asked for, by name -python3 - "$LFS_TOOL" <<'PYBL' -import sys, re, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +# ---- a checksum failure must say WHICH failure it is (1.11.20) ------------ # +# "ERROR: md5sum check failed" was printed for two different things: a file +# whose contents differ from the book, and a file that is not there at all. +# The second is what a fresh system actually hits, and it is not a checksum +# problem -- it is a download that never happened. +# (the check is the shared runner's now: source it and call it) +python3 - "$PHASES_LIB" <<'PYMD' +import sys, subprocess, tempfile, os +d = tempfile.mkdtemp() +sh = os.path.join(d, "m.sh") +open(sh, "w").write('. "%s"\nmd5_sum=abc\npkg=/nonexistent\n_pp_check_md5\n' % sys.argv[1]) problems = [] -# the default must be "none" with NO config at all -- not "none" because some -# other key happened to be unset -lfs.load_config = lambda: {} -if lfs.bootloader() != "none": - problems.append("the default bootloader is %r, not 'none'" % lfs.bootloader()) -lfs.load_config = lambda: {"bootloader": ""} -if lfs.bootloader() != "none": - problems.append("an empty bootloader setting does not mean 'none'") -# and the prompt must say the machine keeps booting as it does now -src = open(sys.argv[1]).read() -m2 = re.search(r'\("bootloader",.*?\),\n\s*\("esp"', src, re.S) -if m2 and "DEFAULT" not in m2.group(0): - problems.append("the prompt does not say that doing nothing is the default") +r = subprocess.run(["bash", sh], capture_output=True, text=True) +out = r.stdout + r.stderr +if r.returncode == 0: + problems.append("a mismatched checksum no longer fails") +if "expected" not in out or "got" not in out: + problems.append("the failure does not say what it expected or got") +if "<no file>" not in out: + problems.append("a missing file is not distinguished from a bad checksum") +# the override must exist, and must NOT be the default +r = subprocess.run(["bash", sh], capture_output=True, text=True, + env={"PM_SKIP_MD5": "1", "PATH": "/usr/bin:/bin"}) +if r.returncode != 0: + problems.append("PM_SKIP_MD5=1 does not let a mismatch through") +if "PM_SKIP_MD5" not in out: + problems.append("the failure does not mention how to override it") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS no bootloader is touched unless it was asked for by name") -PYBL +print(" PASS a checksum failure names the cause, and can be overridden on purpose") +PYMD _count_rc $? -# ---- $LFS may never be the running system (1.11.5) ------------------------ # -# Every path this tool writes is $LFS/something. With $LFS set to the host -- -# a config holding "/", an `export LFS=/` in the wrong shell, a typo in the -# interview -- `restart --run` deletes the top-level directories of the machine -# you are sitting on, and _verify_lfs_ownership runs `chown -R lfs /usr`. -# The chroot bind mounts were guarded; the tree itself was not. -python3 - "$LFS_TOOL" <<'PYHOST' -import sys, os, tempfile, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +# ---- a zero exit is not a finished job (1.11.19) -------------------------- # +# make-ca writes several stores and exits 0 having failed some of them: +# install: cannot create regular file '/etc/ssl/certdata.txt': Permission denied +# p11-kit: couldn't create file: /etc/ssl/certs/...pem: Unknown error 13 +# Failed!!! +# Extracting GNUTLS server auth certificates to: ...Done! +# /etc/ssl/certs belongs to p_openssl, so p_make-ca could not write it. One +# bundle existed elsewhere, _have_ca_bundle said yes, bootstrap reported +# complete, and certificate checking was turned back ON with an empty OpenSSL +# trust store -- which is the store wget itself reads. +python3 - "$pm_src" <<'PYCA' +import sys, os, re, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm_ca', sys.argv[1]).load_module() problems = [] -for fn in ("_is_host_path", "_refuse_host_tree"): - if not hasattr(lfs, fn): - problems.append("there is no %s" % fn) +# p11-kit reports errno 13 as a NUMBER, so a filter looking for "denied" +# skipped the line and the grant found nothing to grant. +log = os.path.join(tempfile.mkdtemp(), "make-ca.log") +open(log, "w", encoding="utf-8").write( + "install: cannot create regular file '/etc/ssl/certdata.txt': Permission denied\n" + "p11-kit: couldn't create file: /etc/ssl/certs/GlobalSign.pem: Unknown error 13\n" + "Extracting GNUTLS server auth certificates to: ...Done!\n") +dirs = [pm._dir_of(p) for p in pm._paths_from_permission_errors(log)] +for want in ("/etc/ssl", "/etc/ssl/certs"): + if want not in dirs: + problems.append("%s is not extracted from make-ca's output" % want) + +src = open(sys.argv[1]).read() +if not re.search(r"^def _run_make_ca\(", src, re.M): + problems.append("make-ca is run without any recovery") +else: + b = re.search(r"def _run_make_ca\(.*?(?=\n\ndef )", src, re.S).group(0) + if "auto_grant_from_log" not in b: + problems.append("a store it could not write is never granted") + if "retry=False" not in b: + problems.append("it never tries again after granting") + if "capture_output" not in b: + problems.append("the output is not read, so nothing can be granted from it") + +# THE TRIGGER MUST MATCH WHAT COUNTS AS DONE. 1.11.19 made "finished" mean a +# bundle AND no empty store, but left stage 5 running only when there was no +# bundle at all. On a tree with the GnuTLS bundle written and /etc/ssl/certs +# empty, stage 5 was skipped and the empty store was then reported as a +# failure -- correctly, and without ever having been attempted. + +# and success must mean the stores are POPULATED, not merely present +if not re.search(r"^def _empty_cert_stores\(", src, re.M): + problems.append("nothing checks that a store has certificates in it") +fin = re.search(r"def _bootstrap_finish\(.*?(?=\n\ndef )", src, re.S).group(0) +if "_empty_cert_stores" not in fin: + problems.append("bootstrap reports success without checking the stores") +_i5 = fin.find('printHeader("5. the certificate bundle")') +_trigger = fin[max(0, _i5 - 200):_i5] if _i5 >= 0 else "" +if _i5 < 0: + problems.append("stage 5's trigger cannot be found") +elif "_empty_cert_stores()" not in _trigger: + problems.append("stage 5 is skipped when the bundle exists but a store is empty") +i_chk = fin.find("_empty_cert_stores") +i_res = fin.find("restore_wget_verification") +if i_chk >= 0 and i_res >= 0 and i_res < i_chk: + problems.append("verification is re-enabled before the stores are checked") + if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) +print(" PASS an empty trust store is a failure, however make-ca exited") +PYCA +_count_rc $? -# the host, by every spelling -for p in ("/", "/usr", "/etc", "/boot", "/var", "/home", "/bin", "/lib"): - if not lfs._is_host_path(p): - problems.append("%s is not recognised as the running system" % p) - -# by another name: a symlink to / -d = tempfile.mkdtemp() -link = os.path.join(d, "root-link") -try: - os.symlink("/", link) - if not lfs._is_host_path(link): - problems.append("a symlink to / is not recognised") -except OSError: - pass - -# a path INSIDE a system tree is the system too, however deep -for p in ("/usr/src/lfs", "/etc/lfs", "/var/tmp/lfs"): - if not lfs._is_host_path(p): - problems.append("%s is not recognised as part of the system" % p) +# ---- the install wrapper must see bundled short options (1.11.18) --------- # +# The wrapper detected directory mode with +# case "$a" in -d|--directory) want_dirs=1 ;; +# which matches -d only as a WHOLE argument. make-ca's Makefile writes +# install -vdm755 /usr/sbin +# in one token, so the rule never fired, the real install(1) ran, and it died on +# the exact message this wrapper's own comment quotes. `install -d` on an +# EXISTING directory still applies the mode, and chmod needs ownership rather +# than write permission -- which is why three rounds of granting groups could +# not help. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _w="$T/wrapper-install" + # lift the wrapper exactly as lfs-helper writes it, and run it for real + sed -n "/cat > \"\$WRAPPERS\/install\" <<'EOF'/,/^EOF$/p" "$_h" \ + | sed '1d;$d' > "$_w" + if [ -s "$_w" ]; then + printf '#!/bin/bash\necho "REAL $*"\n' > "$T/realinstall" + chmod +x "$T/realinstall" "$_w" + sed -i "s|^real=/usr/bin/install$|real=$T/realinstall|" "$_w" + mkdir -p "$T/exists"; rm -rf "$T/fresh" + _p="" + # the reported failure: bundled -d must be recognised, and an existing + # directory left alone + _o="$(bash "$_w" -vdm755 "$T/exists" 2>&1)" + case "$_o" in *REAL*) _p="$_p;bundled -vdm755 still reaches the real install" ;; esac + _o="$(bash "$_w" -dv "$T/exists" 2>&1)" + case "$_o" in *REAL*) _p="$_p;bundled -dv still reaches the real install" ;; esac + # the unbundled spellings must keep working + _o="$(bash "$_w" -m 0755 -d "$T/exists" 2>&1)" + case "$_o" in *REAL*) _p="$_p;-m 0755 -d regressed" ;; esac + _o="$(bash "$_w" --directory "$T/exists" 2>&1)" + case "$_o" in *REAL*) _p="$_p;--directory regressed" ;; esac + # a directory that does NOT exist must still be created + bash "$_w" -vdm755 "$T/fresh" >/dev/null 2>&1 + [ -d "$T/fresh" ] || _p="$_p;a missing directory is no longer created" + # an ordinary file install must still reach install(1), with ownership + # stripped and the mode intact. The source has to EXIST: since 1.12.19 an + # install whose sources are all missing is a skipped optional artifact, + # not a failure, and this fixture never created its file. + : > "$T/f" + _o="$(bash "$_w" -vm644 -o root -g root "$T/f" "$T/exists/" 2>&1)" + case "$_o" in + *"REAL"*) case "$_o" in + *"-m 644"*) ;; + *) _p="$_p;a bundled mode is lost on a file install" ;; + esac ;; + *) _p="$_p;an ordinary file install no longer runs" ;; + esac + case "$_o" in *"REAL"*"-o root"*) _p="$_p;ownership is no longer stripped" ;; esac + # setuid must be refused in EITHER spelling -- bundled slipped past before + _o="$(bash "$_w" -vm4755 f "$T/exists/" 2>&1)" + case "$_o" in *"-m 4755"*"755 instead"*) ;; *) _p="$_p;bundled setuid is not refused" ;; esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the install wrapper reads -vdm755 the same as -d -m 755" + fi + fi +fi -# ...and a real build tree must still be allowed, even though it has the same -# layout. A check that refuses /mnt/lfs/usr refuses every build there is. -tree = os.path.join(d, "lfs") -os.makedirs(os.path.join(tree, "usr"), exist_ok=True) -os.makedirs(os.path.join(tree, "etc"), exist_ok=True) -for p in (tree, "/mnt/lfs", "/home/someone/lfs", "/media/disk/lfs"): - if lfs._is_host_path(p): - problems.append("a legitimate build tree is refused: %s" % p) +# ---- granting cannot fix what is not a group problem (1.11.17) ------------ # +# The grant fired, worked out the right directory, and then: +# /usr/sbin is already group install -- adding 'make-ca' to it +# usermod: user 'make-ca' does not exist +# Two faults in three lines. The account name was the PACKAGE name again -- +# third appearance of that species in this file. And the user was already in +# `install`: /usr/sbin is root:install and the failure was `install -vdm755` +# CHMODing a directory root owns, which is the wrappers' job, not the group's. +# Granting a group someone already holds looks like progress and is not. +if [ -f "$pm_src" ]; then + _p="" + _loop="$(sed -n '/^def run_pm_install(/,/^def /p' "$pm_src")" + printf '%s' "$_loop" | grep -q 'auto_grant_from_log(pkgusr_name(name)' \ + || _p="$_p;the grant is still applied to the package name" + _ag="$(sed -n '/^def auto_grant_from_log(/,/^def /p' "$pm_src")" + printf '%s' "$_ag" | grep -q 'user_supplementary_groups' \ + || _p="$_p;a group the user already has is granted again" + printf '%s' "$_ag" | grep -q 'make-wrappers' \ + || _p="$_p;nothing points at the wrappers when the group is not the problem" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a grant goes to the account, and is not offered where it cannot help" + fi +fi -# ONE rule, not two: the interview and the hard stop must agree, or a path -# refused by one is accepted by the other. -src = open(sys.argv[1]).read() +# ---- the auto-grant must recognise the message it is given (1.11.16) ------ # +# The recovery lfs-helper does during the base build exists here too -- a +# package that must write into another package's directory joins its collector +# group -- but it never fired on the real failure: +# install: cannot change permissions of \u2018/usr/sbin\u2019: Operation not permitted +# coreutils quotes with the locale's style (U+2018/U+2019 under UTF-8) and the +# patterns matched ASCII apostrophes only, so no directory was extracted and +# packagemanager printed the manual command as though it had no opinion. +python3 - "$pm_src" <<'PYPERM' +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm_perm', sys.argv[1]).load_module() +problems = [] +log = os.path.join(tempfile.mkdtemp(), "install.log") +open(log, "w", encoding="utf-8").write( + "install -vdm755 /usr/sbin\n" + "install: cannot change permissions of \u2018/usr/sbin\u2019: Operation not permitted\n" + "cp: cannot create regular file '/usr/lib/foo': Permission denied\n" + "mkdir: cannot create directory \u201c/usr/share/bar\u201d: Permission denied\n" + "gcc -O2 -o thing thing.c\n") +got = pm._paths_from_permission_errors(log) +for want in ("/usr/sbin", "/usr/lib/foo", "/usr/share/bar"): + if want not in got: + problems.append("%s was not extracted from the log" % want) +# and an ordinary build line must not look like a permission failure +if any("thing" in p for p in got): + problems.append("a plain compile line was read as a permission error") +# the retry loop has to consult it, or extracting changes nothing import re -body = re.search(r"def _refuse_host_tree\(lfs\):.*?\n\n\ndef ", src, re.S) -if body and "_is_host_path" not in body.group(0): - problems.append("the hard stop keeps its own copy of the rule") -if "_is_host_path" not in re.search(r"for k, desc, default in _SESSION_KEYS:.*?\n\n", src, re.S).group(0): - problems.append("the interview accepts a mount point the tools will refuse") +src = open(sys.argv[1]).read() +loop = re.search(r"def run_pm_install\(.*?(?=\n\ndef )", src, re.S).group(0) +if "auto_grant_from_log" not in loop: + problems.append("the install never retries after granting") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a permission failure is recognised however coreutils quotes it") +PYPERM +_count_rc $? -# the destructive paths must go through it -for fn in ("require_lfs_for_root", "_verify_lfs_ownership"): - b = re.search(r"def %s\(.*?(?=\n\ndef )" % fn, src, re.S) - if b and "_refuse_host_tree" not in b.group(0): - problems.append("%s does not check the tree is not the host" % fn) +# ---- --run means run, not print (1.11.15) --------------------------------- # +# Stages 3-5 were PRINTED as commands to type, on the reasoning that each has +# to be seen to work before the next is worth trying. That is a poor trade: +# `bootstrap --run` is someone saying "get this system ready", and answering +# with three commands to type refuses the request while looking like an answer. +python3 - "$pm_src" <<'PYRUN' +import sys, re +src = open(sys.argv[1]).read() +problems = [] +if not re.search(r"^def _bootstrap_finish\(", src, re.M): + print(" FAIL nothing does the stages that need a network") + sys.exit(1) +fin = re.search(r"def _bootstrap_finish\(.*?(?=\n\ndef )", src, re.S).group(0) + +# each stage must actually be attempted +for what, needle in (("the book", "fetch"), + ("make-ca", "_install_via_self"), + ("the bundle", "-g")): + if needle not in fin: + problems.append("%s is still only printed" % what) +# make-ca through the ordinary path, with its dependencies +if "recursive=True" not in fin: + problems.append("make-ca is installed without resolving its dependencies") +# --force, or make-ca reports "No update required!" and writes nothing. +# Moved into _run_make_ca in 1.11.19, so look there too. +_mk = re.search(r"def _run_make_ca\(.*?(?=\n\ndef )", src, re.S) +if '"--force"' not in fin + (_mk.group(0) if _mk else ""): + problems.append("the bundle is generated without --force, so it may write nothing") +# a stage whose dependency failed must be SKIPPED, not attempted and blamed +if fin.count("Skipped") < 2: + problems.append("a stage runs even when what it depends on failed") +# and verification comes back on the moment there is something to verify with +if "restore_wget_verification" not in fin: + problems.append("certificate checking is never turned back on") + +# the dry run still explains, rather than doing +body = re.search(r"def cmd_setup\(.*?(?=\n\ndef )", src, re.S).group(0) +if "_bootstrap_finish" not in body: + problems.append("the run path never reaches the later stages") +if "_bootstrap_next_steps" not in body: + problems.append("the dry run no longer says what would happen") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS $LFS is refused when it is the running system, by any spelling") -PYHOST +print(" PASS bootstrap --run does every stage it can, and skips what it cannot") +PYRUN _count_rc $? -# ---- an answer that is collected must be acted on, or said (1.11.4) ------- # -# `strip` is asked for during the interview and nothing does it. The prompt -# says NOT YET IMPLEMENTED, but that was six hours and a whole build ago, and -# the build otherwise ends as though every answer had been acted on. -_h="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_h" ]; then +# ---- the tarball is staged where the home exists (1.11.14, fixed 1.11.20) - # +# get-sources downloads every tarball to /sources before the reboot. The build +# looked only in the build directory, so the first install on a fresh system +# reached for a download tool in order to install that download tool: +# install_p_wget: line 44: wget: command not found +# md5sum: wget-1.25.0.tar.gz: No such file or directory +# +# 1.11.14 put the staging in packagemanager, before the engine ran, guarded by +# "only if the home exists". For a NEW package it never does -- the engine +# creates it -- so the guard made the fix a no-op in exactly the case it was +# written for, and the same error came back unchanged. It belongs in the +# engine, after the account is made. +if [ -f "$helper_src" ]; then _p="" - grep -q '_warn_if_strip_was_asked_for' "$_h" \ - || _p="$_p;nothing tells you the strip answer was ignored" - # it must READ the answer, not assume it - _sf="$(sed -n '/^_warn_if_strip_was_asked_for() {/,/^}/p' "$_h")" - printf '%s' "$_sf" | grep -q 'LFS_STRIP' \ - || _p="$_p;the strip notice does not read LFS_STRIP" - # and only when it was actually asked for - printf '%s' "$_sf" | grep -q 'return 0' \ - || _p="$_p;the strip notice fires even when nobody asked for it" - # said at the END, with the other things you still have to do - _i1="$(grep -n '_warn_if_strip_was_asked_for$' "$_h" | tail -1 | cut -d: -f1)" - _i2="$(grep -n '^ _warn_if_no_login$' "$_h" | tail -1 | cut -d: -f1)" - [ -n "$_i1" ] && [ -n "$_i2" ] \ - || _p="$_p;the notice is not part of the build's last word" + grep -q '^_pm_stage_source()' "$helper_src" \ + || _p="$_p;nothing stages a downloaded tarball" + _sf="$(_slice_fn "$helper_src" _pm_stage_source)" + printf '%s' "$_sf" | grep -q 'LFS_SOURCES_DIR:-/sources' \ + || _p="$_p;the sources directory is not consulted" + # COPIED, never moved: /sources is the record of what was downloaded + printf '%s' "$_sf" | grep -qE '^\s*cp "\$_d/\$_pkg"' \ + || _p="$_p;the tarball is not copied in" + printf '%s' "$_sf" | grep -qE '\bmv "\$_d' \ + && _p="$_p;the build consumes the file out of /sources" + # it must run AFTER the account exists, or the home it copies into is not there + _ip="$(_slice_fn "$helper_src" cmd_pm_install)" + _i_add="$(printf '%s' "$_ip" | grep -n '_pm_add_account' | head -1 | cut -d: -f1)" + _i_stg="$(printf '%s' "$_ip" | grep -n '_pm_stage_source' | head -1 | cut -d: -f1)" + if [ -z "$_i_stg" ]; then + _p="$_p;pm-install never stages anything" + elif [ -n "$_i_add" ] && [ "$_i_stg" -lt "$_i_add" ]; then + _p="$_p;staging runs before the account and its home exist" + fi + # and it must run for real + _sbx="$T/stage"; mkdir -p "$_sbx/src" "$_sbx/home" + echo TARBALL > "$_sbx/src/wget-1.25.0.tar.gz" + printf 'name="wget"\npkg="wget-1.25.0.tar.gz"\n' > "$_sbx/script" + { + echo 'ok(){ :; }' + echo "_recorded_home_for(){ echo \"$_sbx/home\"; }" + printf '%s\n' "$_sf" + echo "_pm_stage_source p_wget $_sbx/script" + } > "$_sbx/run.sh" + LFS_SOURCES_DIR="$_sbx/src" bash "$_sbx/run.sh" >/dev/null 2>&1 + [ -f "$_sbx/home/wget-1.25.0.tar.gz" ] \ + || _p="$_p;the tarball did not reach the package user's home" + [ -f "$_sbx/src/wget-1.25.0.tar.gz" ] \ + || _p="$_p;/sources lost the file the build used" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done else - ok "a build that ignored your strip answer says so at the end" + ok "a downloaded tarball is staged into the home pm-install just made" fi fi -# ---- a suppressed failure is a bug waiting to be found (1.11.4) ----------- # -# `cmd 2>/dev/null || true` has cost this project more debugging than anything -# else: a tree came out with every package home root:root 755 and not one line -# said why. The pattern is not banned -- mkdir -p on a directory that exists is -# fine -- but a failure that CHANGES WHAT THE SYSTEM LOOKS LIKE must be said. -_h="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_h" ]; then +# ---- the account's home is settled where the account is made (1.11.13) ---- # +# add_package_user takes no home argument: it uses /usr/src/<name>, while every +# other tool puts package users in /usr/src/pkgusr/<name>. So the account was +# created, a home was created, and the install then ran in a third place that +# existed in neither: +# tee: /usr/src/p_wget/log/...: No such file or directory +# bash: /usr/src/p_wget/install_p_wget: No such file or directory +# packagemanager reported the right path in the same breath -- logs: +# /usr/src/pkgusr/p_wget/log -- which is the two halves disagreeing, not a +# missing directory. +if [ -f "$helper_src" ]; then _p="" - grep -q '^soft() {' "$_h" || _p="$_p;there is no way to report a non-fatal failure" - # the ownership and permission calls are the ones that matter - _leak="$(grep -nE '(real_chmod|real_chown|real_chgrp|chown|chgrp) .*2>/dev/null \|\| true' "$_h" \ - | grep -v 'SUPPRESSED DELIBERATELY' | wc -l)" - # ANY comment above is not enough -- the surrounding prose explains what the - # call does, not why its failure may be thrown away, and a reintroduced - # suppression inherits whatever comment happened to be there. The line - # immediately above must say which of the two cases this is, in those words. - _undoc=0 - while IFS= read -r _ln; do - [ -n "$_ln" ] || continue - _n="${_ln%%:*}" - sed -n "$((_n-1))p" "$_h" \ - | grep -qE 'SUPPRESSED DELIBERATELY|benign:' || _undoc=$((_undoc+1)) - done <<< "$(grep -nE '(real_chmod|real_chown|real_chgrp) .*2>/dev/null \|\| true' "$_h")" - [ "${_undoc:-0}" = 0 ] \ - || _p="$_p;$_undoc suppressed ownership call(s) are not marked deliberate or benign" - # and the two that are deliberate must say so in those words - grep -q 'SUPPRESSED DELIBERATELY' "$_h" \ - || _p="$_p;the deliberate suppressions are not marked as such" + _au="$(_slice_fn "$helper_src" _pm_add_account)" + # the ONE repair runs at the moment the account is made + printf '%s' "$_au" | grep -q '_repair_misplaced_home' \ + || _p="$_p;creating an account does not settle where it lives" + # and the value computed BEFORE creation must be re-read after it + _ip="$(_slice_fn "$helper_src" cmd_pm_install)" + printf '%s' "$_ip" | grep -q '_now_home' \ + || _p="$_p;the home is still whatever was guessed before the user existed" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done else - ok "every suppressed ownership failure says why it is suppressed" + ok "the home is settled where the account is made, then re-read" fi fi -# lfs-helper must actually offer the doors the other tool now knocks on. -_h="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$_h" ]; then - _p="" - for _c in wrapper-dir make-wrappers grant-dir pkgusr-home owner-name; do - grep -qE "^ $_c\)" "$_h" \ - || _p="$_p;lfs-helper has no '$_c' command" - done - # `case` takes the first match, so a repeated entry is unreachable code - # pretending to be a second decision. Six of them were. - _dups="$(sed -n '/^case "${1:-}" in/,/^esac/p' "$_h" \ - | grep -oE '^ [a-z][a-z0-9|-]*\)' | sort | uniq -d)" - [ -z "$_dups" ] || _p="$_p;the dispatch repeats: $(echo $_dups | tr -d ')')" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "every shared decision is reachable as one lfs-helper command" - fi +# ---- the install engine is handed an ACCOUNT, not a package (1.11.11) ----- # +# Argument 1 of packagemanager_install is used for exactly one thing: the user +# to create, own the files and build as. The bare package name was passed, so: +# $ /usr/bin/packagemanager_install wget /tmp/.../install_Wget-1.25.0 +# id wget || addUser wget +# uid=10089(wget) ... bash: /usr/src/wget/install_wget: No such file +# An account literally called `wget`, in /usr/src/wget -- no prefix, no +# subdirectory, colliding with any real account of that name. A fresh tree +# proved it was created by the install: `id wget` said no such user before. +if [ -f "$pm_src" ]; then + _p="" + _once="$(sed -n '/^def _run_pm_install_once(/,/^def /p' "$pm_src")" + printf '%s' "$_once" | grep -q 'pkgusr_name(name)' \ + || _p="$_p;the engine is still handed a bare package name" + # and the engine resolves defensively, for callers older than this fix + _ip="$(_slice_fn "$helper_src" cmd_pm_install)" + printf '%s' "$_ip" | grep -q 'pkg_owner_name' \ + || _p="$_p;pm-install trusts the name it is handed instead of resolving it" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an install creates a prefixed account, whoever calls it" + fi + + # ---- an already-downloaded tarball is used, not re-fetched ---------------- # + # get-sources puts every tarball in /sources before the reboot. The unpack + # step looked only in the build directory, so the FIRST install on a fresh + # system reached for wget -- the package it was installing -- over a + # connection it cannot verify, because make-ca needs that same wget. + # (the template's unpack is the shared runner's, so that is what is read) + _up="$(sed -n '/^_pp_unpack() {/,/^}/p' "$PHASES_LIB")" + _p="" + printf '%s' "$(sed -n '/^_pp_layout() {/,/^}/p' "$PHASES_LIB")" \ + | grep -q 'LFS_SOURCES_DIR' \ + || _p="$_p;the unpack step never looks in the sources directory" + printf '%s' "$(sed -n '/^_pp_stage_file() {/,/^}/p' "$PHASES_LIB")" \ + | grep -q 'cp -p "\$d/\$f"' \ + || _p="$_p;the staged tarball is not copied in" + # COPIED or linked, never moved: /sources is the record of what was downloaded + printf '%s' "$(sed -n '/^_pp_stage_file() {/,/^}/p' "$PHASES_LIB")" \ + | grep -qE 'mv "\$d' \ + && _p="$_p;the build consumes the file out of /sources" + # and the download must still happen when nothing is staged + printf '%s' "$_up" | grep -q '_pp_fetch "\$link"' \ + || _p="$_p;an unstaged package can no longer be downloaded" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an install uses the tarball get-sources already downloaded" + fi fi -# ---- editing the wrong install script fails silently ------------------------ # -# A package user's home holds two files with nearly identical names: -# install_<name>-<version> the source -- the one to edit -# install_<name> a runtime copy, overwritten every run -# Editing the second changes nothing, the install repeats unchanged, and it -# looks as though the tool threw the edit away. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def _warn_if_wrong_file_edited" "$pmt" \ - && ok "an edit to the runtime copy is pointed out" \ - || bad "editing the wrong file goes unmentioned" - grep -q "To change what it does, edit:" "$pmt" \ - && ok "a failure names the file to edit" \ - || bad "a failure gives no path to edit" +# ---- one key, one meaning: pkgusr_home (1.11.10) -------------------------- # +# `lfs` writes pkgusr_home as the FULL package-user root -- its LAYOUT_DEFAULTS +# say /usr/src/pkgusr -- and packagemanager read the same key as the BASE and +# appended the subdirectory itself. A tree configured by lfs got both: +# logs: /usr/src/pkgusr/pkgusr/p_wget/log +python3 - "$pm_src" "$LFS_TOOL" <<'PYBD' +import sys, os, tempfile, importlib.machinery as m +problems = [] +d = tempfile.mkdtemp() +conf = os.path.join(d, "pm.conf") + +def home_for(value, name="wget"): + open(conf, "w").write("pkgusr_home=%s\n" % value) + os.environ["PKGUSR_CONFIG"] = conf + for k in ("PKGUSR_BASE", "PKGUSR_SUBDIR"): + os.environ.pop(k, None) + mod = m.SourceFileLoader("pm_%d" % abs(hash(value)), sys.argv[1]).load_module() + return mod.pkgusr_home(name) + +# what lfs actually writes +for v in ("/usr/src/pkgusr", "/usr/src/pkgusr/"): + got = home_for(v) + if got != "/usr/src/pkgusr/p_wget": + problems.append("pkgusr_home=%s gives %s" % (v, got)) +# a base that does not end in the subdirectory is taken literally +if home_for("/opt/lfs") != "/opt/lfs/pkgusr/p_wget": + problems.append("a plain base is no longer taken literally") +if home_for("/opt/lfs/pkgusr") != "/opt/lfs/pkgusr/p_wget": + problems.append("a relocated full root is not understood") + +# and lfs must still write the full root -- this fixes the READING, so that +# both tools agree without either changing what it writes +lfs_src = open(sys.argv[2]).read() +if '"target_pkgusr_home": "/usr/src/pkgusr"' not in lfs_src: + problems.append("lfs no longer writes the full package-user root") - # behaviour: warn only when the runtime copy is genuinely newer - python3 - "$pmt" <<'PY61' -import sys, os, time, tempfile, importlib.machinery as m -pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() -base = tempfile.mkdtemp() -pm.BASE_DIR = base -# the directory on disk is the USER's, so build the fixture through the same -# helper the code uses -- the package "p" becomes the account p_p, and -# hard-coding "p" here made this test fail the moment prefixes were turned on -# resolve the home the same way the code does -- the account is p_p AND it -# lives under the package root, so neither part can be hard-coded here -_h = pm.pkgusr_home("p") -os.makedirs(_h) -src = os.path.join(_h, "install_p-1.0") -rt = os.path.join(_h, "install_p") -open(src, "w").write("real") -time.sleep(0.02) -open(rt, "w").write("edited") -import io, contextlib -buf = io.StringIO() -with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - pm._warn_if_wrong_file_edited("p", src) -warned = "runtime copy" in buf.getvalue() -os.utime(src, None) # now the source is newer -buf2 = io.StringIO() -with contextlib.redirect_stdout(buf2), contextlib.redirect_stderr(buf2): - pm._warn_if_wrong_file_edited("p", src) -quiet = "runtime copy" not in buf2.getvalue() -if warned and quiet: - print(" PASS it warns only when the runtime copy is the newer file") -else: - print(" FAIL warned=%s quiet=%s" % (warned, quiet)) +if problems: + for p in problems: print(" FAIL %s" % p) sys.exit(1) -PY61 - _count_rc $? +print(" PASS lfs and packagemanager read pkgusr_home the same way") +PYBD +_count_rc $? + +# ---- a home that is recorded must be the home that exists (1.11.9) -------- # +# add_package_user takes no home argument, so the hint's script uses its own +# default -- /usr/src/<name> -- while init_package_user_home creates and chowns +# the one this tool computes. A real run said both in the same report: +# $ chown -R -h p_urllib3:p_urllib3 /usr/src/pkgusr/p_urllib3 +# home directory created: /usr/src/p_urllib3 +# `su - p_urllib3` then lands in a directory that does not exist. +if [ -f "$pm_src" ]; then + _p="" + _cb="$(sed -n '/^def create_package_user(/,/^def /p' "$pm_src")" + printf '%s' "$_cb" | grep -q '_fix_recorded_home' \ + || _p="$_p;nothing reconciles the recorded home with the real one" + # the useradd branch already passed -d; the add_package_user one did not + printf '%s' "$_cb" | grep -q '"-d", home' \ + || _p="$_p;the useradd fallback no longer sets the home" + _fb="$(sed -n '/^def _fix_recorded_home(/,/^def /p' "$pm_src")" + printf '%s' "$_fb" | grep -q 'usermod' \ + || _p="$_p;the recorded home is never corrected" + printf '%s' "$_fb" | grep -q 'pw_dir' \ + || _p="$_p;it does not read what was actually recorded" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package user's passwd home is the directory its files are in" + fi fi -# ---- installing make-ca does not create any certificates -------------------- # -# It installs the TOOL. Until it is run there is still no CA bundle, wget -# still cannot verify anything, and the /etc/wgetrc workaround stays -- but the -# install reported success, so the missing step is easy to overlook. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def _hint_generate_ca_bundle" "$pmt" \ - && ok "the missing make-ca -g step is pointed out" \ - || bad "make-ca installs and nothing says to run it" - hg="$(sed -n '/^def _hint_generate_ca_bundle/,/^def /p' "$pmt")" - case "$hg" in - *"make-ca -g"*) ok "it gives the exact command" ;; - *) bad "the hint does not say what to run" ;; - esac - # silent once the certificates exist - case "$hg" in - *"certificates exist already"*) ok "it stops once certificates exist" ;; - *) bad "the hint would repeat forever" ;; - esac +# ---- the BLFS book has to be fetched where there is a network (1.11.9) ---- # +# The circle a real build walked into: +# wget: there is no BLFS book on this system to look it up in. +# Could not install: wget +# The built system installs wget by looking it up in the BLFS book, and fetches +# the book with the wget it cannot install. The host has a network and +# install-tools already copies a cached book into the tree -- all that was +# missing was ASKING, at a moment when the answer can still be acted on. +python3 - "$LFS_TOOL" <<'PYBB' +import sys, os, re, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] - # ---- the book's init flavour must match the system ----------------------- # - # A systemd book emits `systemctl` in its install scripts; on a SysV system - # every affected package fails mid-install and has to be hand-edited. - grep -q "^def _warn_init_mismatch" "$pmt" \ - && ok "a systemd/SysV book mismatch is reported" \ - || bad "the wrong book flavour goes unmentioned" - wm="$(sed -n '/^def _warn_init_mismatch/,/^def /p' "$pmt")" - case "$wm" in - *"if book_systemd == have_systemd"*) - ok "it says nothing when the book matches" ;; - *) bad "the mismatch warning would always fire" ;; - esac -fi +# the detector looks in BLFS's own store, not lfs's +d = tempfile.mkdtemp() +os.environ["BLFS_STORE"] = d +if lfs._blfs_book_on_host(): + problems.append("an empty store is reported as having a book") +os.makedirs(os.path.join(d, "books")) +open(os.path.join(d, "books", "BLFS-BOOK-12.4-nochunks.html"), "w").close() +if not lfs._blfs_book_on_host(): + problems.append("a cached BLFS book is not found") + +# it must be a checklist step, and it must come BEFORE the sources: get-sources +# asks the book where wget's tarball is, so with no book that URL is silently +# absent from the download list and the gap only shows on the built system. +body = re.search(r"def _next_steps\(.*?\n\ndef ", src, re.S).group(0) +i_book = body.find("fetch the BLFS book") +i_src = body.find("download the sources") +if i_book < 0: + problems.append("fetching the BLFS book is not a step at all") +elif i_src >= 0 and i_book > i_src: + problems.append("the book is fetched after the sources it is needed for") + +# declining must be remembered, or every resume asks again +h = re.search(r"def _run_choose_blfs_book\(.*?\n\ndef ", src, re.S) +if not h: + problems.append("there is no handler for the decision") +else: + hb = h.group(0) + if "blfs_book" not in hb or "save_config" not in hb: + problems.append("declining the book is not remembered") + if "skip" not in hb: + problems.append("the book cannot be declined") + # and it must actually fetch, not just print advice + if '"fetch"' not in hb: + problems.append("saying yes does not fetch anything") -# ---- make-ca installed is not the same as certificates generated ------------ # -# `make-ca -g` prints "No update required!" and generates nothing when it -# considers its copy of certdata.txt current -- which it is straight after -# install. The install then looks successful while the system still has no -# certificate bundle, and every HTTPS fetch keeps warning. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def check_make_ca_generated" "$pmt" \ - && ok "an installed-but-ungenerated make-ca is noticed" \ - || bad "make-ca can be installed with no certificates and nothing said" - grep -q "make-ca -g --force" "$pmt" \ - && ok "it gives the command that actually generates them" \ - || bad "it does not say how to finish the job" - # and it must go quiet once a bundle exists - python3 - "$pmt" <<'PY62' -import sys, os, io, contextlib, tempfile, importlib.machinery as m -pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the BLFS book is fetched on the host, before the sources need it") +PYBB +_count_rc $? + +# ---- do not ask for what the system already knows (1.11.8) ---------------- # +# `packagemanager setup` opened with a wizard asking for the collector prefix, +# the application-user prefix and the main user -- on a system built by these +# tools, where `lfs build-system session` had collected every one of them, +# written them into config.json and copied that into the chroot. +# +# Being asked for a value the machine is holding teaches you the answer does +# not matter. It matters a great deal: a collector prefix that disagrees with +# the built system gives two parallel sets of groups over the same +# directories, with no error to say so. +if [ -f "$pm_src" ]; then +python3 - "$pm_src" <<'PYFR' +import sys, os, json, tempfile, importlib.machinery as m +problems = [] d = tempfile.mkdtemp() -fake = os.path.join(d, "make-ca") -open(fake, "w").write("#!/bin/sh\nexit 0\n") -os.chmod(fake, 0o755) -os.environ["PATH"] = d + os.pathsep + os.environ["PATH"] -pm._CA_BUNDLES = [os.path.join(d, "bundle.crt")] +store = os.path.join(d, "store"); os.makedirs(store) +json.dump({"collector_prefix": "nimgnu", "pkgusr_prefix": "p", + "cfguser_prefix": "cfg", "main_user": "nim"}, + open(os.path.join(store, "config.json"), "w")) +conf = os.path.join(d, "pm.conf") +os.environ["PKGUSR_CONFIG"] = conf +os.environ["LFS_STORE"] = store +for k in ("LFS_COLLECTOR_PREFIX", "LFS_MAIN_USER", "LFS_PKGUSR_PREFIX"): + os.environ.pop(k, None) +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() + +# it must not read stdin at all when everything is known +import io, contextlib +def _no(*a, **k): + raise AssertionError("asked a question it had the answer to") +pm.input = _no +_buf = io.StringIO() +try: + with contextlib.redirect_stdout(_buf): + pm.first_run_setup() +except AssertionError as e: + problems.append(str(e)) + +got = dict(l.strip().split("=", 1) for l in open(conf) + if "=" in l and not l.startswith("#")) +for k, v in (("collector_prefix", "nimgnu"), ("pkgusr_prefix", "p"), + ("main_user", "nim")): + if got.get(k) != v: + problems.append("%s was saved as %r, not %r" % (k, got.get(k), v)) + +# the DEFAULT is not an opinion. The conflict warning compared the lfs config +# against whatever was in _config, and a built-in default counted -- so a +# system with no packagemanager.conf was told its settings disagreed with +# themselves. +os.remove(conf) +pm._config = None buf = io.StringIO() with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): - pm.check_make_ca_generated() -warned = "no CA certificate bundle" in buf.getvalue() -open(os.path.join(d, "bundle.crt"), "w").write("CERTS") -buf2 = io.StringIO() -with contextlib.redirect_stdout(buf2), contextlib.redirect_stderr(buf2): - pm.check_make_ca_generated() -quiet = "no CA certificate bundle" not in buf2.getvalue() -if warned and quiet: - print(" PASS it warns only while the bundle is genuinely missing") -else: - print(" FAIL warned=%s quiet=%s" % (warned, quiet)); sys.exit(1) -PY62 - _count_rc $? -fi + pm.load_config() +if "note:" in buf.getvalue(): + problems.append("a default is reported as disagreeing with the lfs config") +# ...but a real disagreement must still be said +open(conf, "w").write("collector_prefix=other\n") +pm._config = None +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + cfg = pm.load_config() +if "note:" not in buf.getvalue(): + problems.append("a real conflict with the lfs config is not reported") +if cfg.get("collector_prefix") != "nimgnu": + problems.append("the lfs config did not win the conflict") -# ---- prompts must allow line editing ---------------------------------------- # -# Without readline an arrow key inserts an escape sequence instead of moving -# the cursor, so a typo in a long answer -- a group name, a device path -- -# means retyping the whole thing. -for t in lfs blfs packagemanager; do - f="$(dirname "$LFS_TOOL")/$t" - [ -f "$f" ] || continue - grep -q "import readline" "$f" \ - || bad "$t: input() prompts have no line editing" -done -ok "the python tools import readline" - -# an optional import: a python built without readline must still run -python3 - "$LFS_TOOL" <<'PY64' -import sys -src = open(sys.argv[1]).read() -i = src.index("import readline") -window = src[max(0, i - 200):i + 120] -if "try:" not in window or "ImportError" not in window: - print(" FAIL readline is imported unguarded; a python without it would fail") +if problems: + for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS readline is optional, not required") -PY64 +print(" PASS settings come from what the system was built with, not a prompt") +PYFR _count_rc $? - -for f in lfs-helper packagemanager_install; do - p="$(dirname "$LFS_TOOL")/$f" - [ -f "$p" ] || continue - # every prompt that reads an ANSWER needs -e; loops reading files do not - bare="$(grep -nE "read (-r )?(ans|check|answer|reply)\b" "$p" \ - | grep -v -- "-e" | head -3)" - [ -z "$bare" ] \ - || bad "$f: prompt without line editing: $(echo "$bare" | head -1)" -done -ok "the bash prompts use read -e" - -# and -r as well, or a backslash in an answer is eaten -if grep -qE "read -e [^r-]" "$(dirname "$LFS_TOOL")/packagemanager_install" 2>/dev/null; then - bad "a prompt uses -e without -r" -else - ok "prompts keep backslashes literal (-r)" fi -# ---- starting the whole build over -------------------------------------------# -# Deleting a built system is the most destructive thing these tools can do -- -# potentially weeks of compilation -- so it has to say exactly what goes, what -# stays, and refuse outright in the one case that could damage the host. -python3 - "$LFS_TOOL" <<'PY65' -import sys, importlib.machinery as m +# ---- a native gcc means opposite things on either side of the build (1.11.7) # +# Chapter 6 installs only under $LFS_TGT, so DURING the build a native gcc on +# top of the cross one is the toolchain being clobbered. Chapter 8 REPLACES +# the cross toolchain with exactly that native gcc: it is the finished state, +# the thing the build is for. One rule for both told someone snapshotting a +# complete, working system that its toolchain was BROKEN. +python3 - "$LFS_TOOL" <<'PYTC' +import sys, os, tempfile, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -src = open(sys.argv[1]).read() -if not hasattr(lfs, "cmd_bs_restart"): - print(" FAIL no way to start the build over"); sys.exit(1) -body = src[src.index("def cmd_bs_restart"):src.index("def cmd_bs_set_book")] problems = [] -# deleting through a bind-mounted /dev would reach the running system -if "the chroot is still mounted" not in body: - problems.append("it does not refuse while the chroot is mounted") -for d in ('"dev/pts", "dev/shm", "dev", "proc", "sys", "run"',): - if d not in body: - problems.append("the mount check does not cover every virtual filesystem") -# dry run by default, and a confirmation that cannot be typed by accident -if "args.run" not in body: - problems.append("restart is not dry-run by default") -if "Type the mount point" not in body: - problems.append("the confirmation is too easy to give by accident") -# sources are expensive to re-download and unchanged by a build -if "keep_sources" not in body: - problems.append("the downloaded sources are not kept") -# and it should point at the cheaper alternative -if "snapshot restore" not in body: - problems.append("it does not mention restoring a snapshot instead") +lfs.lfs_tgt = lambda: "x86_64-lfs-linux-gnu" + +def tree(steps_built, steps_total=105): + d = tempfile.mkdtemp() + h = os.path.join(d, "usr/include/c++/14.2.0/x86_64-pc-linux-gnu/bits") + os.makedirs(h) + open(os.path.join(h, "c++config.h"), "w").close() + os.makedirs(os.path.join(d, "usr/lib/gcc/x86_64-pc-linux-gnu")) + prog = lfs.pkgusr_state(d, "progress") + os.makedirs(prog, exist_ok=True) + open(os.path.join(prog, "steporder"), "w").write( + "\n".join("s%d" % i for i in range(steps_total))) + open(os.path.join(prog, "steps-built"), "w").write( + "\n".join("s%d" % i for i in range(steps_built))) + return d + +st = lfs._toolchain_state(tree(105)) +if st.startswith("BROKEN"): + problems.append("a finished system is reported as BROKEN: %s" % st) +st = lfs._toolchain_state(tree(30)) +if not st.startswith("BROKEN"): + problems.append("a clobbered cross toolchain mid-build is not reported: %s" % st) +# the WARNING in `verify` has the same two sides +if any(l == "WARNING" for l, _v in lfs._tree_state(tree(105))): + problems.append("a finished tree still gets a native-gcc WARNING") + if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS restart warns, confirms, keeps the sources and spares the host") -PY65 +print(" PASS a native gcc is read as chapter 8 on a finished tree, damage before") +PYTC _count_rc $? -# ---- sources and build scratch are two directories -------------------------- # -# They used to be one, and every problem in this area came from that. A build -# unpacked into the tarball store, dropped .cc-* markers there, and left both -# owned by whoever built -- so a rebuild had to pick debris out of the thing it -# most needed to keep. Split, and the whole class of problem is structural: -# /sources holds only what get-sources downloaded, and BUILD_ROOT is deleted -# whole. -python3 - "$LFS_TOOL" <<'PY65B' -import sys, os, re, importlib.machinery as m +# ---- the kernel and the bootloader are yours (1.11.6) --------------------- # +# Both are decisions about the whole MACHINE, and getting either wrong costs +# the system you are building on. The build must not do them by default, and +# must not let you find that out after the reboot. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '_say_what_is_yours_to_finish' "$_h" \ + || _p="$_p;the build never says the kernel and bootloader are yours" + _sf="$(sed -n '/^_say_what_is_yours_to_finish() {/,/^}/p' "$_h")" + printf '%s' "$_sf" | grep -qi 'kernel' \ + || _p="$_p;the closing note does not mention the kernel" + printf '%s' "$_sf" | grep -qi 'refind' \ + || _p="$_p;the closing note does not say how to add a bootloader entry" + # it is a note, not an action: nothing in it may write anything + printf '%s' "$_sf" | grep -qE '^\s*(cp|mv|rm|dd|mkfs|chmod|chown|install) ' \ + && _p="$_p;the closing note runs commands instead of printing them" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build ends by saying the kernel and bootloader are yours" + fi +fi + +# and no bootloader is set up unless it was asked for, by name +python3 - "$LFS_TOOL" <<'PYBL' +import sys, re, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() problems = [] -body = open(sys.argv[1]).read() - -if 'BUILD_ROOT="${BUILD_ROOT:-$LFS/sources}"' in body: - problems.append("BUILD_ROOT still defaults to the tarball store") -for need in ('SOURCES_DIR="${SOURCES_DIR:-$LFS/sources}"', - 'BUILD_ROOT="${BUILD_ROOT:-$LFS/build}"'): - if need not in body: - problems.append("generated scripts do not set %s" % need.split("=")[0]) - -# generate a real script and read what it does -cmds = ["mkdir -v build\ncd build", "../configure", "make", "make install"] -s = lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-1.0", - "demo-1.0.tar.*", cmds) -# the markers must live in scratch, never in the tarball store -for marker in (".cc-build-demo", ".cc-dir-demo"): - for line in s.splitlines(): - if marker in line and "SOURCES_DIR" in line: - problems.append("%s is written to the tarball store" % marker) -if '$BUILD_ROOT/.cc-build-demo' not in s: - problems.append("the build marker is not written to BUILD_ROOT") -# unpack must extract into scratch -if 'cd "$BUILD_ROOT"' not in s: - problems.append("unpack does not work in BUILD_ROOT") -# and the symlink farm must be built before the tarball is looked for -u = s[s.index("unpack_pkg()"):s.index("#### UNPACK DONE ####")] -if "_link_sources" not in u: - problems.append("unpack does not build the sources symlink farm") - -# the chroot copy must map BOTH directories, not just one -c = lfs._chrootify(s) -if 'SOURCES_DIR="${SOURCES_DIR:-/sources}"' not in c: - problems.append("the chroot script does not point at /sources") -if 'BUILD_ROOT="${BUILD_ROOT:-/build}"' not in c: - problems.append("the chroot script does not point at /build") - +# the default must be "none" with NO config at all -- not "none" because some +# other key happened to be unset +lfs.load_config = lambda: {} +if lfs.bootloader() != "none": + problems.append("the default bootloader is %r, not 'none'" % lfs.bootloader()) +lfs.load_config = lambda: {"bootloader": ""} +if lfs.bootloader() != "none": + problems.append("an empty bootloader setting does not mean 'none'") +# and the prompt must say the machine keeps booting as it does now +src = open(sys.argv[1]).read() +m2 = re.search(r'\("bootloader",.*?\),\n\s*\("esp"', src, re.S) +if m2 and "DEFAULT" not in m2.group(0): + problems.append("the prompt does not say that doing nothing is the default") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS tarball store and build scratch are separate directories") -PY65B +print(" PASS no bootloader is touched unless it was asked for by name") +PYBL _count_rc $? -# ---- a book command may reach a sibling tarball through .. ------------------ # -# The book does this, in Tcl: -# -# cd .. -# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 -# -# `cd ..` leaves the build subdirectory, so ".." is then the package root's -# parent -- which used to be the sources directory and is now BUILD_ROOT. -# Splitting the two would break every command of this shape. -# -# The fix must not be a list of packages known to do it: that would be a second -# thing to keep in sync with the book, and wrong the first time a new release -# adds another. unpack_pkg symlinks EVERY downloaded file into BUILD_ROOT, so -# "../<anything-we-downloaded>" resolves for any package in any future book. -# This test builds a package that does exactly what Tcl does. -_sbx="$T/split" -mkdir -p "$_sbx/lfs/sources" "$_sbx/lfs/out" "$_sbx/mk/demo-2.0" "$_sbx/mk/doc" -cat > "$_sbx/mk/demo-2.0/configure" <<'EOF' -#!/bin/bash -cat > Makefile <<'MK' -all: - @echo compiling -install: - @true -MK -EOF -chmod +x "$_sbx/mk/demo-2.0/configure" -( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0.tar.gz" demo-2.0 ) -echo html > "$_sbx/mk/doc/index.html" -( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0-html.tar.gz" doc ) -echo patch > "$_sbx/lfs/sources/demo-2.0-fix-1.patch" - -python3 - "$LFS_TOOL" "$_sbx/demo.sh" <<'PYSPLIT' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -cmds = [ - "mkdir -v build\ncd build", - "../configure", - "make", - "make install", - # the book's Tcl shape, verbatim in structure - "cd ..\ntar -xf ../demo-2.0-html.tar.gz -C ../../out\n" - "cp -v ../demo-2.0-fix-1.patch ../../out/patch.seen", -] -open(sys.argv[2], "w").write( - lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-2.0", - "demo-2.0.tar.*", cmds)) -PYSPLIT - -# The suite exports SOURCES_DIR/BUILD_ROOT for other tests; clear them here so -# this exercises the DEFAULTS a real build would get. -( cd "$_sbx" && env -u SOURCES_DIR -u BUILD_ROOT LFS="$_sbx/lfs" \ - bash demo.sh all ) >"$_sbx/log" 2>&1 -_sp="" -[ -d "$_sbx/lfs/out/doc" ] || _sp="$_sp;a sibling tarball reached through .. was not found" -[ -f "$_sbx/lfs/out/patch.seen" ] || _sp="$_sp;a patch reached through .. was not found" -# and the whole point: a full build must not have touched the tarball store -_left="$(ls -A "$_sbx/lfs/sources" | grep -v '^demo-2\.0' | tr '\n' ' ' || true)" -[ -z "$_left" ] || _sp="$_sp;a build wrote into the tarball store: $_left" -[ -d "$_sbx/lfs/build/demo-2.0" ] || _sp="$_sp;the package was not unpacked into BUILD_ROOT" -[ -f "$_sbx/lfs/build/.cc-dir-demo" ] || _sp="$_sp;build markers did not land in BUILD_ROOT" -[ -L "$_sbx/lfs/build/demo-2.0-html.tar.gz" ] || _sp="$_sp;the farm entry is not a symlink" -if [ -n "$_sp" ]; then - printf '%s\n' "${_sp#;}" | tr ';' '\n' | while IFS= read -r _m; do - [ -n "$_m" ] && bad "$_m" - done - sed 's/^/ /' "$_sbx/log" | tail -8 -else - ok "'../<tarball>' resolves for any package, and /sources stays clean" -fi - -# ---- /sources has ONE owner convention, and it is the book's ---------------- # -# Three rules had drifted apart, and the result on disk was a mixture: -# -# drwxrwxrwt 1 9998 install /mnt/lfs/sources/ -# -rw-r--r-- 1 lfs 9998 acl-2.3.2.tar.xz -# -# * lfs's _chown_tree_to_lfs added "sources" to the 4.3 handover set, so it -# chowned /sources to the HOST's lfs user. Book 4.3 does no such thing. -# * lfs-helper carried /sources in install_dirs_list AS WELL AS in -# never_adopt_list -- the exact conflation the comment above -# never_adopt_list warns about, the one that made /root root:install. -# verify --fix then made it root:install 775 and the seal made it 1775, -# dropping the o+w that lets the unprivileged build user unpack at all. -# * cmd_clean_sources reset unknown owners to root:root. -# -# The host lfs account is created by book 4.3's own useradd, which pins no -# uid -- it was 10753 on a Debian host. Package users start at 10000, so that -# is not merely "an unnamed UID" as the book puts it: it COLLIDES with a real -# package user, and tarballs report as owned by some unrelated package. -# -# Book 3.1 settles it and the reasoning is given there in full: -# chmod -v a+wt $LFS/sources -> root:root 1777 -# chown root:root $LFS/sources/* -> the contents too -# World-writable is what grants the build user access, so no group is involved. -python3 - "$LFS_TOOL" <<'PY65C' -import sys, os, stat, tempfile, importlib.machinery as m +# ---- $LFS may never be the running system (1.11.5) ------------------------ # +# Every path this tool writes is $LFS/something. With $LFS set to the host -- +# a config holding "/", an `export LFS=/` in the wrong shell, a typo in the +# interview -- `restart --run` deletes the top-level directories of the machine +# you are sitting on, and _verify_lfs_ownership runs `chown -R lfs /usr`. +# The chroot bind mounts were guarded; the tree itself was not. +python3 - "$LFS_TOOL" <<'PYHOST' +import sys, os, tempfile, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() problems = [] -if not hasattr(lfs, "_normalize_sources"): - print(" FAIL no single rule for /sources ownership") +for fn in ("_is_host_path", "_refuse_host_tree"): + if not hasattr(lfs, fn): + problems.append("there is no %s" % fn) +if problems: + for p in problems: print(" FAIL %s" % p) sys.exit(1) -# 4.3 hands over the book's directories and NOT /sources -if "sources" in lfs.LFS_HANDOVER_DIRS: - problems.append("/sources is in the 4.3 handover set") -body = open(sys.argv[1]).read() -h = body[body.index("def _chown_tree_to_lfs"):body.index("def _ensure_lfs_home")] -if 'LFS_HANDOVER_DIRS + ("sources",)' in h: - problems.append("the 4.3 chown still adds /sources to the handover") +# the host, by every spelling +for p in ("/", "/usr", "/etc", "/boot", "/var", "/home", "/bin", "/lib"): + if not lfs._is_host_path(p): + problems.append("%s is not recognised as the running system" % p) +# by another name: a symlink to / d = tempfile.mkdtemp() -src = os.path.join(d, "sources"); os.makedirs(src) -tar = os.path.join(src, "bash-5.3.tar.gz"); open(tar, "w").close() -tree = os.path.join(src, "acl-2.3.2"); os.makedirs(tree) -os.chmod(src, 0o775) # what verify --fix used to leave - +link = os.path.join(d, "root-link") try: - os.chown(tar, 10753, 10753) # a host uid inside the package range - os.chown(tree, 10753, 10753) + os.symlink("/", link) + if not lfs._is_host_path(link): + problems.append("a symlink to / is not recognised") except OSError: - print(" SKIP cannot chown here (needs root)") -else: - lfs._normalize_sources(d, run=True, quiet=True) - st = os.stat(src) - if stat.S_IMODE(st.st_mode) != (stat.S_ISVTX | 0o777): - problems.append("/sources is not 1777 -- the build user cannot unpack " - "(mode %o)" % stat.S_IMODE(st.st_mode)) - if st.st_uid != 0 or st.st_gid != 0: - problems.append("/sources itself is not root:root") - if os.lstat(tar).st_uid != 0: - problems.append("a downloaded file keeps a host uid (book 3.1)") - # an unpacked tree belongs to whoever is building it; taking it away - # mid-build would stop that build. Book 3.1 runs before any unpack. - if os.lstat(tree).st_uid != 10753: - problems.append("an unpacked tree was chowned out from under its build") - # running it twice must change nothing - if lfs._normalize_sources(d, run=True, quiet=True) != (0, False): - problems.append("the rule is not idempotent") - -# and it must actually be applied where sources appear or survive -g = body[body.index("def cmd_bs_get_sources"):body.index("def _title_pkgver")] -if "_normalize_sources" not in g: - problems.append("get-sources does not apply book 3.1's chown") -r = body[body.index("def cmd_bs_restart"):body.index("def cmd_bs_set_book")] -if "_normalize_sources" not in r: - problems.append("restart does not apply book 3.1's chown") + pass -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS /sources is root:root 1777 with root-owned contents (book 3.1)") -PY65C -_count_rc $? +# a path INSIDE a system tree is the system too, however deep +for p in ("/usr/src/lfs", "/etc/lfs", "/var/tmp/lfs"): + if not lfs._is_host_path(p): + problems.append("%s is not recognised as part of the system" % p) -# ---- /sources is scratch, not a shared install directory -------------------- # -# Being in install_dirs_list meant verify --fix set it root:install 775 and -# seal-install-dirs set 1775 -- both drop the o+w of book 3.1's `a+wt`, and -# o+w is the only thing that lets the unprivileged build user unpack there. -# It belongs in never_adopt_list (no package may own it) and nowhere else. -if [ -r "$(dirname "$LFS_TOOL")/lfs-helper" ]; then - HELPER="$(dirname "$LFS_TOOL")/lfs-helper" -else - HELPER="" -fi -if [ -n "$HELPER" ]; then - _p="" - # the DEFAULT list, i.e. the heredoc, must not carry it - if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then - _p="$_p;/sources is still a shared install directory" - fi - # but it must still be a directory no package may adopt - if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then - _p="$_p;/sources may now be adopted by a package" - fi - # an installdirs.lst from an older run must not bring it back - if ! sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -q 'grep -vx.*/sources'; then - _p="$_p;a stale installdirs.lst can still reintroduce /sources" - fi - # the build scratch is the same kind of thing and needs the same rules - if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then - _p="$_p;/build may be adopted by a package" - fi - if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then - _p="$_p;/build is a shared install directory" - fi - # every tree scan must skip it -- otherwise each snapshot, ownership pass - # and collector-group map walks every unpacked source tree - if [ "$(grep -c 'not -path "\${\?r}\?/build/\*"' "$HELPER")" -lt 3 ]; then - _p="$_p;a tree scan still descends into the build scratch" - fi - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "/sources and /build are scratch, not shared install directories" - fi -else - bad "lfs-helper not found beside $LFS_TOOL" -fi +# ...and a real build tree must still be allowed, even though it has the same +# layout. A check that refuses /mnt/lfs/usr refuses every build there is. +tree = os.path.join(d, "lfs") +os.makedirs(os.path.join(tree, "usr"), exist_ok=True) +os.makedirs(os.path.join(tree, "etc"), exist_ok=True) +for p in (tree, "/mnt/lfs", "/home/someone/lfs", "/media/disk/lfs"): + if lfs._is_host_path(p): + problems.append("a legitimate build tree is refused: %s" % p) -# ---- snapshots must not read live kernel filesystems ------------------------ # -# Excluding "./sys/*" skips the CONTENTS but still archives the ./sys directory -# itself. Reading a live kernel filesystem's metadata gives -# tar: ./sys: file changed as we read it -# and exit 1, so the snapshot failed -- then succeeded on a retry, because the -# timing happened to work out. --one-file-system does not descend into -# anything mounted inside the tree at all. -python3 - "$LFS_TOOL" <<'PY66' -import sys +# ONE rule, not two: the interview and the hard stop must agree, or a path +# refused by one is accepted by the other. src = open(sys.argv[1]).read() -body = src[src.index('cmd = ("tar -C'):] -body = body[:body.index("prog = ")] if "prog = " in body else body[:2000] -problems = [] -if "--one-file-system" not in body: - problems.append("tar still descends into mounted filesystems") -# tar's exit 1 is "files differ", a warning on a live tree -- verify, don't die -if "rc == 1" not in body: - problems.append("a tar warning is still treated as a failure") -if "tar -tf" not in body: - problems.append("the archive is not verified before being accepted") -# but an unreadable archive must NOT be kept -if "os.remove(dest)" not in body: - problems.append("a corrupt archive would be left in place") +import re +body = re.search(r"def _refuse_host_tree\(lfs\):.*?\n\n\ndef ", src, re.S) +if body and "_is_host_path" not in body.group(0): + problems.append("the hard stop keeps its own copy of the rule") +if "_is_host_path" not in re.search(r"for k, desc, default in _SESSION_KEYS:.*?\n\n", src, re.S).group(0): + problems.append("the interview accepts a mount point the tools will refuse") + +# the destructive paths must go through it +for fn in ("require_lfs_for_root", "_verify_lfs_ownership"): + b = re.search(r"def %s\(.*?(?=\n\ndef )" % fn, src, re.S) + if b and "_refuse_host_tree" not in b.group(0): + problems.append("%s does not check the tree is not the host" % fn) + if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS snapshots skip live filesystems and verify what they wrote") -PY66 +print(" PASS $LFS is refused when it is the running system, by any spelling") +PYHOST _count_rc $? -# ---- an application user must be usable, not just exist --------------------- # -# `user create` made the account and stopped there: you could not become it, -# its home was world-readable, and nothing launched the program. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def _setup_application_user" "$pmt" \ - && ok "creating an application user sets up access to it" \ - || bad "user create leaves an account nobody can use" - su_body="$(sed -n '/^def _setup_application_user/,/^def _home_of/p' "$pmt")" - case "$su_body" in - *'"chmod", "0750", home'*) ok "the account's home is private" ;; - *) bad "the application's home stays world-readable" ;; - esac - case "$su_body" in - *'"usermod", "-a", "-G", u, main'*) - ok "your own login joins the account's group" ;; - *) bad "a shared directory would not be reachable" ;; - esac - # without a main user there is nobody to grant access TO - case "$su_body" in - *"No main user is set"*) ok "it says so when no main user is set" ;; - *) bad "it silently does nothing when main_user is unset" ;; - esac - # the PAM file is not part of stock LFS: absence must not be an error - pam="$(sed -n '/^def _allow_su_via_pam/,/^def _write_launcher/p' "$pmt")" - case "$pam" in - *"not present -- \`su - "*) ok "a missing PAM file is handled, not fatal" ;; - *) bad "it assumes the hint's PAM file exists" ;; - esac - grep -q "_APPLICATION_USER_HELP" "$pmt" \ - && ok "application users have their own help section" \ - || bad "no help explaining what an application user is" +# ---- an answer that is collected must be acted on, or said (1.11.4) ------- # +# `strip` used to be asked for and then ignored; since 1.11.27 it is done. +# The notice survives for the one case left -- the run failed partway -- and +# these assertions are about the notice's mechanics, which have not changed: +# it reads the answer, stays quiet when nobody asked, and is said at the END. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '_warn_if_strip_was_asked_for' "$_h" \ + || _p="$_p;nothing tells you the strip answer was ignored" + # it must READ the answer, not assume it + _sf="$(sed -n '/^_warn_if_strip_was_asked_for() {/,/^}/p' "$_h")" + printf '%s' "$_sf" | grep -q 'LFS_STRIP' \ + || _p="$_p;the strip notice does not read LFS_STRIP" + # and only when it was actually asked for + printf '%s' "$_sf" | grep -q 'return 0' \ + || _p="$_p;the strip notice fires even when nobody asked for it" + # said at the END, with the other things you still have to do + _i1="$(grep -n '_warn_if_strip_was_asked_for$' "$_h" | tail -1 | cut -d: -f1)" + _i2="$(grep -n '^ _warn_if_no_login$' "$_h" | tail -1 | cut -d: -f1)" + [ -n "$_i1" ] && [ -n "$_i2" ] \ + || _p="$_p;the notice is not part of the build's last word" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a build that ignored your strip answer says so at the end" + fi fi -# ---- shell completion -------------------------------------------------------- # -comp="$(dirname "$LFS_TOOL")/lfs-completion.bash" -if [ -f "$comp" ]; then - bash -n "$comp" \ - && ok "the completion script parses" \ - || bad "the completion script has a syntax error" - # subcommands must come from the tools themselves, or the list goes stale - grep -q "_lfs_tool_subcommands" "$comp" \ - && ok "completions are read from each tool's own help" \ - || bad "the completion keeps a second list of commands" - for t in lfs blfs packagemanager lfs-helper; do - grep -q "complete -F _lfs_complete $t\b" "$comp" \ - || bad "no completion registered for $t" - done - ok "all four tools have completion registered" +# ---- strip is implemented, and it runs as each file's owner (1.11.27) ----- # +# The interview collected the answer for years and the build ended with a NOT +# YET IMPLEMENTED notice. Now `lfs-helper strip` does the work. The property +# that matters: strip REPLACES each file, and the replacement belongs to +# whoever ran it -- so the files must be grouped by their owner, read from the +# filesystem, and each group stripped AS that owner. Anything else silently +# hands the tree to root and the ownership record is gone. +if [ -f "$_h" ]; then + _p="" + grep -qE '^cmd_strip\(\) \{' "$_h" || _p="$_p;there is no cmd_strip" + grep -qE '^ strip\) +shift; cmd_strip' "$_h" \ + || _p="$_p;strip is not dispatched" + _usg="$(_slice_fn "$_h" _usage_text)" + printf '%s' "$_usg" | grep -q 'lfs-helper strip' \ + || _p="$_p;usage does not mention strip" + _sf="$(_slice_fn "$_h" cmd_strip)" + [ -n "$_sf" ] || _p="$_p;cmd_strip cannot be sliced" + # each group is stripped as its owner -- the whole point of the command + printf '%s' "$_sf" | grep -q 'su -s /bin/bash' \ + || _p="$_p;stripping is not run as the owning account" + # and the owner comes from the file itself, not from an assumption + printf '%s' "$_sf" | grep -q '%U' \ + || _p="$_p;the owner is not read from the filesystem" + # the book's .la cleanup is there, and refusable + printf '%s' "$_sf" | grep -q 'keep_la' \ + || _p="$_p;the libtool .la cleanup cannot be kept back" + # dry run is the default here like everywhere else + printf '%s' "$_sf" | grep -q '"$run" = 0' \ + || _p="$_p;strip has no dry run" + # a file that fails to strip is reported, not swallowed + printf '%s' "$_sf" | grep -q 'FAILSTRIP' \ + || _p="$_p;a strip failure would be silent" + # the interview answer is acted on at the end of build-all, through the ONE + # implementation -- not a second copy of the loop + _rf="$(_slice_fn "$_h" _strip_if_asked_for)" + [ -n "$_rf" ] || _p="$_p;nothing runs strip when it was asked for" + printf '%s' "$_rf" | grep -q 'LFS_STRIP' \ + || _p="$_p;the strip run does not read the interview answer" + printf '%s' "$_rf" | grep -q 'cmd_strip --run' \ + || _p="$_p;the strip run does not go through cmd_strip" + grep -q '^ _strip_if_asked_for$' "$_h" \ + || _p="$_p;build-all does not run strip at the end" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "strip is implemented, per owner, through one door" + fi fi -# ---- a system you can actually log into -------------------------------------- # -# A freshly built system has root with NO password. Depending on the login -# manager that is either "anyone can log in as root" or "nobody can log in at -# all" -- discovered after a reboot, with no way in. +# ---- the install record has ONE writer, with two strategies (1.11.39) ----- # +# Item 6, step 1. What a package installed was written twice, by two drivers, +# in two places -- lfs-helper's manifest (scan by TIMESTAMP) and +# packagemanager_install's pkg.lst (scan by OWNER) -- and each tool read only +# its own. That is why `packagemanager update` reported "0 of 97 judged" on a +# system where every package was recorded, in the other file. # -# It lives in lfs-helper now, as the built-in init-accounts step. It could not -# move to `packagemanager setup` with the rest of the old final step: setup runs -# on the BOOTED system, and you need to log in to run it. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _ia="$(_slice_fn "$helper_src" cmd_init_accounts)" +# Both strategies stay: early in the build a package's files are not owned by +# its package user yet, so an owner scan returns nothing, and the timestamp +# scan is also the only one that sees a config step REWRITE a file. What must +# not stay is two drivers each deciding for themselves. +if [ -f "$_h" ]; then _p="" - case "$_ia" in - *'passwd root'*) ;; - *) _p="$_p;the root password is never set" ;; - esac - case "$_ia" in - *'NO password yet'*) ;; - *) _p="$_p;nothing explains why a root password matters" ;; - esac - # a scripted run has no terminal: it must say what is missing, not hang - case "$_ia" in - *'! -t 0'*) ;; - *) _p="$_p;a non-interactive run would block on a prompt" ;; - esac - case "$_ia" in - *'useradd -m'*) ;; - *) _p="$_p;no login account is ever created" ;; - esac + grep -q '^cmd_record_install()' "$_h" \ + || _p="$_p;there is no single recorder" + grep -q '^_record_strategy()' "$_h" \ + || _p="$_p;the strategy is not chosen in one named place" + _rs="$(_slice_fn "$_h" _record_strategy)" + printf '%s' "$_rs" | grep -q 'ownership_established' \ + || _p="$_p;the strategy does not turn on whether ownership exists" + _wp="$(_slice_fn "$_h" _write_pkg_lst)" + [ -n "$_wp" ] || _p="$_p;_write_pkg_lst cannot be sliced" + # an empty scan must never overwrite a real record -- empty-as-pass has cost + # this project three separate bugs + printf '%s' "$_wp" | grep -q 'n:-0.*-eq 0\|-eq 0' \ + || _p="$_p;an empty owner scan would overwrite the record" + # and it must not sweep the build scratch in + printf '%s' "$_wp" | grep -q 'scan_prune_set' \ + || _p="$_p;the owner scan does not prune the build scratch" + # a package built before ownership gets its pkg.lst without a rebuild + grep -q '^cmd_reconcile_records()' "$_h" \ + || _p="$_p;a build-stage package never gets a pkg.lst" + grep -q 'record-install)' "$_h" \ + || _p="$_p;record-install is not reachable as a command" + # BOTH drivers go through it + grep -q 'cmd_record_install "$name"' "$_h" \ + || _p="$_p;lfs-helper's own build does not use the recorder" + _slice_fn "$helper_src" cmd_pm_install | grep -q 'cmd_record_install' \ + || _p="$_p;pm-install's old-style path never records the install" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done else - ok "the build ends with a system you can log into" + ok "one recorder, two strategies, and both drivers call it" fi -fi + # ---- one runner, three privilege cases (item 6, step 3b -- 1.11.46) ----- # + # Both drivers ran the install script and spelled it differently: cmd_build + # as `su - "$owner" -c "env ... bash SCRIPT PHASE" | tee`, + # packagemanager_install as `su - -c "set -o pipefail; PATH=...; bash ... | + # tee" <user>`. The differences were exactly where the bugs lived -- which + # PATH the build sees (no wrappers => "install: cannot change permissions of + # '/usr/sbin'") and whether the status comes from the script or from tee (a + # failing build reported success, and cost a day). + _p="" + grep -q '^run_phase_as()' "$_h" || _p="$_p;there is no shared runner" + _rp="$(_slice_fn "$_h" run_phase_as)" + [ -n "$_rp" ] || _p="$_p;run_phase_as cannot be sliced" + printf '%s' "$_rp" | grep -q 'PIPESTATUS' \ + || _p="$_p;the runner would report tee's status, not the build's" + printf '%s' "$_rp" | grep -q 'have_su' \ + || _p="$_p;the runner cannot tell the no-su case apart" + # cmd_build must USE it rather than keep its own copy + _cb2="$(_slice_fn "$_h" cmd_build)" + printf '%s' "$_cb2" | grep -q 'run_phase_as ' \ + || _p="$_p;the build still runs the script its own way" + printf '%s' "$_cb2" | grep -qE 'su - "\$owner" -c' \ + && _p="$_p;a second su invocation survives in the build" + # and it is reachable for a caller that is not this script + grep -q '^cmd_run_script()' "$_h" \ + || _p="$_p;there is no run-script command" + grep -q 'run-script)' "$_h" \ + || _p="$_p;run-script is not dispatched" + _pmci="$(_slice_fn "$helper_src" cmd_pm_install)" + { printf '%s' "$_pmci" | grep -q 'cmd_build ' \ + && printf '%s' "$_pmci" | grep -q 'run_phase_as '; } \ + || _p="$_p;pm-install runs scripts some way other than the one loop/runner" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "one runner, and both drivers go through it" + fi -# password detection must read /etc/shadow correctly -shd="$T/shadowcheck"; mkdir -p "$shd" -_haspw() { - grep -qE "^root:[^:]*:" "$1" && ! grep -qE "^root:(\*|!|)?:" "$1" -} -printf 'root::19000:0:99999:7:::\n' > "$shd/empty" -printf 'root:!:19000:0:99999:7:::\n' > "$shd/locked" -printf 'root:$6$a$b:19000:0:99999:7:::\n' > "$shd/set" -if ! _haspw "$shd/empty" && ! _haspw "$shd/locked" && _haspw "$shd/set"; then - ok "an empty, locked or real root password are told apart" -else - bad "root password detection is wrong" -fi + # functionally: the status is the SCRIPT's, and the phase reaches it + _rt="$(mktemp -d)" + printf '#!/bin/bash\necho "phase=$1"\nexit 7\n' > "$_rt/f.sh" + printf '#!/bin/bash\nexit 0\n' > "$_rt/o.sh" + chmod +x "$_rt"/*.sh + ( + # shellcheck disable=SC1090 + source <(sed -n '/^run_phase_as() {/,/^}/p' "$_h") + have_su() { return 1; } + user_exists() { return 1; } + run_phase_as root "$_rt/f.sh" install "$_rt/f.log" "" "" 1 >/dev/null 2>&1 + _frc=$? + run_phase_as root "$_rt/o.sh" install "$_rt/o.log" "" "" 1 >/dev/null 2>&1 + _orc=$? + [ "$_frc" = 7 ] || echo "BADRC failing build reported $_frc, not 7" + [ "$_orc" = 0 ] || echo "BADRC passing build reported $_orc, not 0" + grep -q 'phase=install' "$_rt/f.log" || echo "BADRC the phase never reached the script" + ) > "$_rt/out" 2>&1 + if grep -q BADRC "$_rt/out"; then + sed 's/^BADRC / /' "$_rt/out" | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the runner reports the build's own exit status" + fi + rm -rf "$_rt" + + # ---- phase tracking lives in the runner (item 6, step 4) --------------- # + # It was cmd_build's private bookkeeping, so a packagemanager install that + # died during `make install` left no record and "resume" meant starting + # over. The runner is the one place that knows which phase ran and how it + # ended -- whichever tool asked. + _pt="$(mktemp -d)"; mkdir -p "$_pt/state" + printf '#!/bin/bash\ninstall_pkg() { :; }\ncase "${1:-all}" in\n build) echo ok ;;\n install) echo "phase '"'"'install'"'"' FAILED"; exit 1 ;;\n all) echo ok; echo "phase '"'"'install'"'"' FAILED"; exit 1 ;;\nesac\n' > "$_pt/install_p_dummy" + printf '#!/bin/bash\necho oldstyle\n' > "$_pt/install_old" + _pt_run() { env LFS_PKGUSR_DIR="$_pt/state" LFS_ETC="$_pt" LFS_PKGUSR_PREFIX=p \ + bash "$_h" run-script --script "$_pt/install_p_dummy" "$@" ; } + printf 'root:x:0:0::/root:/bin/bash\n' > "$_pt/passwd" + _p="" + _pt_run --user p_dummy --phase all --log "$_pt/a.log" >/dev/null 2>&1 + [ "$(tr '\n' ' ' < "$_pt/state/progress/phases/dummy" 2>/dev/null)" = "unpack build " ] \ + || _p="$_p;a failed 'all' run did not record the phases that finished" + rm -f "$_pt/state/progress/phases/dummy" + _pt_run --user p_dummy --phase build --log "$_pt/b.log" >/dev/null 2>&1 + grep -qx build "$_pt/state/progress/phases/dummy" 2>/dev/null \ + || _p="$_p;a finished phase is not recorded under the STEP name" + _pt_run --user p_dummy --phase install --log "$_pt/c.log" >/dev/null 2>&1 + grep -qx install "$_pt/state/progress/phases/dummy" 2>/dev/null \ + && _p="$_p;a FAILED phase was recorded as done" + rm -f "$_pt/state/progress/phases/dummy" + _pt_run --user p_dummy --phase build --name '' --log "$_pt/d.log" >/dev/null 2>&1 + [ -e "$_pt/state/progress/phases/dummy" ] \ + && _p="$_p;--name '' does not disable tracking" + env LFS_PKGUSR_DIR="$_pt/state" LFS_ETC="$_pt" LFS_PKGUSR_PREFIX=p \ + bash "$_h" run-script --user p_old --script "$_pt/install_old" \ + --phase build --log "$_pt/e.log" >/dev/null 2>&1 + [ -e "$_pt/state/progress/phases/old" ] \ + && _p="$_p;an old-style script (bash script unpack runs EVERYTHING) got a phase record" + # and cmd_build must not keep a second copy of the bookkeeping + _cb3="$(_slice_fn "$_h" cmd_build)" + printf '%s' "$_cb3" | grep -q 'clear_phases' \ + && _p="$_p;cmd_build still clears phases itself" + printf '%s' "$_cb3" | grep -qE "phase '\[a-z\]\+' FAILED" \ + && _p="$_p;cmd_build still infers finished phases from the log itself" + printf '%s' "$_cb3" | grep -q 'record_phase ' \ + && _p="$_p;cmd_build still records phases itself" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "phase tracking is the runner's, for every caller, and only for phased scripts" + fi + rm -rf "$_pt" -# ---- decisions already made must not be asked again ------------------------- # -# The export copied into the tree said -# dir|/usr/libexec/gcc|nimgnu_gcc -# yet the build still stopped to ask which group should share that directory. -# Two faults: nothing applied the file, and `import-groups --run` read the -# wrong path anyway. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *"cmd_import_groups --run"*) - ok "a copied-in groups export is applied automatically" ;; - *) bad "the export sits unapplied until run by hand" ;; - esac - grep -q "^_groups_already_imported()" "$helper_src" \ - && ok "it is applied once, not on every build" \ - || bad "the import would run on every build-all" - - # `--run` used to be taken as the FILENAME, then overwrite the path - ig="$(sed -n '/^cmd_import_groups() {/,/^}/p' "$helper_src")" - case "$ig" in - *'in="$STATE/collector-groups.export"; }'*) - bad "import-groups --run still overwrites the file it found" ;; - *) ok "import-groups --run reads the file it chose" ;; - esac - case "$ig" in - *'local in="" run=0 a'*) ok "the arguments are parsed in one place" ;; - *) bad "the run flag can still be wiped by a later declaration" ;; - esac - - # and the prompt itself must consult the export, imported or not - cg="$(sed -n '/^choose_collector_group() {/,/^}/p' "$helper_src")" - case "$cg" in - *'collector-groups.import'*) - ok "the prompt checks the copied-in decisions first" ;; - *) bad "a decision in the tree would still be asked about" ;; - esac + # ---- a heading that promises errors must show some (1.11.47) ----------- # + # m4 failed with "no M4-1.4.21.tar.* ... in /sources" -- a clear message + # that matched none of the compiler-shaped patterns, so the report printed + # "--- first errors in the log ---" followed by nothing at all. No pattern + # matching is a statement about the PATTERNS, not about the log. + _et="$(mktemp -d)" + printf 'configure: ok\nno M4-1.4.21.tar.* in /sources -- run: get-sources\n' \ + > "$_et/nomatch.log" + printf 'foo.c:1: error: bad\n' > "$_et/match.log" + ( + # shellcheck disable=SC1090 + source <(sed -n '/^show_build_errors() {/,/^}/p' "$_h") + say() { echo "$@"; }; fail() { echo "$@"; } + show_build_errors "$_et/nomatch.log" > "$_et/out1" 2>&1 + show_build_errors "$_et/match.log" > "$_et/out2" 2>&1 + ) >/dev/null 2>&1 + _p="" + grep -q 'in /sources' "$_et/out1" 2>/dev/null \ + || _p="$_p;a failure with no recognised pattern shows nothing at all" + grep -q 'error: bad' "$_et/out2" 2>/dev/null \ + || _p="$_p;a recognised error is no longer reported" + grep -q 'first errors in the log' "$_et/out1" 2>/dev/null \ + && _p="$_p;it still promises errors it did not find" + # and the new command explains itself, like every other one + ( "$_h" run-script --help >"$_et/h" 2>&1 ) || true + grep -q 'run one phase\|Runs one phase' "$_et/h" 2>/dev/null \ + || _p="$_p;run-script does not answer --help" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a failure report shows the log when the patterns miss" + fi + rm -rf "$_et" + + # ---- a run's record must not outlive the run (1.11.49) ----------------- # + # Building m4 ended with "Collector groups used during this run:" listing + # nine grants for p_urllib3, make-ca and p11-kit -- real grants, made weeks + # earlier. The file is only cleared by the report, so any build that + # granted without reporting (a failure, another path) left its entries for + # the next successful build to claim. + _p="" + _cb3="$(_slice_fn "$_h" cmd_build)" + printf '%s' "$_cb3" | grep -q ': > "$GRANTED"' \ + || _p="$_p;a build inherits grants made by earlier runs" + _rg="$(_slice_fn "$_h" report_granted_groups)" + printf '%s' "$_rg" | grep -q 'used during this run' \ + && _p="$_p;the report still claims older grants as this run's" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the grant report names only what this build needed" + fi fi -# end to end: the answer is in the file, so nothing is asked -igd="$T/importdec"; mkdir -p "$igd/.lfs-pkgusr" "$igd/etc" "$igd/usr/src" -printf 'root:x:0:\ninstall:x:9999:\n' > "$igd/etc/group" -printf 'root:x:0:0:root:/root:/bin/bash\n' > "$igd/etc/passwd" -printf '# prefix|nimgnu\ndir|/usr/libexec/gcc|nimgnu_gcc\n' \ - > "$igd/.lfs-pkgusr/collector-groups.import" -got="$(STATE="$igd/.lfs-pkgusr" awk -F'|' -v d=/usr/libexec/gcc \ - '$1 == "dir" && $2 == d { print $3; exit }' \ - "$igd/.lfs-pkgusr/collector-groups.import")" -[ "$got" = "nimgnu_gcc" ] \ - && ok "a recorded decision resolves without asking" \ - || bad "the recorded decision was not found (got '$got')" +_pm_tool="$(dirname "$LFS_TOOL")/packagemanager" +# ---- silence is not a clean bill of health (1.11.40) ---------------------- # +# `packagemanager update` printed "all current with the LFS book" while `lfs +# update` reported thirty stale packages. _lfs_update_report returned [] BOTH +# for "everything is current" and for "the call produced nothing", and the +# caller printed the same reassuring line for both -- empty-as-pass, in the +# very helper written to stop packages disappearing quietly. +# +# Also here: a script generated by `lfs` from the LFS book was reported as +# OLD-format, because only blfs stamps 'script format vN'. The advice that +# came with it was worse than the warning -- migrate-scripts would rewrite an +# LFS-book script into a BLFS one. Two generators, two dialects: item 6's +# disease, and until they are one thing this tool must not misread the +# other's work. +if [ -f "$_pm_tool" ]; then + python3 - "$_pm_tool" "$LFS_TOOL" <<'PYSILENCE' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +problems = [] +# three outcomes, told apart +seg = open(sys.argv[1]).read() +lr = seg[seg.index("def _lfs_update_report("):] +lr = lr[:lr.index("\ndef ", 10)] +for token in ('"failed"', '"current"', '"lines"'): + if token not in lr: + problems.append("the LFS consult cannot report %s" % token) +if "if not out:" not in lr: + problems.append("empty output is still indistinguishable from 'current'") +# and the caller says so instead of smoothing it over +rn = seg[seg.index("def _report_no_book("):] +rn = rn[:rn.index("\ndef ", 10)] +if "UNCHECKED" not in rn: + problems.append("a failed consult is not reported as unchecked") +# functional: with no lfs present the answer is 'failed', never 'current' +_which = pm.shutil.which +pm.shutil.which = lambda x: None +try: + _r = pm._lfs_update_report([]) + kind = _r[0] if isinstance(_r, tuple) else "current" if _r == [] else "lines" + if kind != "failed": + problems.append("a missing lfs reads as a clean bill of health") +finally: + pm.shutil.which = _which +# a script from the other book is not an old script +lfs_hdr = "### lfs-crosschain-script v17: shadow\n" +if not hasattr(pm, "_is_lfs_book_script"): + problems.append("nothing tells an LFS-book script from an old one") + pm._is_lfs_book_script = lambda t: False +# ---- one stamp, read by everyone (item 6, step 3 -- 1.11.44) ---- +# Both generators now write `script format vN origin=<tool>`, so no tool has +# to recognise the other's dialect. The counters are PER GENERATOR (blfs v4, +# lfs v17) and must never be compared with each other -- a foreign script is +# left to the tool that writes it. +if not hasattr(pm, "_script_origin"): + problems.append("there is no shared way to ask what a script is") +else: + for txt, want, label in ( + ("### generated by blfs 1.0 -- script format v4 origin=blfs", + "blfs", "a blfs script"), + ("### script format v17 origin=lfs", "lfs", "an lfs script"), + ("### generated by blfs 1.0 -- script format v4", "blfs", + "a legacy blfs script"), + ("### lfs-crosschain-script v17: shadow", "lfs", + "a legacy lfs script"), + ("#!/bin/bash\nmake install", "", "an unstamped script")): + if pm._script_origin(txt) != want: + problems.append("%s is not identified" % label) + _pso2 = seg[seg.index("def _package_script_outdated("):] + _pso2 = _pso2[:_pso2.index("\ndef ", 10)] + if 'origin != "blfs"' not in _pso2: + problems.append("scripts from another generator are still judged here") +try: + _pso = seg[seg.index("def _package_script_outdated("):] + _pso = _pso[:_pso.index("\ndef ", 10)] +except ValueError: + _pso = "" +# the property: a script this generator does not own is not judged by it. +# (This first asserted that _package_script_outdated CALLS +# _is_lfs_book_script -- the mechanism -- and broke the moment the check +# became the better one, an origin comparison. Assert intent.) +if "origin" not in _pso and "_is_lfs_book_script" not in _pso: + problems.append("LFS-book scripts are still flagged as OLD-format") +if pm._is_lfs_book_script("### generated by blfs 1.0 -- script format v3"): + problems.append("a blfs script is misread as an LFS one") +elif not pm._is_lfs_book_script(lfs_hdr) and hasattr(pm, "_is_lfs_book_script"): + pass +# the book is parsed once per path, not once per package name +lsrc = open(sys.argv[2]).read() +ls = lsrc[lsrc.index("def load_soup("):] +ls = ls[:ls.index("\ndef ", 10)] +if "_SOUP_CACHE" not in ls: + problems.append("the book is re-parsed for every package looked up") +# the generator really emits it -- generate a body and read it back +try: + lfsmod = m.SourceFileLoader('lfsmod', sys.argv[2]).load_module() + body = lfsmod._crosschain_script_body("demo", "ch-demo", "Demo-1.0", + "demo-*.tar.*", + ["./configure", "make", "make install"]) + if pm._script_origin(body) != "lfs": + problems.append("the lfs generator does not write the shared stamp") +except Exception as e: + problems.append("could not generate a script to check the stamp: %s" % e) +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a failed check says so, and a foreign script is not an old one") +PYSILENCE + _count_rc $? +fi -# ---- 7.2 must actually hand the tree to root -------------------------------- # -# The handover used `chown --from lfs`, which changes only files owned by -# exactly that user. A tree owned by a uid with no matching account -- or by -# an `lfs` whose uid changed since -- was silently left alone, so `prepare` -# reported "give the tree to root", changed nothing, and `enter` then refused. -# The loop in `run` caught it as a step that ran but stayed pending. -python3 - "$LFS_TOOL" <<'PY68' -import sys +# ---- ask a question whose answer can be no (1.11.41) --------------------- # +# `lfs update` ended with "In the BLFS book -- update those with blfs: ... +# gcc-pass1 cfg_fstab init-dirs libstdcpp ..." -- LFS build STEPS, offered as +# BLFS packages. `_blfs_knows` asked `blfs debug <name>`, which is a +# DIAGNOSTIC: it prints "resolve: NOT FOUND" and exits 0, so the check said +# yes to every string it was ever given. +# +# The same output offered to rebuild util-linux-tmp, python-tmp and +# gettext-tmp -- the TEMPORARY chapter-7 toolchain -- onto a finished system. +python3 - "$LFS_TOOL" <<'PYSTAGE' +import sys, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] src = open(sys.argv[1]).read() -i = src.index('"give the tree to root"') -cmd = src[i:i + 400] -if "--from lfs" in cmd: - print(" FAIL the handover still only matches files owned by 'lfs'") +bk = src[src.index("def _blfs_knows("):] +bk = bk[:bk.index("\ndef ", 10)] +if re.search(r'\["debug"|\+ \["debug', bk): + problems.append("BLFS membership is still decided by a diagnostic command") +if "_blfs_anchor_set" not in bk: + problems.append("BLFS membership is not decided from the book's own anchors") +# the stage variants are excluded, and ONLY the variants +cu = src[src.index("def cmd_update("):] +cu = cu[:cu.index("\ndef ", 10)] +if "_stage_steps" not in cu: + problems.append("the temporary chapter-7 builds are offered for reinstall") +else: + st = {n.lower() for n, _s in lfs.CHROOT_STEPS_7 if n.lower().endswith("-tmp")} + st |= {n.lower() for n, _s, _p in lfs.CROSSCHAIN_STEPS + if re.search(r"-(pass\d+|tmp)$", n.lower())} + for must in ("util-linux-tmp", "python-tmp", "gcc-pass1", "binutils-pass2"): + if must not in st: + problems.append("%s would be rebuilt onto a finished system" % must) + # chapter 6 builds these under their own names -- excluding them would + # hide the packages most worth knowing about + for keep in ("glibc", "coreutils", "linux-headers", "libstdcpp"): + if keep in st: + problems.append("%s is hidden from update as if it were a stage" % keep) +# ---- the book ships the URLs; use them (1.11.48) ---- +# `lfs install --reinstall m4` failed with "no M4-1.4.21.tar.* in /sources" +# after the book moved to 13.0 -- while the URL for exactly that tarball sat +# in the wget-list this tool already caches per book. Only get-sources ever +# read it, and only in bulk. +import tempfile as _tf, os as _os +if not hasattr(lfs, "_wget_list_url_for"): + problems.append("a single package's source URL cannot be looked up") +else: + _d = _tf.mkdtemp(); _p = _os.path.join(_d, "wl.txt") + open(_p, "w").write( + "https://ftp.gnu.org/gnu/m4/m4-1.4.21.tar.xz\n" + "https://github.com/util-linux/util-linux/releases/util-linux-2.41.3.tar.xz\n") + _orig = lfs.wget_list_path + lfs.wget_list_path = lambda v: _p + try: + if not (lfs._wget_list_url_for("x", "M4-1.4.21") or "").endswith("m4-1.4.21.tar.xz"): + problems.append("the wget-list lookup misses a plain package name") + # hyphenated names are where a naive split goes wrong + if not (lfs._wget_list_url_for("x", "Util-linux-2.41.3") or "").endswith( + "util-linux-2.41.3.tar.xz"): + problems.append("the wget-list lookup misses a hyphenated name") + if lfs._wget_list_url_for("x", "Nope-9.9"): + problems.append("the lookup invents a URL for a package not listed") + finally: + lfs.wget_list_path = _orig +if not hasattr(lfs, "_have_source_for"): + problems.append("nothing checks whether the tarball is already there") +else: + _t2 = _tf.mkdtemp(); _os.makedirs(_os.path.join(_t2, "sources")) + open(_os.path.join(_t2, "sources", "m4-1.4.20.tar.xz"), "w").write("x") + if not lfs._have_source_for(_t2, "M4-1.4.20"): + problems.append("an existing tarball is not recognised") + if lfs._have_source_for(_t2, "M4-1.4.21"): + problems.append("a different version reads as already downloaded") +# ---- rebuild in BOOK order, not alphabetically (1.11.52) ---- +# The stale list came out alphabetically: Python, binutils, coreutils ... +# glibc. Rebuilt in that order the earlier ones link against the OLD glibc +# and are stale again the moment it is replaced. Chapter 8's own sequence +# is the safe one. +_cu3 = src[src.index("def cmd_update("):] +_cu3 = _cu3[:_cu3.index("\ndef ", 10)] +if "iter_sections" not in _cu3 or "_order" not in _cu3: + problems.append("the stale list is not ordered by the book") +else: + _o = {"glibc": 5, "binutils": 30, "python": 55} + _names = ["Python", "binutils", "glibc", "cfg_fstab"] + _sorted = sorted(_names, key=lambda n: (_o.get(n.lower(), 10 ** 6), n.lower())) + if _sorted.index("glibc") > _sorted.index("Python"): + problems.append("Python would be rebuilt before glibc") + if _sorted[-1] != "cfg_fstab": + problems.append("a package the book does not name jumps the queue") + +_ci = src[src.index("def cmd_install("):] +_ci = _ci[:_ci.index("\ndef ", 10)] +if "_fetch_one_source" not in _ci: + problems.append("installing still fails instead of fetching what it needs") + +if problems: + for p in problems: print(" FAIL %s" % p) sys.exit(1) -if "chown -h -R root:root" not in cmd: - print(" FAIL the handover does not chown recursively"); sys.exit(1) -# and the refusal should say who DOES own it -if "_owner_name_of" not in src: - print(" FAIL the refusal does not name the actual owner"); sys.exit(1) -print(" PASS 7.2 hands the tree to root whoever owned it") -PY68 +# ---- merged from the SysV-book session (1.11.54) ---- +# A parallel session branched at 1.11.26 and fixed three things here. Ported +# rather than overwritten, because that branch never saw 1.11.27-1.11.53. +if "sqlite" not in src.lower() and "INFIX" not in src: + problems.append("the infix tarball pattern (SQLite-3510200) is missing") +_pg = src[src.index("def _pkg_glob_for("):] +_pg = _pg[:_pg.index("\ndef ", 10)] +if "m.group(1).lower()" not in _pg: + problems.append("a package whose tarball has an infix cannot be found") +_g = lfs._pkg_glob_for("SQLite-3510200", "sqlite") +if "sqlite-*3510200.tar.*" not in _g: + problems.append("SQLite's autoconf tarball would not be matched") +# ...and the doc filter must keep that pattern off sqlite-doc-* +if "*-doc-*|*-docs-*|*-html-*|*-man-pages-*|*-tests-*" not in src: + problems.append("the infix pattern could grab a -doc- tarball") +# the sources step is done only when the CURRENT book's list is complete +if "src_missing" not in src: + problems.append("leftover tarballs still count as 'sources downloaded'") + +print(" PASS BLFS membership is a fact, and stage builds are not upgrades") +PYSTAGE _count_rc $? -# behaviour: a tree owned by an unknown uid must still be handed over -hod="$T/handover"; mkdir -p "$hod/usr/bin" "$hod/etc" -chown -R 4242 "$hod/usr" "$hod/etc" 2>/dev/null || true -if [ "$(stat -c %u "$hod/usr")" = "4242" ]; then - ( for d in usr etc; do [ -e "$hod/$d" ] && chown -h -R root:root "$hod/$d"; done; true ) - [ "$(stat -c %u "$hod/usr")" = "0" ] \ - && ok "a tree owned by an unknown uid is handed to root" \ - || bad "the handover left the tree owned by $(stat -c %u "$hod/usr")" -else - ok "(cannot test unknown-uid handover as this user)" -fi - -# ---- a shared user reaches the session -------------------------------------- # -# A "shared user" is one that belongs to the group on the main user's XDG -# runtime directory: -# drwxrws--- n76310 u_xdg_runtime /tmp/xdg-n76310 -# The display socket, the session bus and audio live there, so without that -# group a graphical program starts and then cannot draw anything. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def _join_xdg_runtime_group" "$pmt" \ - && ok "a shared user joins the session group" \ - || bad "a shared user cannot reach the display or audio" - jx="$(sed -n '/^def _join_xdg_runtime_group/,/^def _setup_application_user/p' "$pmt")" - # the group must be READ from the directory, never assumed by name - case "$jx" in - *"getgrgid(os.stat(rt).st_gid)"*) - ok "the group is read from the runtime directory" ;; - *) bad "the session group name is hardcoded" ;; - esac - case "$jx" in - *'u_xdg_runtime"'*) bad "a specific group name is hardcoded" ;; - *) ok "no group name is baked in" ;; - esac - # a personal group is not a shared group - case "$jx" in - *"which is not a shared "*) - ok "it refuses to add the user to a personal group" ;; - *) bad "an application user could join the person's own group" ;; - esac - # both /run/user/<uid> and /tmp/xdg-<user> layouts - xd="$(sed -n '/^def _xdg_runtime_dir_of/,/^def _join_xdg/p' "$pmt")" - for p in "/run/user" "/tmp/xdg-"; do - case "$xd" in - *"$p"*) ;; - *) bad "the runtime directory search misses $p" ;; - esac - done - ok "both runtime-directory layouts are searched" - # --shared and --share-dir are different things - grep -q '"--share-dir"' "$pmt" \ - && ok "the session group and a shared directory are separate options" \ - || bad "--shared conflates the session group with a directory" -fi +# ---- entering refreshes every tool, and keeps the tree's own book --------- # +# Entering the chroot refreshed lfs-helper ALONE, so the python tools inside +# stayed older than the host's and every packagemanager run opened with "These +# tools differ from the copies on the host" -- advice to sync, from the path +# that had just synced one file of five. (1.11.42) +# +# And the other half: install-tools copied the host's config.json over the +# tree's, so `lfs set-default 13.0-systemd` INSIDE the system was undone by +# the next entry, silently, back to whatever the host had. +python3 - "$LFS_TOOL" <<'PYREFRESH' +import sys, os, json, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +# every tool, not just the bash one +t = tempfile.mkdtemp() +os.makedirs(os.path.join(t, "usr/bin")) +for tool in ("lfs", "lfs-helper", "packagemanager", "packagemanager_install", + "blfs"): + open(os.path.join(t, "usr/bin", tool), "w").write("stale\n") +try: + got = sorted(lfs._refresh_chroot_tools(t, quiet=True) or []) +except Exception as e: + got = [] + problems.append("the refresh raised: %s" % e) +want = sorted(["lfs", "lfs-helper", "packagemanager", "blfs"]) +# the RETIRED tool must be swept off the chroot PATH, not refreshed -- +# a stale copy running old code is what this sync exists to prevent +if os.path.exists(os.path.join(t, "usr/bin", "packagemanager_install")): + problems.append("the retired packagemanager_install survives in the chroot") +missing = [x for x in want if x not in got] +if missing: + problems.append("entering leaves these stale in the chroot: %s" + % ", ".join(missing)) +# and it does nothing when there is nothing to do +if got and lfs._refresh_chroot_tools(t, quiet=True): + problems.append("the refresh copies again when nothing changed") +# the tree's own book choice survives an install-tools run +src = open(sys.argv[1]).read() +it = src[src.index("def cmd_bs_install_tools("):] +it = it[:it.index("\ndef ", 10)] +if "shutil.copy(cfgsrc, os.path.join(store" in it: + problems.append("the host's config still overwrites the tree's") +# (1.13.65 added target-config.json between the two, so the comment now +# reads "the tree overrides both" -- what matters is that the tree wins) +if "the tree overrides" not in it: + problems.append("nothing keeps the tree's own book default") +# the version you would move to is emphasised, and not in a pipe +cu = src[src.index("def cmd_update("):] +cu = cu[:cu.index("\ndef ", 10)] +if "_C_OK" not in cu: + problems.append("the LFS report does not colour the new version") +if "_tty_out()" not in cu: + problems.append("the colour would land in a piped report") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +# a kept script says WHICH version it builds when the book has moved on: +# `lfs script fetch m4` printed the book's title (M4-1.4.21) above a script +# that builds M4-1.4.20, with nothing to say they differ. (1.11.45) +import io, contextlib +es = src[src.index("def _ensure_lfs_script("):] +es = es[:es.index("\ndef ", 10)] +if "kept your existing script" not in es: + problems.append("a kept script does not say it was kept") +t2 = tempfile.mkdtemp() +os.makedirs(os.path.join(t2, "usr/src/lfs-pkgusr/scripts")) +_sp = os.path.join(t2, "usr/src/lfs-pkgusr/scripts/m4.sh") +open(_sp, "w").write('#!/bin/bash\nname_version="M4-1.4.20"\n') +_buf = io.StringIO() +try: + with contextlib.redirect_stdout(_buf): + lfs._ensure_lfs_script(type("A", (), {})(), t2, "m4", "ch-m4", + "8.14. M4-1.4.21", {"commands": ["make"]}, False) + _out = _buf.getvalue() + if "M4-1.4.20" not in _out or "M4-1.4.21" not in _out: + problems.append("a script older than the book does not say so") + if "--regenerate" not in _out: + problems.append("nothing says how to rebuild it from the book") +except Exception as e: + problems.append("keeping a script raised: %s" % e) -# ---- the setup must be applicable after the fact ---------------------------- # -# An account made earlier -- or by hand -- should be able to reach the same -# state without deleting and recreating it. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def cmd_user_setup" "$pmt" \ - && ok "an existing user can be set up after the fact" \ - || bad "the setup only happens at creation time" - grep -q '"setup"' "$pmt" \ - && ok "`user setup` is registered" \ - || bad "the setup command is not reachable" - # it must refuse on a user that does not exist, rather than half-creating - cs="$(sed -n '/^def cmd_user_setup/,/^def _write_desktop_entry/p' "$pmt")" - case "$cs" in - *"no such user"*) ok "setup refuses on a user that does not exist" ;; - *) bad "setup would act on a missing user" ;; - esac - # the same code path as create, or the two drift - case "$cs" in - *_setup_application_user*) ok "create and setup share one code path" ;; - *) bad "setup reimplements what create does" ;; - esac +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS entering refreshes every tool and keeps the tree's book") +PYREFRESH +_count_rc $? - # menu entry - grep -q "^def _write_desktop_entry" "$pmt" \ - && ok "a menu entry is written" \ - || bad "no desktop entry is created" - de="$(sed -n '/^def _write_desktop_entry/,/^def _write_launcher/p' "$pmt")" - for field in "Desktop Entry" "Exec=" "StartupWMClass=" "Icon="; do - case "$de" in - *"$field"*) ;; - *) bad "the desktop entry has no $field" ;; - esac - done - ok "the desktop entry has the fields a menu needs" - # a launcher for a program that is not installed is a broken command - sa="$(sed -n '/# 6. a launcher and a menu entry/,/_write_desktop_entry(main, u, app)/p' "$pmt")" - case "$sa" in - *"no launcher or menu entry"*) - ok "nothing is written when the program is not installed" ;; - *) bad "a launcher could point at a program that does not exist" ;; - esac - # and neither file is clobbered if it is already there - case "$de" in - *"already exists -- left alone"*) ok "an existing menu entry is kept" ;; - *) bad "the desktop entry would be overwritten" ;; - esac -fi +# ---- a refresh that does not update the stamps announces its own work ------ # +# The refresh copied all five tools in and left tool-stamps.json alone -- the +# file `warn_if_tools_are_stale` compares against -- so packagemanager opened +# with "These tools differ from the copies on the host" about tools it had +# just been handed, and advised syncing, on the path that had synced them. +# And a copy that silently did not stick showed up only as the same refresh +# message on every entry, so the copy is verified now. (1.11.43) +python3 - "$LFS_TOOL" <<'PYSTAMP' +import sys, os, json, hashlib, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +src = open(sys.argv[1]).read() +rc = src[src.index("def _refresh_chroot_tools("):] +rc = rc[:rc.index("\ndef ", 10)] +if "_write_tool_stamps" not in rc: + problems.append("the refresh leaves the staleness stamps behind") +if "did not change" not in rc: + problems.append("a copy that does not stick is never noticed") +t = tempfile.mkdtemp() +os.makedirs(os.path.join(t, "usr/bin")) +for tool in ("lfs", "lfs-helper", "packagemanager", "blfs"): + open(os.path.join(t, "usr/bin", tool), "w").write("stale\n") +try: + lfs._refresh_chroot_tools(t, quiet=True) +except Exception as e: + problems.append("the refresh raised: %s" % e) +sp = os.path.join(t, "usr/share/lfs/tool-stamps.json") +if not os.path.isfile(sp): + problems.append("no stamps were written, so the warning cannot clear") +else: + for name, want in json.load(open(sp)).items(): + p = os.path.join(t, "usr/bin", name) + have = hashlib.md5(open(p, "rb").read()).hexdigest() \ + if os.path.exists(p) else None + if have != want: + problems.append("the stamp for %s does not match what was " + "installed -- the warning would keep firing" % name) + break +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the refresh records what it installed, and checks it landed") +PYSTAMP +_count_rc $? -# ---- changing LFS_TGT invalidates the toolchain ----------------------------- # -# gcc-pass2 installs its C++ headers under $LFS_TGT. Change LFS_TGT afterwards -# and the compiler looks under the new name while the headers sit under the -# old one -- present, readable, and invisible. It surfaces forty packages -# later as "bits/c++config.h: No such file or directory", which reads like a -# missing file rather than a mismatch. -python3 - "$LFS_TOOL" <<'PY69' +# the prompt must stop claiming the answer is ignored, now that it is not +python3 - "$LFS_TOOL" <<'PYSTRIP' import sys, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -src = open(sys.argv[1]).read() problems = [] -if not hasattr(lfs, "_crosschain_tgt_mismatch"): - problems.append("nothing detects a changed target triplet") -# the triplet must be RECORDED with the build, or there is nothing to compare -mk = src[src.index("def _mark_crosschain_done"):] -mk = mk[:mk.index("def lfs_tgt")] -if '"lfs_tgt"' not in mk: - problems.append("the target is not recorded with the build progress") -# and checked before the chroot, not after -if "this tree was built for a different target" not in src: - problems.append("a changed target is not reported") -i_check = src.find("this tree was built for a different target") -i_prep = src.find('if action == "prepare":') -if i_check < i_prep: - problems.append("the check is not inside chroot prepare") +prompts = {k: p for k, p, _ in lfs._SESSION_KEYS} +if "strip" not in prompts: + problems.append("'strip' is no longer asked during session") +elif "NOT YET IMPLEMENTED" in prompts["strip"]: + problems.append("the strip prompt still claims it is not implemented") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS a changed target triplet is caught before the chroot") -PY69 +print(" PASS the strip prompt no longer disclaims the work") +PYSTRIP _count_rc $? -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_warn_on_triplet_mismatch()" "$helper_src" \ - && ok "the mismatch is also reported inside the chroot" \ - || bad "inside the chroot the mismatch is only a missing-file error" - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *_warn_on_triplet_mismatch*) - ok "it is checked before building, not after failing" ;; - *) bad "the mismatch check never runs" ;; - esac +# ---- a suppressed failure is a bug waiting to be found (1.11.4) ----------- # +# `cmd 2>/dev/null || true` has cost this project more debugging than anything +# else: a tree came out with every package home root:root 755 and not one line +# said why. The pattern is not banned -- mkdir -p on a directory that exists is +# fine -- but a failure that CHANGES WHAT THE SYSTEM LOOKS LIKE must be said. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + grep -q '^soft() {' "$_h" || _p="$_p;there is no way to report a non-fatal failure" + # the ownership and permission calls are the ones that matter + _leak="$(grep -nE '(real_chmod|real_chown|real_chgrp|chown|chgrp) .*2>/dev/null \|\| true' "$_h" \ + | grep -v 'SUPPRESSED DELIBERATELY' | wc -l)" + # ANY comment above is not enough -- the surrounding prose explains what the + # call does, not why its failure may be thrown away, and a reintroduced + # suppression inherits whatever comment happened to be there. The line + # immediately above must say which of the two cases this is, in those words. + _undoc=0 + while IFS= read -r _ln; do + [ -n "$_ln" ] || continue + _n="${_ln%%:*}" + sed -n "$((_n-1))p" "$_h" \ + | grep -qE 'SUPPRESSED DELIBERATELY|benign:' || _undoc=$((_undoc+1)) + done <<< "$(grep -nE '(real_chmod|real_chown|real_chgrp) .*2>/dev/null \|\| true' "$_h")" + [ "${_undoc:-0}" = 0 ] \ + || _p="$_p;$_undoc suppressed ownership call(s) are not marked deliberate or benign" + # and the two that are deliberate must say so in those words + grep -q 'SUPPRESSED DELIBERATELY' "$_h" \ + || _p="$_p;the deliberate suppressions are not marked as such" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every suppressed ownership failure says why it is suppressed" + fi fi -# ---- three causes, one error message ---------------------------------------- # -# "bits/c++config.h: No such file or directory" is produced by three different -# situations needing three different fixes. The compiler's own configure line -# tells them apart: -# no --host and no --target -> a NATIVE gcc, which only chapter 8 builds: -# its install is half-finished -# both are cross names -> LFS_TGT changed since chapter 6 -# no headers at all -> chapter 6 never finished -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_explain_triplet_mismatch()" "$helper_src" \ - && ok "the cause of a mismatch is named, not just the symptom" \ - || bad "all three causes give the same message" - ex="$(sed -n '/^_explain_triplet_mismatch() {/,/^}/p' "$helper_src")" - case "$ex" in - *"it is a NATIVE build"*) - ok "a native compiler is recognised as chapter 8's" ;; - *) bad "a half-installed chapter-8 gcc is not identified" ;; - esac - case "$ex" in - *"LFS_TGT changed between"*) - ok "two cross names are recognised as a changed LFS_TGT" ;; - *) bad "a changed LFS_TGT is not identified" ;; - esac - # the two cases must give DIFFERENT fixes -- that is the whole point - # the two causes need different fixes -- that is the whole point - case "$ex" in - *"lfs config lfs_tgt"*"--phase all --force"*|*"--phase all --force"*"lfs config lfs_tgt"*) - ok "each cause gets the fix that applies to it" ;; - *) bad "the same fix is offered for different causes" ;; - esac - # and it must say nothing when the names agree - case "$ex" in - *'[ "$have" = "$mine" ] && return 0'*) - ok "it is silent when there is no mismatch" ;; - *) bad "it would explain a mismatch that does not exist" ;; - esac +# lfs-helper must actually offer the doors the other tool now knocks on. +_h="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_h" ]; then + _p="" + for _c in wrapper-dir make-wrappers grant-dir pkgusr-home owner-name; do + grep -qE "^ $_c\)" "$_h" \ + || _p="$_p;lfs-helper has no '$_c' command" + done + # `case` takes the first match, so a repeated entry is unreachable code + # pretending to be a second decision. Six of them were. + _dups="$(sed -n '/^case "${1:-}" in/,/^esac/p' "$_h" \ + | grep -oE '^ [a-z][a-z0-9|-]*\)' | sort | uniq -d)" + [ -z "$_dups" ] || _p="$_p;the dispatch repeats: $(echo $_dups | tr -d ')')" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every shared decision is reachable as one lfs-helper command" + fi fi -# ---- installs are staged, not written straight into the system -------------- # -# `make install` writes into the live system one file at a time, in Makefile -# order. Interrupt it and the system is neither the old version nor the new -# one -- and for gcc that is unrecoverable: it installs its driver early and -# its C++ headers late, so a cancel in between leaves a compiler that cannot -# compile C++, which is what you need to rebuild it. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_merge_stage()" "$helper_src" \ - && ok "installs go to a staging tree first" \ - || bad "installs still write straight into the live system" - grep -q "DESTDIR='\$_stage'" "$helper_src" \ - && ok "DESTDIR points the install at the staging tree" \ - || bad "the staging tree is created but never used" +# ---- editing the wrong install script fails silently ------------------------ # +# A package user's home holds two files with nearly identical names: +# install_<name>-<version> the source -- the one to edit +# install_<name> a runtime copy, overwritten every run +# Editing the second changes nothing, the install repeats unchanged, and it +# looks as though the tool threw the edit away. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _warn_if_wrong_file_edited" "$pmt" \ + && ok "an edit to the runtime copy is pointed out" \ + || bad "editing the wrong file goes unmentioned" + grep -q "To change what it does, edit:" "$pmt" \ + && ok "a failure names the file to edit" \ + || bad "a failure gives no path to edit" - # a FAILED install must leave the system untouched -- the whole point - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - case "$cb" in - *'if [ "${rc:-0}" = 0 ]; then'*'_merge_stage'*) - ok "the staged tree is moved in only on success" ;; - *) bad "a failed install could still be merged" ;; - esac - case "$cb" in - *"the system was not touched"*) - ok "a failed install says the system is unchanged" ;; - *) bad "a failed install does not say what state things are in" ;; - esac + # behaviour: warn only when the runtime copy is genuinely newer + python3 - "$pmt" <<'PY61' +import sys, os, time, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +base = tempfile.mkdtemp() +pm.BASE_DIR = base +# the directory on disk is the USER's, so build the fixture through the same +# helper the code uses -- the package "p" becomes the account p_p, and +# hard-coding "p" here made this test fail the moment prefixes were turned on +# resolve the home the same way the code does -- the account is p_p AND it +# lives under the package root, so neither part can be hard-coded here +_h = pm.pkgusr_home("p") +os.makedirs(_h) +src = os.path.join(_h, "install_p-1.0") +rt = os.path.join(_h, "install_p") +open(src, "w").write("real") +time.sleep(0.02) +open(rt, "w").write("edited") +import io, contextlib +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + pm._warn_if_wrong_file_edited("p", src) +warned = "runtime copy" in buf.getvalue() +os.utime(src, None) # now the source is newer +buf2 = io.StringIO() +with contextlib.redirect_stdout(buf2), contextlib.redirect_stderr(buf2): + pm._warn_if_wrong_file_edited("p", src) +quiet = "runtime copy" not in buf2.getvalue() +if warned and quiet: + print(" PASS it warns only when the runtime copy is the newer file") +else: + print(" FAIL warned=%s quiet=%s" % (warned, quiet)) + sys.exit(1) +PY61 + _count_rc $? +fi - # a package that ignores DESTDIR writes to the system anyway: say so - ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" - case "$ms" in - *"return 2"*) ok "an empty staging tree is detected" ;; - *) bad "a package ignoring DESTDIR would look like a clean install" ;; - esac - case "$cb" in - *"ignores DESTDIR"*) ok "and reported, not silently trusted" ;; - *) bad "ignoring DESTDIR is not reported" ;; +# ---- installing make-ca does not create any certificates -------------------- # +# It installs the TOOL. Until it is run there is still no CA bundle, wget +# still cannot verify anything, and the /etc/wgetrc workaround stays -- but the +# install reported success, so the missing step is easy to overlook. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _hint_generate_ca_bundle" "$pmt" \ + && ok "the missing make-ca -g step is pointed out" \ + || bad "make-ca installs and nothing says to run it" + hg="$(sed -n '/^def _hint_generate_ca_bundle/,/^def /p' "$pmt")" + case "$hg" in + *"make-ca -g"*) ok "it gives the exact command" ;; + *) bad "the hint does not say what to run" ;; esac - # merging must target the tracked root, not a hardcoded / - # the installer, not the merge wrapper, is what writes - it2="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" - case "$it2" in - *'root="${SNAP_ROOT:-/}"'*) ok "the merge targets the tracked root" ;; - *) bad "the merge writes to a hardcoded /" ;; + # silent once the certificates exist + case "$hg" in + *"certificates exist already"*) ok "it stops once certificates exist" ;; + *) bad "the hint would repeat forever" ;; esac - # the workarounds this replaces must be gone - grep -q "cmd_fix_toolchain" "$helper_src" \ - && bad "the symlink workaround is still present" \ - || ok "the triplet symlink workaround is gone" - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *"Finishing its install before going on"*) - bad "the auto-finish workaround is still present" ;; - *) ok "the auto-finish workaround is gone" ;; + # ---- the book's init flavour must match the system ----------------------- # + # A systemd book emits `systemctl` in its install scripts; on a SysV system + # every affected package fails mid-install and has to be hand-edited. + grep -q "^def _warn_init_mismatch" "$pmt" \ + && ok "a systemd/SysV book mismatch is reported" \ + || bad "the wrong book flavour goes unmentioned" + wm="$(sed -n '/^def _warn_init_mismatch/,/^def /p' "$pmt")" + case "$wm" in + *"if book_systemd == have_systemd"*) + ok "it says nothing when the book matches" ;; + *) bad "the mismatch warning would always fire" ;; esac -fi -# ---- a shared user must know where the session is --------------------------- # -# Being in the session group is necessary but not sufficient: XDG_RUNTIME_DIR, -# WAYLAND_DISPLAY and the package-user PATH all come from -# /etc/pkgusr/skel-u_xdg/.bash_profile. Without it the account can reach the -# socket and still not find it. -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def _link_xdg_profile" "$pmt" \ - && ok "a shared user gets the XDG profile" \ - || bad "a shared user has no XDG_RUNTIME_DIR" - lp="$(sed -n '/^def _link_xdg_profile/,/^def _setup_application_user/p' "$pmt")" - # a symlink, so editing the file once reaches every shared account - case "$lp" in - *"os.symlink(XDG_SKEL, target)"*) ok "it is linked, not copied" ;; - *) bad "each account gets its own copy, which then drift" ;; - esac - # the runtime directory belongs to whoever is logged in: fill it in, do - # not ship someone else's path - case "$lp" in - *"@XDG_RUNTIME_DIR@"*) ok "the runtime directory is filled in, not shipped" ;; - *) bad "a hardcoded runtime directory would be installed" ;; + # ---- the mismatch is asked about ONCE, and the answer is kept ------------ # + # The SysV book is no longer maintained, so running the systemd book on a + # SysV system is a reasonable, deliberate choice -- and a warning printed + # on EVERY invocation punishes it forever. (1.11.30) + case "$wm" in + *init_mismatch_ok*) ok "an accepted mismatch is remembered" ;; + *) bad "the mismatch warning nags on every run" ;; esac - case "$lp" in - *"kept the old one as"*) ok "an existing profile is kept, not destroyed" ;; - *) bad "the user's own .bash_profile would be overwritten" ;; - esac - # and it must be wired into the shared path - sa="$(sed -n '/# 4. the session/,/# 5. a shared directory/p' "$pmt")" - case "$sa" in - *_link_xdg_profile*) ok "--shared links the profile as well as the group" ;; - *) bad "the profile is never linked" ;; + case "$wm" in + *ask_yes_no*) ok "it is asked, not just announced" ;; + *) bad "there is no way to accept the mismatch" ;; esac fi -# the file must ship, and the Makefile must install it without clobbering -skel="$(dirname "$LFS_TOOL")/skel-u_xdg/.bash_profile" -[ -f "$skel" ] \ - && ok "the XDG profile ships with the tools" \ - || bad "the XDG profile is not shipped" -mkf="$(dirname "$LFS_TOOL")/Makefile" -if [ -f "$mkf" ]; then - grep -q "skel-u_xdg" "$mkf" \ - && ok "make install installs it" \ - || bad "make install does not install the XDG profile" - grep -q "kept your existing .bash_profile" "$mkf" \ - && ok "reinstalling does not overwrite your edits" \ - || bad "a reinstall would clobber the XDG profile" +# ---- generated scripts retry downloads and survive a missing systemctl ------ # +# GNU's ftpmirror 502s on the first tries more often than not, and a plain +# wget gives up on any HTTP error -- the third manual re-run succeeding via a +# mirror redirect is exactly what a retry loop does unattended. And a +# systemd book's `systemctl` on a SysV system aborted the phase mid-install; +# a fallback function turns that into a visible skip. (1.11.30) +_bt="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt" ]; then + _p="" + grep -q '_pp_fetch() {' "$PHASES_LIB" || _p="$_p;generated scripts have no retrying fetch" + # RETRYING A DEAD MIRROR IS NOT A RETRY: ftpmirror.gnu.org returned 502 + # and the script asked the same broken host three times. + # A PACKAGE MUST NOT QUIETLY BUILD A DEPENDENCY IT COULD NOT FIND. + # cairo's meson fell back to bundled subprojects and installed its own + # freetype/fontconfig -- owned by p_cairo, with no development symlinks, + # and freetype2 was then never built at all because a libfreetype was on + # disk. Every later "cannot find -lfreetype" came from that. + python3 - "$_bt" <<'PYWRAP' || _p="$_p;meson may silently vendor a missing dependency" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._no_vendored_subprojects("meson setup .. --prefix=/usr &&\nninja") +assert "--wrap-mode=nofallback" in out, out +assert out.splitlines()[1] == "ninja" +# not added twice, and a commented line is left alone +again = b._no_vendored_subprojects(out) +assert again.count("--wrap-mode=nofallback") == 1, again +assert b._no_vendored_subprojects("# meson setup ..").count("wrap-mode") == 0 +PYWRAP + # THE WHOLE SYSTEMD FAMILY, not just systemctl. LLVM built all 4343 + # targets and then died on `systemd-run: command not found` -- the + # systemd book uses several of these, and each missing one aborts the + # phase exactly as systemctl would have. + # (the names are emitted from a list in the generator, so check the list) + _shims="$(sed -n '/^_pp_systemd_shims() {/,/^}/p' "$PHASES_LIB")" + for _sd in systemd-run systemd-tmpfiles loginctl journalctl udevadm; do + printf '%s' "$_shims" | grep -qw "$_sd" \ + || _p="$_p;$_sd is not shimmed, and it aborts the phase on a SysV system" + done + # and only when the real binary is absent, so a systemd host is untouched + printf '%s' "$_shims" | grep -q 'command -v "\$c" >/dev/null 2>&1 && continue' \ + || _p="$_p;the systemd shims are defined unconditionally" + + # A META PAGE HAS NO TARBALL. "Xorg Libraries" is a LIST of packages: + # no pkg=, a directory for link=, and its own commands fetch the list. + # Fetching link= downloaded index.html and `tar xf ../` said + # "tar: ../: Cannot read: Is a directory". + grep -q 'no single tarball' "$PHASES_LIB" \ + || _p="$_p;a page with no tarball still tries to fetch and unpack one" + # THE BOOK'S "START A SUBSHELL AND PASTE THIS" IDIOM. A person types + # `bash -e`, pastes the loop, then `exit`. A script that runs `bash -e` + # gets a shell reading the terminal and waits there forever -- which is + # exactly how a build sat idle for twenty minutes with no compiler under + # it. + python3 - "$_bt" <<'PYSUB' || _p="$_p;a bare `bash -e` from the book still hangs the build" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._unwrap_interactive_subshell("bash -e\nfor p in a b; do echo $p; done\nexit") +assert not any(l.strip() == "bash -e" for l in out.splitlines()), out +assert not any(l.strip() == "exit" for l in out.splitlines()), out +assert "for p in a b" in out +# a package with no such wrapper is untouched +assert b._unwrap_interactive_subshell("make\nmake install") == "make\nmake install" +PYSUB + + # AN ABSOLUTE PATH WALKS PAST EVERY WRAPPER. The wrappers live on the + # package user's PATH, so the book's `as_root /sbin/ldconfig` ran the + # real one and failed on the sticky /etc while the wrapper sat unused. + python3 - "$_bt" <<'PYWRAPTOOL' || _p="$_p;an absolute path bypasses the package-user wrappers" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +assert b._use_wrapped_tools("as_root /sbin/ldconfig") == "as_root ldconfig" +assert b._use_wrapped_tools("/usr/bin/install -m644 x /usr/lib").startswith("install ") +# a comment is left alone, and a longer name is not a match +assert b._use_wrapped_tools("# /sbin/ldconfig here").startswith("# /sbin/") +assert "ldconfigXYZ" in b._use_wrapped_tools("echo /sbin/ldconfigXYZ") +PYWRAPTOOL + + # as_root IS DEFINED IN THE BOOK'S PROSE, which we keep as a comment -- + # so the multi-package loop called a function that did not exist: + # install_xorg7-lib: line 198: as_root: command not found + # In this model there is nothing to elevate: the package user owns what + # it installs into. + python3 - "$_bt" <<'PYASROOT' || _p="$_p;the book's as_root is used but never defined" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._define_as_root("for p in a; do as_root make install; done") +assert 'as_root() { "$@"; }' in out, out +assert out.index('as_root() {') < out.index('as_root make'), out +# a package that never calls it gains nothing +assert b._define_as_root("make install") == "make install" +# ...and a mention in a COMMENT is not a call +assert b._define_as_root("# as_root make install\nmake install").count("as_root() {") == 0 +PYASROOT + + # A FINISHED HOST IS NOT THE CHROOT. inside_chroot() asked "does + # /usr/src/lfs-pkgusr exist" -- true of ANY system these tools built -- so + # the user's own machine refused every build command with "you are inside + # the LFS chroot". An inference true of the destination cannot identify + # the journey; the chroot says so itself now (LFS_IN_CHROOT). + grep -q 'LFS_IN_CHROOT=1' "$LFS_TOOL" \ + || _p="$_p;the chroot does not mark itself, so it has to be guessed" + python3 - "$LFS_TOOL" <<'PYCHR' || _p="$_p;inside_chroot misreads a host or a chroot" +import sys, os, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +def check(env): + old = dict(os.environ) + os.environ.clear(); os.environ.update(env) + try: + return lfs.inside_chroot() + finally: + os.environ.clear(); os.environ.update(old) +assert check({"LFS_IN_CHROOT": "1"}) is True # the chroot said so + +# ...AND THE GUARD ON THAT INFERENCE FAILED AT THE ONE MOMENT IT MATTERED. +# The fallback was "/usr/src/lfs-pkgusr exists AND /mnt/lfs/usr/src does +# not". On a FRESH build the partition is mounted and still EMPTY, so on a +# host these tools had built, `lfs run` refused with "you are inside the LFS +# chroot". Ask the kernel: a chroot is a root that is not PID 1's. +real = lfs._root_differs_from_pid1 +try: + lfs._root_differs_from_pid1 = lambda: False # kernel: same root + assert check({}) is False, "the kernel says host and it is not believed" + lfs._root_differs_from_pid1 = lambda: True # kernel: a chroot + assert check({"LFS": "/mnt/lfs"}) is True, "a stray $LFS outvotes the kernel" + # and when the kernel cannot be asked (no /proc, not root), the last + # resort tests the CONFIGURED mount point, not a hardcoded one + lfs._root_differs_from_pid1 = lambda: None + assert check({"LFS": "/mnt/lfs"}) is False + src = open(sys.argv[1]).read() + i = src.index("def inside_chroot():") + assert '"/mnt/lfs/usr/src"' not in src[i:i + 2000], \ + "the fallback still tests a hardcoded mount point" + assert 'load_config().get("lfs_mount")' in src[i:i + 2000] +finally: + lfs._root_differs_from_pid1 = real + +# A REFUSAL MUST IDENTIFY ITSELF. A host was refused for days by a message +# 1.14.1 had already fixed -- the binary being run was 1.11.26, because +# `make install` was going somewhere else. Nothing on screen said which copy +# was talking, so a stale install looked like a bug in the current one. +src = open(sys.argv[1]).read() +i = src.index("def _refuse_inside_chroot(") +blk = src[i:i + 3000] +for need in ("sys.argv[0]", "LFS_VERSION", "_build_id()", "because"): + assert need in blk, "the chroot refusal does not say %s" % need +PYCHR + + # PRESENT-AND-EMPTY IS AN ANSWER. `"pkgusr_prefix": ""` means NO prefix. + # The bash reader failed `[ -n "$v" ]` on it and fell back to the default + # `p`, so lfs-helper called the account p_mako while packagemanager -- + # which honours empty -- asked for mako. Two tools, one account, two + # names. + _pfxd="$T/prefixcfg"; mkdir -p "$_pfxd" + printf '{\n "pkgusr_prefix": "",\n "collector_prefix": "nimgnu"\n}\n' > "$_pfxd/empty.json" + printf '{\n "pkgusr_prefix": "pkg",\n "collector_prefix": "nimgnu"\n}\n' > "$_pfxd/set.json" + for _c in empty set; do + _pv="$({ _slice_fn "$helper_src" _read_pkgusr_prefix \ + | sed "s|/usr/share/lfs/config.json|$_pfxd/$_c.json|" + echo 'v="$(_read_pkgusr_prefix)"; echo "$?:[$v]"'; } | bash 2>/dev/null)" + case "$_c:$_pv" in + empty:0:\[\]|set:0:\[pkg\]) ;; + *) _p="$_p;the $_c prefix config is read as $_pv" ;; + esac + done + + # AND "CANNOT JUDGE" IS NOT AN ANSWER. `update --reinstall mako` on a + # package that was never installed reported "no completed-install record" + # and stopped -- a true statement about a question the user did not mean + # to ask. When they NAME packages, point at install. + grep -q 'they are NOT ' "$pm_src" \ + || _p="$_p;update reports no record without saying to install instead" + grep -q 'packagemanager install %s --run' "$pm_src" \ + || _p="$_p;the not-installed hint does not give the command" + + # TYPE WHAT YOU WERE SHOWN. BLFS anchors are not all lower case (Mako, + # PyYAML, Cython keep their project capitalisation), search is + # case-insensitive and every other command was not: + # blfs search mako -> Mako + # packagemanager update mako -> ERROR: 'mako' not found + grep -q '^def resolve_anchor' "$_bt" \ + || _p="$_p;an anchor typed in the wrong case is not found" + python3 - "$_bt" <<'PYCASE' || _p="$_p;resolve_anchor does not match case-insensitively" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +class FakeBook: + packages = {"Mako": 1, "libxml2": 1} +fb = FakeBook() +assert b.resolve_anchor(fb, "mako") == "Mako" +assert b.resolve_anchor(fb, "Mako") == "Mako" +assert b.resolve_anchor(fb, "libxml2") == "libxml2" +# an anchor that really is absent comes back unchanged, for the error message +assert b.resolve_anchor(fb, "nosuch") == "nosuch" +PYCASE + + # AND THE EDITED SCRIPT MUST ACTUALLY BE FOUND. _script_name_version + # matched only name_version="..." while the generator has emitted book + # strings SINGLE-quoted since 1.12.18 -- so the local script never + # matched and every user edit was silently ignored. + python3 - "$pm_src" <<'PYQUOTE' || _p="$_p;an edited local script is not recognised (quote style)" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp() +for q in ("'", '"'): + p = os.path.join(d, "s" + ("1" if q == "'" else "2")) + open(p, "w").write("name_version=%sxmlto-0.0.29%s\n" % (q, q)) + assert pm._script_name_version(p) == "xmlto-0.0.29", (q, pm._script_name_version(p)) +PYQUOTE + + # AN EDIT MUST NOT VANISH IN SILENCE. The script in the package home is + # regenerated whenever the tools move on, and a user who had fixed a dead + # download URL in it lost the fix with no message. + grep -q 'your version of %s differed' "$pm_src" \ + || _p="$_p;a regenerated script overwrites the user's edits silently" + # ...and says what actually happened to it. The old wording claimed the + # versioned file "is never regenerated" while regenerating it (1.14.21). + grep -q 'this file IS the one you edit' "$pm_src" \ + || _p="$_p;nothing says which file was rewritten, or why" + grep -q 'Your copy is the .edited file' "$pm_src" \ + || _p="$_p;nothing says where the user's version went" + + # A GENERIC NAME IN THE ENVIRONMENT IS A COLLISION. libvpx's Makefile + # has `BUILD_ROOT?=.` and uses it in CFLAGS -- and `?=` yields to the + # environment, so exporting our BUILD_ROOT sent every compile looking + # for vpx_config.h in the source root. + grep -q 'LFS_BUILD_ROOT' "$PHASES_LIB" \ + || _p="$_p;the generated script does not take the runner's private build-root name" + grep -qE "envpass=.*[^_]BUILD_ROOT='" "$helper_src" \ + && _p="$_p;the runner still exports a bare BUILD_ROOT into every build" + # (the emitted line itself: the runner's value must win over a stray one) + grep -q 'LFS_BUILD_ROOT:-' "$PHASES_LIB" \ + || _p="$_p;the script does not prefer the runner's build root" + # ...and when nothing says otherwise the sources are in ~/src, not $PWD: + # `su - p_rust -c 'bash ~/install_p_rust install'` starts in the HOME, so + # _enter_build looked beside the script and rust's ./x.py vanished. + grep -q 'BUILD_ROOT="\$HOME/src"' "$PHASES_LIB" \ + || _p="$_p;a phase run from the home cannot find the unpacked sources" + + # A CONDITIONAL BLOCK IS CONDITIONAL WHEREVER IT APPEARS. The phased + # extractor guarded them; extract_commands (the CONFIGURATION section) + # did not, so cyrus-sasl ran `make install-saslauthd` -- introduced by + # "If you need to run the saslauthd daemon at system startup ... install + # the saslauthd.service unit included in blfs-systemd-units". + _ec="$(sed -n '/^def extract_commands(/,/^def /p' "$_bt")" + printf '%s' "$_ec" | grep -q '_conditional_guard' \ + || _p="$_p;conditional blocks in the configuration section are run unconditionally" + + # A ONE-PACKAGE PLAN WITH MISSING DEPENDENCIES IS A TRAP. `install + # ffmpeg --run` (no --recursive) planned one package, said "1 to build", + # and died on `libass >= 0.11.0 not found` -- libass was never installed + # and was never going to be. The plan knew. + grep -q 'this plan builds ONLY' "$pm_src" \ + || _p="$_p;a single-package plan does not mention its missing dependencies" + grep -q -- '--run --recursive' "$pm_src" \ + || _p="$_p;nothing suggests --recursive when dependencies are missing" + + # ...AND IT IS REFRESHED, NOT JUST REPORTED. Telling the user to pass + # --regenerate cost several rounds each on brotli, rust, xcb-utilities and + # ffmpeg. A stale script the user has NOT edited is regenerated; one + # they have edited is kept, with the command to take the new one. + grep -q '_stale_unedited' "$pm_src" \ + || _p="$_p;a stale script is reported but still used" + python3 - "$pm_src" <<'PYEDITED' || _p="$_p;the edited-script test is wrong" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp() +plain = os.path.join(d, "plain"); open(plain, "w").write("### generated by blfs 1.1\n") +assert pm._script_is_user_edited(plain) is False +marked = os.path.join(d, "marked"); open(marked, "w").write("# EDITED by me\n") +assert pm._script_is_user_edited(marked) is True +bak = os.path.join(d, "bak"); open(bak, "w").write("x\n"); open(bak + ".edited", "w").write("y\n") +assert pm._script_is_user_edited(bak) is True +PYEDITED + + # AND A LOCAL SCRIPT FROM AN OLDER GENERATOR CARRIES THE OLD BUGS. + # brotli failed twice on the same line after the generator was fixed: the + # script in the package home was written by the previous blfs, and the + # only notice was a one-line hint lost in a 200-package run. + # (1.13.40: an unedited stale script is regenerated instead of merely + # reported; an EDITED one is kept and the user is told how to take the + # new generator's version) + grep -q 'this file has YOUR EDITS -- keeping it' "$pm_src" \ + || _p="$_p;a script from an older generator is used without saying so" + + # `packagemanager script <phase> <pkg>` MUST USE THE ACCOUNT. It took the + # anchor verbatim: `su - rust -c 'bash ~/install_rust install'` -- no such + # account, no such script. + # two names: the ACCOUNT for su and the script path, the ANCHOR for blfs + grep -q 'name = pkgusr_name(anchor)' "$pm_src" \ + || _p="$_p;packagemanager script runs as the anchor instead of the package user" + grep -q 'run_blfs(\["script", anchor' "$pm_src" \ + || _p="$_p;packagemanager script asks the book for the account name" + # ...and it must DELETE the account, not the anchor: `remove --purge mako` + # ran `userdel mako`, left p_mako in /etc/passwd, and printed + # "mako purged." with the files and home already gone. + grep -q 'userdel", _acct' "$pm_src" \ + || _p="$_p;purge deletes the anchor instead of the package user" + grep -q 'NOT fully purged' "$pm_src" \ + || _p="$_p;purge claims success even when the account survives" + + # `remove <anchor> --purge` looked the ANCHOR up in passwd and crashed: + # KeyError: getpwnam(): name not found: 'cargo-c' + grep -q 'pwd.getpwnam(pkgusr_name(name)).pw_dir' "$pm_src" \ + || _p="$_p;remove --purge looks up the anchor instead of the account" + + # AND AN INSTALL IT RAN MUST BE RECORDED: only pm-install writes the + # manifest, so `packagemanager script install rust` succeeded and the + # planner still called Rustc "new" on the next run. + grep -q '"lfs-helper", "record-install", name' "$pm_src" \ + || _p="$_p;a phase that installed is not recorded, so it is built again" + # a local `import pwd` inside one branch shadows the module-level one, so + # a later `if pwd is not None` crashed with UnboundLocalError + python3 -c "import ast,sys; ast.parse(open(sys.argv[1]).read())" "$pm_src" \ + || _p="$_p;packagemanager does not parse" + python3 - "$pm_src" <<'PYPWD' || _p="$_p;a branch-local import of pwd shadows the module-level one" +import sys, ast +tree = ast.parse(open(sys.argv[1]).read()) +for fn in [n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)]: + # An `import pwd` makes the name local to the WHOLE function. That is + # safe when the use sits in the same block as the import (import then + # use, one try); it is a latent UnboundLocalError when a use appears + # BEFORE the import line or in a different branch -- which is exactly + # what crashed cmd_script. + imports = [n.lineno for n in ast.walk(fn) + if isinstance(n, ast.Import) + and any(a.name == "pwd" and not a.asname for a in n.names)] + if not imports: + continue + first = min(imports) + bad = [n.lineno for n in ast.walk(fn) + if isinstance(n, ast.Name) and n.id == "pwd" + and isinstance(n.ctx, ast.Load) and n.lineno < first] + # a use far below the import block is also suspect: require it to be + # within a few lines of an import of pwd + for n in ast.walk(fn): + if (isinstance(n, ast.Name) and n.id == "pwd" + and isinstance(n.ctx, ast.Load) + and not any(0 <= n.lineno - i <= 3 for i in imports)): + bad.append(n.lineno) + assert not bad, "%s: bare pwd at %s with a function-local import" % (fn.name, bad) +PYPWD + + # WRITING A LOGIN PROFILE IS NOT RUNNING IT. rust's config writes + # /etc/profile.d/rustc.sh and the page sources it; those snippets call + # pathprepend, a BLFS login-profile helper absent from a build shell, so + # a package that had installed 2390 files failed at the last line. + python3 - "$_bt" <<'PYPROF' || _p="$_p;a package sources a login profile during its build" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_profile_sourcing( + "cat > /etc/profile.d/rustc.sh << EOF\nX\nEOF\nsource /etc/profile.d/rustc.sh") +assert "# source /etc/profile.d/rustc.sh" in out, out +assert out.splitlines()[0].startswith("cat >"), out # writing it is kept +PYPROF + + # RUST DRIVES EVERYTHING THROUGH x.py, and two things went wrong: + # `./x.py test` (41425 tests) was not recognised as a test driver, and the + # book's "If sudo or su is invoked for switching to the root user..." + # caused `./x.py install` -- the whole point of the package -- to be + # commented as conditional. A sentence about privilege mechanics guards + # nothing. + python3 - "$_bt" <<'PYRUST' || _p="$_p;rust's tests run, or its install is skipped" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_test_runs( + "./x.py build\n./x.py test --verbose | tee rustc-testlog\n" + "grep '^test result:' rustc-testlog |\n awk '{print}'\n./x.py install") +lines = out.splitlines() +assert lines[0] == "./x.py build", out +assert lines[-1] == "./x.py install", out # build and install survive +assert "# ./x.py test" in out, out # the run goes +assert "# grep" in out and "# " in lines[-2], out # and the whole pipeline +class P: + name = "p" + def __init__(s, t): s.t = t + def get_text(s): return s.t +class Pre: + def __init__(s, t): s.p = P(t) + def find_previous(s, names): return s.p +assert b._conditional_guard(Pre( + "If sudo or su is invoked for switching to the root user, ensure X")) is None +PYRUST + + # COMMENTING A COMMAND MUST NOT STRAND THE ONE BEFORE IT. libpwquality + # ended `make install &&` with the pip line after it commented (1.13.3), + # and bash refused the whole file: syntax error near unexpected token `}'. + python3 - "$_bt" <<'PYDANGLE' || _p="$_p;commenting a command leaves the previous one ending in &&" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._repair_dangling_operators("make install &&\n# pip3 install x") +assert out.splitlines()[0] == "make install", out +# a real chain is untouched +keep = b._repair_dangling_operators("./configure &&\nmake &&\nmake install") +assert keep == "./configure &&\nmake &&\nmake install", keep +# and a pipe with nothing after it goes too +assert b._repair_dangling_operators("grep x f |\n# awk y").splitlines()[0] == "grep x f" +PYDANGLE + + # A PIP INSTALL WHOSE WHEEL NOTHING BUILT. brotli's optional Python + # bindings are two steps; the `pip3 wheel -w dist` is conditional and + # commented, the `pip3 install --find-links dist` was not -- so pip + # looked in a directory nobody made, AFTER the C library installed fine. + python3 - "$_bt" <<'PYPIP' || _p="$_p;a pip install runs against a wheel directory nothing fills" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +orphan = b._comment_orphan_pip_installs( + "# pip3 wheel -w dist $PWD\npip3 install --no-index --find-links dist Brotli") +assert "# pip3 install" in orphan, orphan +kept = b._comment_orphan_pip_installs( + "pip3 wheel -w dist $PWD\npip3 install --find-links dist Foo") +assert not kept.splitlines()[1].startswith("#"), kept +PYPIP + + # A FUNCTION DEFINITION IS NOT A COMMAND TO SKIP. Xorg's loop defines + # do_build/do_test/do_install and calls them later; the test filter + # commented `do_test() { make check; }` and the loop then called a + # function that did not exist -- xorg7-lib "succeeded" without installing + # libX11, and mesa died on `Dependency "x11" not found` with xorg7-lib + # recorded as installed. + python3 - "$_bt" <<'PYDEFS' || _p="$_p;a filter comments a function definition" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +src = ("do_build() { make; }\n" + "do_test() { make check; }\n" + "do_pdf() { make -C doc pdf; }\n" + "do_x() { pip3 install --find-links dist X; }\n" + "make check") +for fn in ("_comment_test_runs", "_comment_doc_format_builds", + "_comment_orphan_pip_installs"): + out = getattr(b, fn)(src) + for line in out.splitlines(): + assert not (line.startswith("# ") and "()" in line), (fn, line) +# ...and a real command is still skipped +assert "# make check" in b._comment_test_runs(src) +PYDEFS + + # RUNNING THE TEST SUITE IS NOT BUILDING THE PACKAGE. NSS's page puts + # `cd tests && ./all.sh` in the BUILD block, so an ordinary install ran + # 77549 tests -- hours -- and then failed the package on a test artifact. + # The script already has a test phase, gated on PM_TEST=1. + python3 - "$_bt" <<'PYTESTRUN' || _p="$_p;a book test suite still runs during a normal build" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_test_runs("make\ncd tests &&\nHOST=x ./all.sh\ncd ../\nmake install") +lines = out.splitlines() +assert lines[0] == "make" and lines[-1] == "make install", out +# the run, the cd in and the cd back out all go +assert all(l.startswith("#") for l in lines[1:-1]), out +# a package with no test run is untouched +assert b._comment_test_runs("make\nmake install") == "make\nmake install" +# and the make-based drivers are caught too +assert "# make check" in b._comment_test_runs("make check") +PYTESTRUN + + # A MATCH ON A CONTINUATION LINE belongs to the command above it: + # rsync -vrltLW --delete ... \ + # rsync://fate-suite.ffmpeg.org/... fate-suite/ + # only the second line matches, and the first ran on its own. + python3 - "$_bt" <<'PYCONT' || _p="$_p;a test command split over lines is only half commented" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_test_runs("make install &&\nrsync -a \\\n rsync://host/fate x") +lines = out.splitlines() +assert lines[0] == "make install &&", out +assert lines[1].startswith("# rsync -a"), out +assert lines[-1].startswith("#"), out +PYCONT + + # A LOOP THAT TYPESETS goes as a unit. ffmpeg wraps texi2pdf/texi2dvi/ + # dvips in `pushd doc && for ... done &&`; commenting only the texi2* + # lines would leave an empty loop body. And `make fate` is ffmpeg's test + # suite, which fetches 40GB of samples over rsync first. + python3 - "$_bt" <<'PYTYPESET' || _p="$_p;a typesetting loop or the fate suite still runs" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_typeset_blocks( + "make &&\npushd doc &&\nfor D in x\ndo\n texi2pdf $D.texi\ndone &&\npopd &&\nmake install") +lines = out.splitlines() +assert lines[0] == "make &&", out +assert lines[-1] == "make install", out +assert all(l.startswith("#") for l in lines[2:-1]), out # the whole block +# a loop that does NOT typeset is untouched +keep = b._comment_typeset_blocks("for D in x\ndo\n make -C $D\ndone") +assert not any(l.startswith("#") for l in keep.splitlines()), keep +# fate is a test driver +assert "# make fate" in b._comment_test_runs("make fate SAMPLES=x") +assert "# make fate-rsync" in b._comment_test_runs("make fate-rsync SAMPLES=x") +PYTYPESET + + # TYPESETTING THE MANUAL IS NOT INSTALLING THE PACKAGE. libassuan's page + # ends with `make -C doc pdf ps`, which needs TeX -- and --disable-doc + # does not help, because nothing is misconfigured: the command asks for + # PDF outright. (That wrong fix shipped three times before the script + # was read.) + python3 - "$_bt" <<'PYDOCFMT' || _p="$_p;a typeset manual can still fail a library build" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_doc_format_builds("make\nmake -C doc html\nmake -C doc pdf ps") +lines = out.splitlines() +assert lines[0] == "make", out +assert "make -C doc html" in out and "# make -C doc html" not in out, out +assert "# make -C doc pdf ps" in out, out +PYDOCFMT + + # A COMMAND WHOSE INPUT NOTHING PRODUCES is not part of installing: + # "grep -A9 summary *make_check.log" reads logs written by the optional + # test step, and under set -e it failed the whole page. + python3 - "$_bt" <<'PYLOG' || _p="$_p;a command reading a log nothing writes still runs" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +orphan = b._comment_orphan_log_readers("make install\ngrep -A9 summary *make_check.log") +assert "# grep -A9" in orphan, orphan +kept = b._comment_orphan_log_readers("make check | tee x.log\ngrep Summary x.log") +assert not kept.splitlines()[1].startswith("#"), kept +PYLOG + _bk_meta="$(ls "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*.html 2>/dev/null | head -1)" + if [ -n "$_bk_meta" ]; then + _metad="$T/meta"; mkdir -p "$_metad" + python3 "$_bt" --book-file "$_bk_meta" script xorg7-lib -o "$_metad" >/dev/null 2>&1 + _metaf="$(ls "$_metad"/install_Xorg-Libraries 2>/dev/null | head -1)" + if [ -n "$_metaf" ]; then + _up="$(sed -n '/^_pp_unpack() {/,/^}/p' "$PHASES_LIB")" + printf '%s' "$_up" | grep -q 'no single tarball' \ + || _p="$_p;the meta page's unpack does not bail out before fetching" + grep -q "^pkg=''" "$_metaf" \ + || _p="$_p;the meta page's script claims a tarball" + # ...the build must RESUME where those commands left off (the + # page does `mkdir lib && cd lib`, and its loop reads + # ../lib-7.md5, which only resolves from inside lib/) ... + printf '%s' "$(awk '/^_pp_enter_source\(\) \{/,/^\}/' "$PHASES_LIB")" \ + | grep -q '_pp_cwd_marker' \ + || _p="$_p;a meta build starts at the root, not where its downloads went" + # ...and a loop over a list file must FAIL when the file is gone, + # instead of iterating zero times and exiting 0 + grep -q 'the list this build loops over is missing or empty' "$_metaf" \ + || _p="$_p;an empty list loop reports success" + # ...and the page's OWN download commands must run before that + # early return: Xorg Libraries writes lib-7.md5 and wgets its + # list from it, and returning first left the build loop reading + # a file nobody had created -- it built nothing and exited 0. + # (they are the page's prepare_pkg, which the runner calls + # before it decides there is no tarball) + sed -n '/^prepare_pkg() {/,/^}/p' "$_metaf" | grep -q 'lib-7.md5' \ + || _p="$_p;the meta page's own download commands are not run before unpack" + _dl="$(printf '%s' "$_up" | grep -n 'prepare_pkg$' | head -1 | cut -d: -f1)" + _rt="$(printf '%s' "$_up" | grep -n 'no single tarball' | head -1 | cut -d: -f1)" + if [ -n "$_dl" ] && [ -n "$_rt" ] && [ "$_dl" -gt "$_rt" ]; then + _p="$_p;the meta page returns before running its own download commands" + fi + # ...and the BUILD phase must not then demand an unpacked source: + # XCB-Utilities died there after unpack had correctly done nothing + _eb="$(awk '/^_pp_enter_source\(\) \{/,/^\}/' "$PHASES_LIB")" + printf '%s' "$_eb" | grep -q '\[ -z "\${pkg:-}" \]' \ + || _p="$_p;a page with no tarball fails when the build phase enters it" + bash -n "$_metaf" || _p="$_p;the meta page's script does not parse" + fi + rm -rf "$_metad" + fi + + # THE SAME PROBLEM, GENERALLY: a variable the book expects the user to + # export. Unset, the shell expands it to nothing and configure gets an + # empty flag -- "expected an absolute directory name for --datarootdir:". + python3 - "$_bt" <<'PYENV' || _p="$_p;a variable the book expects the user to set is used empty" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +assert "TEXLIVE_PREFIX to be set" in b._required_env_guard("./configure --prefix=$TEXLIVE_PREFIX", set()) +assert b._required_env_guard("BUILD_ROOT=x\ncd $BUILD_ROOT", {"BUILD_ROOT"}).count("to be set") == 0 +assert b._required_env_guard('tar xf "$pkg"', {"pkg"}).count("to be set") == 0 +# THE SHELL'S OWN VARIABLES ARE NOT THE USER'S TO SET. A list containing +# EUID teaches the user to ignore the list -- ten of eleven entries in the +# first real run were false positives. +for shellvar in ("EUID", "UID", "PPID", "RANDOM", "LANG", "PWD"): + assert b._required_env_guard("echo $%s" % shellvar, set()).count("to be set") == 0, shellvar +# SINGLE-QUOTED TEXT IS NOT THE SHELL'S: $PACKAGE_PREFIX_DIR lives inside a +# sed expression that writes CMake, so the shell never expands it and the +# user was being asked about another language's variables. +sq = b._required_env_guard( + "sed -e '/X/i set(SAVE_PACKAGE_PREFIX_DIR \"${PACKAGE_PREFIX_DIR}\")' f &&\n" + "cmake -D CMAKE_INSTALL_PREFIX=$KF6_PREFIX ..", set()) +assert "PACKAGE_PREFIX_DIR to be set" not in sq, sq +assert "KF6_PREFIX to be set" in sq, sq +PYENV + # AND THE BOOK'S OWN ANSWER COMES FIRST: TeX Live's build uses + # $TEXLIVE_PREFIX and the book sets it in "Setting the PATH for TeX Live" + # -- a DIFFERENT section. Asking the user to supply it is asking them to + # repeat the book. + _bk_env="$(ls "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*.html 2>/dev/null | head -1)" + if [ -n "$_bk_env" ]; then + _envd="$T/pageenv"; mkdir -p "$_envd" + python3 "$_bt" --book-file "$_bk_env" script texlive -o "$_envd" >/dev/null 2>&1 + _envf="$(ls "$_envd"/install_texlive-* 2>/dev/null | head -1)" + if [ -n "$_envf" ]; then + grep -q 'TEXLIVE_PREFIX="/opt/texlive' "$_envf" \ + || _p="$_p;the book's own definition of TEXLIVE_PREFIX is not used" + grep -q 'needs TEXLIVE_PREFIX to be set' "$_envf" \ + && _p="$_p;the user is asked for a variable the book already defines" + fi + rm -rf "$_envd" + fi + python3 - "$pm_src" <<'PYWARN' || _p="$_p;nothing warns about an unset variable BEFORE the build starts" +import sys, os, io, contextlib, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp(); f = os.path.join(d, "install_x") +open(f, "w").write('[ -n "${SOME_PREFIX:-}" ] || { echo "this package needs SOME_PREFIX to be set"; }\n') +os.environ.pop("SOME_PREFIX", None) +err = io.StringIO() +with contextlib.redirect_stderr(err), contextlib.redirect_stdout(err): + pm._warn_unset_env("somepkg", f) +assert "SOME_PREFIX" in err.getvalue() +os.environ["SOME_PREFIX"] = "/x" +quiet = io.StringIO() +with contextlib.redirect_stderr(quiet), contextlib.redirect_stdout(quiet): + pm._warn_unset_env("somepkg", f) +assert quiet.getvalue().strip() == "" +PYWARN + grep -q '_pp_mirrors_for() {' "$PHASES_LIB" \ + || _p="$_p;a failed download is retried against the same host only" + # ($_bt is the GENERATOR here, not a generated script, so check what it + # emits rather than trying to run it.) + grep -q 'ftp.gnu.org/gnu/\$rest' "$PHASES_LIB" \ + || _p="$_p;the GNU fallback does not rewrite to a real ftp.gnu.org path" + grep -q 'mirrors.kernel.org/gnu/\$rest' "$PHASES_LIB" \ + || _p="$_p;there is only one GNU fallback mirror" + grep -q 'blfs/downloads' "$_bt" && grep -q 'pkgusr_book_mirror' "$PHASES_LIB" \ + || _p="$_p;there is no last-resort mirror for non-GNU hosts" + grep -q 'try \* 15' "$PHASES_LIB" \ + || _p="$_p;the retry backoff does not grow (5s three times is not a retry)" + grep -q 'wget -4 -c' "$PHASES_LIB" \ + || _p="$_p;a retried partial download would be saved beside itself" + grep -q 'try in 1 2 3' "$PHASES_LIB" || _p="$_p;the fetch does not retry" + grep -q '_pp_fetch "\$link"' "$PHASES_LIB" \ + || _p="$_p;the main tarball download does not go through the retry" + # The one loop sets a real BUILD_ROOT, so "look in ." stopped finding the + # tarball packagemanager staged into $HOME -- wget's own install tried to + # DOWNLOAD wget on a fresh 13.0-sysv bootstrap. The script owns the + # search: HOME, then /sources, then and only then the network. + grep -q 'for d in "\$HOME" "\$SOURCES_DIR" /sources' "$PHASES_LIB" \ + || _p="$_p;a staged tarball is invisible: the script only looks in its cwd before downloading" + grep -q '_pp_stage_file "\$f" || _pp_fetch "\$li"' "$PHASES_LIB" \ + || _p="$_p;additional files are always downloaded, even when already on the system" + # conditional blocks are recognised by the SENTENCE PATTERN, not a verb + # list: the list caught libuv's "If you installed sphinx..." and missed + # "If you BUILT the man page, install it", so the page was never made and + # its install failed on a missing file + grep -q 'if\\b|optionally\\b|should you\\b' "$_bt" \ + || _p="$_p;conditional command blocks are matched by a verb list, which will keep missing wordings" + grep -q '_pp_fetch "\$li"' "$PHASES_LIB" \ + || _p="$_p;additional downloads do not go through the retry" + grep -q 'wget -4 "\$link"' "$_bt" "$PHASES_LIB" \ + && _p="$_p;a bare non-retrying wget emission survives" + # the shims are emitted from a list now (1.12.79), so check the list and + # the shape rather than one hard-coded name + grep -qw 'systemctl' "$PHASES_LIB" \ + || _p="$_p;a missing systemctl still aborts generated scripts" + grep -q 'command -v "\$c" >/dev/null 2>&1 && continue' "$PHASES_LIB" \ + || _p="$_p;the systemd shims would shadow the real commands" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "generated scripts retry downloads and shim a missing systemctl" + fi fi -# ---- a snapshot must say what state it captured ----------------------------- # -# Restoring a snapshot named "chrosscompile_done" brought back a tree whose -# toolchain was ALREADY broken -- the snapshot had been taken after chapter 8 -# damaged it. Nothing said so, at save or at restore, so it looked like the -# restore had failed rather than the snapshot being bad. -python3 - "$LFS_TOOL" <<'PY70' -import sys, os, json, tempfile, importlib.machinery as m -store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store -os.makedirs(os.path.join(store, "books"), exist_ok=True) -json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, - open(os.path.join(store, "config.json"), "w")) -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -if not hasattr(lfs, "_toolchain_state"): - print(" FAIL a snapshot does not record the toolchain state"); sys.exit(1) +# ---- make-ca installed is not the same as certificates generated ------------ # +# `make-ca -g` prints "No update required!" and generates nothing when it +# considers its copy of certdata.txt current -- which it is straight after +# install. The install then looks successful while the system still has no +# certificate bundle, and every HTTPS fetch keeps warning. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def check_make_ca_generated" "$pmt" \ + && ok "an installed-but-ungenerated make-ca is noticed" \ + || bad "make-ca can be installed with no certificates and nothing said" + grep -q "make-ca -g --force" "$pmt" \ + && ok "it gives the command that actually generates them" \ + || bad "it does not say how to finish the job" + # and it must go quiet once a bundle exists + python3 - "$pmt" <<'PY62' +import sys, os, io, contextlib, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp() +fake = os.path.join(d, "make-ca") +open(fake, "w").write("#!/bin/sh\nexit 0\n") +os.chmod(fake, 0o755) +os.environ["PATH"] = d + os.pathsep + os.environ["PATH"] +pm._CA_BUNDLES = [os.path.join(d, "bundle.crt")] +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + pm.check_make_ca_generated() +warned = "no CA certificate bundle" in buf.getvalue() +open(os.path.join(d, "bundle.crt"), "w").write("CERTS") +buf2 = io.StringIO() +with contextlib.redirect_stdout(buf2), contextlib.redirect_stderr(buf2): + pm.check_make_ca_generated() +quiet = "no CA certificate bundle" not in buf2.getvalue() +if warned and quiet: + print(" PASS it warns only while the bundle is genuinely missing") +else: + print(" FAIL warned=%s quiet=%s" % (warned, quiet)); sys.exit(1) +PY62 + _count_rc $? +fi -def tree(headers=None, native=False): - t = tempfile.mkdtemp() - if headers: - p = os.path.join(t, "usr", "include", "c++", "15.2.0", headers, "bits") - os.makedirs(p) - open(os.path.join(p, "c++config.h"), "w").close() - if native: - os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) - return t -problems = [] -if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu")).startswith("ok"): - problems.append("a clean chapter-6 tree is not recognised") -if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu", native=True)) \ - .startswith("BROKEN"): - problems.append("a native gcc over the cross one is not detected") -if "unfinished" not in lfs._toolchain_state(tree()): - problems.append("a tree with no C++ headers is not detected") -# headers under a different target than configured -st = lfs._toolchain_state(tree("x86_64-lfs-linux-gnu")) -if not st.startswith("BROKEN"): - problems.append("headers under the wrong target are not detected") +# ---- prompts must allow line editing ---------------------------------------- # +# Without readline an arrow key inserts an escape sequence instead of moving +# the cursor, so a typo in a long answer -- a group name, a device path -- +# means retyping the whole thing. +for t in lfs blfs packagemanager; do + f="$(dirname "$LFS_TOOL")/$t" + [ -f "$f" ] || continue + grep -q "import readline" "$f" \ + || bad "$t: input() prompts have no line editing" +done +ok "the python tools import readline" + +# an optional import: a python built without readline must still run +python3 - "$LFS_TOOL" <<'PY64' +import sys src = open(sys.argv[1]).read() -if "Restoring it later will restore the breakage too" not in src: - problems.append("saving a broken tree gives no warning") -if problems: - for p in problems: print(" FAIL %s" % p) +i = src.index("import readline") +window = src[max(0, i - 200):i + 120] +if "try:" not in window or "ImportError" not in window: + print(" FAIL readline is imported unguarded; a python without it would fail") sys.exit(1) -print(" PASS a snapshot records and reports the state it captured") -PY70 +print(" PASS readline is optional, not required") +PY64 _count_rc $? -# ---- a snapshot label must not hide what it captured ------------------------ # -# "chapters 5-6: 22/22 steps" was all a snapshot recorded, so a tree where -# chapter 8 had already run -- and already broken the toolchain -- looked -# pristine. Restoring it restored the damage, and it read as a failed restore -# rather than a bad snapshot. -python3 - "$LFS_TOOL" <<'PY71' -import sys, os, json, tempfile, importlib.machinery as m -store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store -os.makedirs(os.path.join(store, "books"), exist_ok=True) -json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, - open(os.path.join(store, "config.json"), "w")) -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -problems = [] -if not hasattr(lfs, "_tree_state"): - problems.append("nothing reports the state of a tree") -if not hasattr(lfs, "cmd_bs_verify"): - problems.append("there is no way to ask what state a tree is in") +for f in lfs-helper; do + p="$(dirname "$LFS_TOOL")/$f" + [ -f "$p" ] || continue + # every prompt that reads an ANSWER needs -e; loops reading files do not + bare="$(grep -nE "read (-r )?(ans|check|answer|reply)\b" "$p" \ + | grep -v -- "-e" | head -3)" + [ -z "$bare" ] \ + || bad "$f: prompt without line editing: $(echo "$bare" | head -1)" +done +ok "the bash prompts use read -e" -t = tempfile.mkdtemp() -os.makedirs(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress")) -os.makedirs(os.path.join(t, "usr", "src", "zlib")) -os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) -p = os.path.join(t, "usr", "include", "c++", "15.2.0", - "x86_64-nimgnu-linux-gnu", "bits") -os.makedirs(p); open(os.path.join(p, "c++config.h"), "w").close() -names = [n for n, _s, _t in lfs.CROSSCHAIN_STEPS] -json.dump({"book": "12.4", "done": names, "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, - open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "crosschain-progress.json"), "w")) -open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "steps-built"), "w").write("zlib\n") -state = dict(lfs._tree_state(t)) -# chapters 5-6 alone is not the whole story -if "chapters 7-9" not in state: - problems.append("chapter 7-9 progress is not reported") -if "WARNING" not in state: - problems.append("a native gcc in the tree is not flagged") -if not state.get("toolchain", "").startswith("BROKEN"): - problems.append("the toolchain state is not reported") -# and the snapshot must carry it +# and -r as well, or a backslash in an answer is eaten +if grep -qE "read -e [^r-]" "$(dirname "$LFS_TOOL")/lfs-helper" 2>/dev/null; then + bad "a prompt uses -e without -r" +else + ok "prompts keep backslashes literal (-r)" +fi + +# ---- starting the whole build over -------------------------------------------# +# Deleting a built system is the most destructive thing these tools can do -- +# potentially weeks of compilation -- so it has to say exactly what goes, what +# stays, and refuse outright in the one case that could damage the host. +python3 - "$LFS_TOOL" <<'PY65' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() src = open(sys.argv[1]).read() -if '"state":' not in src: - problems.append("the snapshot metadata does not record the tree state") +if not hasattr(lfs, "cmd_bs_restart"): + print(" FAIL no way to start the build over"); sys.exit(1) +body = src[src.index("def cmd_bs_restart"):src.index("def cmd_bs_set_book")] +problems = [] +# deleting through a bind-mounted /dev would reach the running system +if "the chroot is still mounted" not in body: + problems.append("it does not refuse while the chroot is mounted") +for d in ('"dev/pts", "dev/shm", "dev", "proc", "sys", "run"',): + if d not in body: + problems.append("the mount check does not cover every virtual filesystem") +# dry run by default, and a confirmation that cannot be typed by accident +if "args.run" not in body: + problems.append("restart is not dry-run by default") +if "Type the mount point" not in body: + problems.append("the confirmation is too easy to give by accident") +# sources are expensive to re-download and unchanged by a build +if "keep_sources" not in body: + problems.append("the downloaded sources are not kept") +# and it should point at the cheaper alternative +if "snapshot restore" not in body: + problems.append("it does not mention restoring a snapshot instead") if problems: - for p2 in problems: print(" FAIL %s" % p2) + for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS a tree's real state is reported and recorded with snapshots") -PY71 +print(" PASS restart warns, confirms, keeps the sources and spares the host") +PY65 _count_rc $? -# ---- staged installs must cope with the merged-/usr symlinks ---------------- # -# Book 4.2 makes /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib. A -# package installing into /bin produces a REAL bin/ directory in the staging -# tree, and copying that over the symlink fails: -# cp: cannot overwrite non-directory '/bin' with directory '.../bin' -# That broke every package installing into those paths -- most of chapter 7 -- -# on a completely fresh build. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_resolve_staged_symlink_dirs()" "$helper_src" \ - && ok "staged directories are folded onto the tree's symlinks" \ - || bad "a package installing into /bin cannot be merged" - ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" - i_fold="$(printf '%s' "$ms" | grep -n "_resolve_staged_symlink_dirs" | head -1 | cut -d: -f1)" - i_cp="$(printf '%s' "$ms" | grep -n '_install_staged_tree' | head -1 | cut -d: -f1)" - if [ -n "$i_fold" ] && [ -n "$i_cp" ] && [ "$i_fold" -lt "$i_cp" ]; then - ok "the folding happens before the copy" - else - bad "the copy runs first and still fails on the symlinks" - fi - # driven by the live tree, not a hardcoded list of names - rs="$(sed -n '/^_resolve_staged_symlink_dirs() {/,/^}/p' "$helper_src")" - case "$rs" in - *'[ -L "$live" ] || continue'*) - ok "it folds whatever the tree actually has as a symlink" ;; - *) bad "the symlink names are hardcoded" ;; - esac +# ---- sources and build scratch are two directories -------------------------- # +# They used to be one, and every problem in this area came from that. A build +# unpacked into the tarball store, dropped .cc-* markers there, and left both +# owned by whoever built -- so a rebuild had to pick debris out of the thing it +# most needed to keep. Split, and the whole class of problem is structural: +# /sources holds only what get-sources downloaded, and BUILD_ROOT is deleted +# whole. +python3 - "$LFS_TOOL" <<'PY65B' +import sys, os, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +# the runner decides both directories, from the script's stage variable +body = open(os.path.join(os.path.dirname(sys.argv[1]), "lfs-phases")).read() +layout = body[body.index("_pp_layout() {"):body.index("_pp_systemd_shims() {")] - # behaviour: bin/ and sbin/ fold, usr/bin/ is left alone - sd="$T/stagedirs"; mkdir -p "$sd/usr/bin" "$sd/usr/sbin" - ln -sf usr/bin "$sd/bin"; ln -sf usr/sbin "$sd/sbin" - mkdir -p "$sd/stage/bin" "$sd/stage/usr/bin" - echo m > "$sd/stage/bin/mount"; echo w > "$sd/stage/usr/bin/wall" - ( SNAP_ROOT="$sd"; say() { :; } - eval "$rs" - _resolve_staged_symlink_dirs "$sd/stage" ) >/dev/null 2>&1 - if [ -f "$sd/stage/usr/bin/mount" ] && [ ! -d "$sd/stage/bin" ] \ - && [ -f "$sd/stage/usr/bin/wall" ]; then - ok "bin/ is folded into usr/bin/ and nothing else is disturbed" - else - bad "the folding did not produce the expected staging tree" - fi -fi +if 'BUILD_ROOT="$LFS/sources"' in layout: + problems.append("BUILD_ROOT still defaults to the tarball store") +for need in ('SOURCES_DIR="$LFS/sources"', 'BUILD_ROOT="$LFS/build"'): + if need not in layout: + problems.append("the runner does not set %s for a cross build" % need.split("=")[0]) -# ---- replacing a library a running system is using -------------------------- # -# `cp -a` opens the destination with O_TRUNC and writes. For a shared library -# that is currently mapped that is fatal: every running process sees the file -# shrink under it. Merging glibc that way killed the shell mid-install -- -# /usr/bin/env: error while loading shared libraries: -# /usr/lib/libc.so.6: file too short -# -- and the chroot could not start any program afterwards. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_install_staged_tree()" "$helper_src" \ - && ok "staged files are installed one at a time" \ - || bad "the merge still overwrites files in place" - it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" - case "$it" in - *'mv -f "$tmp" "$dest"'*) - ok "each file is renamed into place, never truncated" ;; - *) bad "files are still written over in place" ;; - esac - # the temp file must be in the SAME directory, or mv is a copy, not a rename - case "$it" in - *'tmp="$dest.pkgusr-new.$$"'*) - ok "the temporary is beside its destination, so mv is atomic" ;; - *) bad "the temporary is elsewhere and mv would copy" ;; - esac - ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" - case "$ms" in - *'cp -a "$stage"/. '*) bad "the wholesale cp -a is still there" ;; - *) ok "the wholesale copy is gone" ;; - esac +# generate a real script and read what it does +cmds = ["mkdir -v build\ncd build", "../configure", "make", "make install"] +s = lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-1.0", + "demo-1.0.tar.*", cmds) +if 'pkgusr_stage="cross"' not in s: + problems.append("a cross script does not say so") +# the markers must live in scratch, never in the tarball store +for marker in ("_pp_dir_marker=", "_pp_cwd_marker="): + for line in layout.splitlines(): + if marker in line and "SOURCES_DIR" in line: + problems.append("%s is written to the tarball store" % marker) + if marker in line and "$BUILD_ROOT/" not in line: + problems.append("%s is not written to BUILD_ROOT" % marker) +# unpack must extract into scratch +u = body[body.index("_pp_unpack() {"):body.index("_pp_enter_source() {")] +if 'cd "$BUILD_ROOT"' not in u: + problems.append("unpack does not work in BUILD_ROOT") +# and the symlink farm must be built before the tarball is looked for +if u.index("_pp_link_sources") > u.index("_pp_pick_glob"): + problems.append("unpack does not build the sources symlink farm first") - # behaviour: replacing a file a process holds open must not disturb it - ad="$T/atomic"; mkdir -p "$ad/usr/lib" "$ad/stage/usr/lib" - printf 'OLDOLDOLD' > "$ad/usr/lib/lib.so" - printf 'NEWNEWNEWNEW' > "$ad/stage/usr/lib/lib.so" - before_ino="$(stat -c %i "$ad/usr/lib/lib.so")" - ( SNAP_ROOT="$ad"; say() { :; }; warn() { :; } - eval "$it" - _install_staged_tree "$ad/stage" ) >/dev/null 2>&1 - after_ino="$(stat -c %i "$ad/usr/lib/lib.so")" - if [ "$before_ino" != "$after_ino" ] \ - && [ "$(cat "$ad/usr/lib/lib.so")" = "NEWNEWNEWNEW" ]; then - ok "the replacement is a new inode, so open files keep the old one" - else - bad "the file was overwritten in place (inode unchanged)" - fi -fi +# the chroot copy must map BOTH directories, not just one +c = lfs._chrootify(s) +if 'pkgusr_stage="chroot"' not in c: + problems.append("the chroot script does not say so") +if 'SOURCES_DIR="/sources"' not in layout: + problems.append("the runner does not point a chroot script at /sources") +if 'BUILD_ROOT="/build"' not in layout: + problems.append("the runner does not point a chroot script at /build") -# ---- staging is not worth paying for on every package ----------------------- # -# Staging costs roughly an extra copy of everything installed -- measurably -# slower on a big package. It is worth it only where an interrupted install -# cannot be recovered from: interrupt man-pages and you rebuild man-pages; -# interrupt gcc or glibc and there is no compiler and no libc left to rebuild -# them WITH. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^STAGE_PACKAGES_DEFAULT=" "$helper_src" \ - && ok "only the packages that cannot be recovered are staged" \ - || bad "every package pays the staging cost" - for p in gcc glibc binutils bash coreutils; do - grep -q "^STAGE_PACKAGES_DEFAULT=.*$p" "$helper_src" \ - || bad "$p is not staged, but an interrupted install would be fatal" - done - ok "the toolchain packages are all in the default list" +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS tarball store and build scratch are separate directories") +PY65B +_count_rc $? - ss="$(sed -n '/^_should_stage() {/,/^}/p' "$helper_src")" - # -tmp and -passN are the same package at an earlier stage - case "$ss" in - *'base="${name%-tmp}"'*) ok "the -tmp and -passN variants count too" ;; - *) bad "gcc-pass1 would not be staged" ;; - esac - case "$ss" in - *"LFS_STAGE_PACKAGES"*) ok "the list is configurable" ;; - *) bad "the list is hardcoded" ;; - esac - case "$ss" in - *"all) return 0"*"none) return 1"*) ok "all / none are accepted" ;; - *) bad "there is no way to turn staging fully on or off" ;; - esac - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - case "$cb" in - *"--stage)"*"--no-stage)"*) ok "a single build can override it" ;; - *) bad "staging cannot be forced or skipped per build" ;; - esac +# ---- a book command may reach a sibling tarball through .. ------------------ # +# The book does this, in Tcl: +# +# cd .. +# tar -xf ../tcl8.6.16-html.tar.gz --strip-components=1 +# +# `cd ..` leaves the build subdirectory, so ".." is then the package root's +# parent -- which used to be the sources directory and is now BUILD_ROOT. +# Splitting the two would break every command of this shape. +# +# The fix must not be a list of packages known to do it: that would be a second +# thing to keep in sync with the book, and wrong the first time a new release +# adds another. unpack_pkg symlinks EVERY downloaded file into BUILD_ROOT, so +# "../<anything-we-downloaded>" resolves for any package in any future book. +# This test builds a package that does exactly what Tcl does. +_sbx="$T/split" +mkdir -p "$_sbx/lfs/sources" "$_sbx/lfs/out" "$_sbx/mk/demo-2.0" "$_sbx/mk/doc" +cat > "$_sbx/mk/demo-2.0/configure" <<'EOF' +#!/bin/bash +cat > Makefile <<'MK' +all: + @echo compiling +install: + @true +MK +EOF +chmod +x "$_sbx/mk/demo-2.0/configure" +( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0.tar.gz" demo-2.0 ) +echo html > "$_sbx/mk/doc/index.html" +( cd "$_sbx/mk" && tar czf "$_sbx/lfs/sources/demo-2.0-html.tar.gz" doc ) +echo patch > "$_sbx/lfs/sources/demo-2.0-fix-1.patch" - # behaviour - ( STAGE_PACKAGES_DEFAULT="$(sed -n 's/^STAGE_PACKAGES_DEFAULT=\"\(.*\)\"$/\1/p' \ - "$helper_src" | head -1)" - eval "$ss" - _should_stage gcc && _should_stage glibc && _should_stage gcc-pass2 \ - && ! _should_stage man-pages && ! _should_stage zlib ) \ - && ok "gcc and glibc stage; man-pages and zlib do not" \ - || bad "the wrong packages are being staged" -fi +python3 - "$LFS_TOOL" "$_sbx/demo.sh" <<'PYSPLIT' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +cmds = [ + "mkdir -v build\ncd build", + "../configure", + "make", + "make install", + # the book's Tcl shape, verbatim in structure + "cd ..\ntar -xf ../demo-2.0-html.tar.gz -C ../../out\n" + "cp -v ../demo-2.0-fix-1.patch ../../out/patch.seen", +] +open(sys.argv[2], "w").write( + lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-2.0", + "demo-2.0.tar.*", cmds)) +PYSPLIT -# ---- the staging directory must belong to whoever builds -------------------- # -# It is created by root, but the build runs as the PACKAGE USER, so -# `make install DESTDIR=...` could not write into it: -# mkdir: cannot create directory '/.lfs-pkgusr/stage/gcc': Permission denied -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - case "$cb" in - *'chown "$owner:$owner" "$_stage"'*) - ok "the staging directory is given to the package user" ;; - *) bad "the package user cannot write into its own staging directory" ;; - esac - # and the parent must be traversable, or the chown alone is not enough - case "$cb" in - *'chmod 755 "$(dirname "$_stage")"'*) - ok "the staging parent is traversable" ;; - *) bad "the package user cannot reach into the staging directory" ;; - esac - # a root step needs no chown -- there is no package user to give it to - case "$cb" in - *'[ "$as_root" != 1 ] && user_exists "$owner"'*) - ok "a root step is not given away to a user that may not exist" ;; - *) bad "the chown would fail on a root step" ;; - esac - # the owner must be known BEFORE the stage is made, or it chowns to nothing - i_owner="$(printf '%s' "$cb" | grep -n 'owner="\$(pkg_owner_name' | head -1 | cut -d: -f1)" - i_stage="$(printf '%s' "$cb" | grep -n '_stage="\$(_stage_dir_for' | head -1 | cut -d: -f1)" - if [ -n "$i_owner" ] && [ -n "$i_stage" ] && [ "$i_owner" -lt "$i_stage" ]; then - ok "the owner is resolved before the staging directory is made" - else - bad "the staging directory is created before the owner is known" - fi +# The suite exports SOURCES_DIR/BUILD_ROOT for other tests; clear them here so +# this exercises the DEFAULTS a real build would get. +( cd "$_sbx" && env -u SOURCES_DIR -u BUILD_ROOT LFS="$_sbx/lfs" \ + bash demo.sh all ) >"$_sbx/log" 2>&1 +_sp="" +[ -d "$_sbx/lfs/out/doc" ] || _sp="$_sp;a sibling tarball reached through .. was not found" +[ -f "$_sbx/lfs/out/patch.seen" ] || _sp="$_sp;a patch reached through .. was not found" +# and the whole point: a full build must not have touched the tarball store +_left="$(ls -A "$_sbx/lfs/sources" | grep -v '^demo-2\.0' | tr '\n' ' ' || true)" +[ -z "$_left" ] || _sp="$_sp;a build wrote into the tarball store: $_left" +[ -d "$_sbx/lfs/build/demo-2.0" ] || _sp="$_sp;the package was not unpacked into BUILD_ROOT" +[ -f "$_sbx/lfs/build/.pkgusr-demo-2.0.dir" ] || _sp="$_sp;build markers did not land in BUILD_ROOT" +[ -L "$_sbx/lfs/build/demo-2.0-html.tar.gz" ] || _sp="$_sp;the farm entry is not a symlink" +if [ -n "$_sp" ]; then + printf '%s\n' "${_sp#;}" | tr ';' '\n' | while IFS= read -r _m; do + [ -n "$_m" ] && bad "$_m" + done + sed 's/^/ /' "$_sbx/log" | tail -8 +else + ok "'../<tarball>' resolves for any package, and /sources stays clean" fi -# ---- a package must own the files its earlier stages installed -------------- # -# The same package is built more than once: ncurses in chapter 6 (as the lfs -# user), perl-tmp in chapter 7 (as root), then the real one in chapter 8 as -# its package user. The earlier files keep the earlier owner, and the package -# user cannot overwrite or re-mode them: -# cp: cannot create regular file '/usr/bin/tic': Permission denied -# Couldn't chmod 644 /usr/lib/perl5/.../perl.pod: Operation not permitted -# One cause, two chapters -- so one mechanism, applied to every package. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_claim_earlier_stages()" "$helper_src" \ - && ok "a package claims the files its earlier stages installed" \ - || bad "a rebuild cannot overwrite its own earlier files" - ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src")" - # every naming shape the same package takes across the book - # the stages are built from a loop now, so check the suffixes it appends - case "$ce" in - *'"$stem-tmp" "$stem-pass1" "$stem-pass2"'*) - ok "the -tmp and -passN stages are all claimed" ;; - *) bad "an earlier stage suffix is missing" ;; - esac - # ONLY what the build left behind -- never another package's file - case "$ce" in - *"root|lfs|UNKNOWN) ;;"*) - ok "only root- and lfs-owned paths are claimed" ;; - *) bad "it could take a file belonging to another package" ;; - esac - # directories too, or the package cannot create files in them - case "$ce" in - *'.dirs'*) ok "the directories are claimed as well as the files" ;; - *) bad "the package could not create files in its own directories" ;; - esac - # shared install dirs stay shared - case "$ce" in - *"is_install_dir"*) ok "shared install directories are left alone" ;; - *) bad "a shared directory could be taken by one package" ;; - esac - # and it must run BEFORE the build, not after it fails - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - i_claim="$(printf '%s' "$cb" | grep -n '_claim_earlier_stages' | head -1 | cut -d: -f1)" - i_build="$(printf '%s' "$cb" | grep -n 'building \$name' | head -1 | cut -d: -f1)" - if [ -n "$i_claim" ] && [ -n "$i_build" ] && [ "$i_claim" -lt "$i_build" ]; then - ok "the claim happens before the build starts" - else - bad "the claim runs too late to help" - fi -fi - -# ---- checks that silently always passed --------------------------------------# -# From an independent review of `lfs` against the book. Each of these hid a -# real failure rather than reporting it. -python3 - "$LFS_TOOL" <<'PY73' -import sys, importlib.machinery as m +# ---- /sources has ONE owner convention, and it is the book's ---------------- # +# Three rules had drifted apart, and the result on disk was a mixture: +# +# drwxrwxrwt 1 9998 install /mnt/lfs/sources/ +# -rw-r--r-- 1 lfs 9998 acl-2.3.2.tar.xz +# +# * lfs's _chown_tree_to_lfs added "sources" to the 4.3 handover set, so it +# chowned /sources to the HOST's lfs user. Book 4.3 does no such thing. +# * lfs-helper carried /sources in install_dirs_list AS WELL AS in +# never_adopt_list -- the exact conflation the comment above +# never_adopt_list warns about, the one that made /root root:install. +# verify --fix then made it root:install 775 and the seal made it 1775, +# dropping the o+w that lets the unprivileged build user unpack at all. +# * cmd_clean_sources reset unknown owners to root:root. +# +# The host lfs account is created by book 4.3's own useradd, which pins no +# uid -- it was 10753 on a Debian host. Package users start at 10000, so that +# is not merely "an unnamed UID" as the book puts it: it COLLIDES with a real +# package user, and tarballs report as owned by some unrelated package. +# +# Book 3.1 settles it and the reasoning is given there in full: +# chmod -v a+wt $LFS/sources -> root:root 1777 +# chown root:root $LFS/sources/* -> the contents too +# World-writable is what grants the build user access, so no group is involved. +python3 - "$LFS_TOOL" <<'PY65C' +import sys, os, stat, tempfile, importlib.machinery as m lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -src = open(sys.argv[1]).read() problems = [] -# md5 verification: `md5sum -c … | grep -v ": OK"` returned 0 on a CORRUPTED -# tree -- grep inverts the verdict and the pipe discards md5sum's status. -i = src.index("verifying md5sums") -seg = src[i:i + 900] -# ignore comment lines: the fix is documented with the broken command quoted -code = "\n".join(l for l in seg.splitlines() if not l.lstrip().startswith("#")) -if 'grep -v ": OK"' in code: - problems.append("md5 verification still inverts its own verdict") -if "md5sum -c md5sums --quiet" not in seg or "sys.exit(1)" not in seg: - problems.append("a bad md5sum does not fail the download") - -# The book's toolchain gate must stay a gate. These five greps MUST produce -# output -- no match means the build is linked against the host -- and the -# blanket `|| true` turned the only real check into a guaranteed pass. -gates = [ - 'grep -E -o "/usr/lib.*/S?crt[1in].*succeeded" dummy.log', - 'grep -B3 "^ /usr/include" dummy.log', - "grep 'SEARCH.*/usr/lib' dummy.log", - 'grep "/lib.*/libc.so.6 " dummy.log', - 'grep found dummy.log', -] -out = lfs._soften_diagnostics("\n".join(gates)) -for line in out.splitlines(): - if "|| true" in line: - problems.append("a toolchain gate is still softened: %s" % line[:40]) -# but an ordinary diagnostic grep must STILL be softened -if "|| true" not in lfs._soften_diagnostics("grep something optional.txt"): - problems.append("ordinary diagnostic greps are no longer softened") - -# 9.9's `cat > /etc/shells` block is <pre class="root">, not "userinput", so -# cfg_shells resolved to zero commands and /etc/shells was never written. -pi = src.index("classes = tag.get(\"class\") or []") -if '"root", "install"' not in src[pi:pi + 700]: - problems.append("only 'userinput' blocks are parsed; /etc/shells is missed") -# the accepted-class tuple itself must not include "screen" (output samples) -accept = src[src.index("any(c in classes for c in ("):][:120] -if "screen" in accept: - problems.append("'screen' blocks (output samples) would be run as commands") +if not hasattr(lfs, "_normalize_sources"): + print(" FAIL no single rule for /sources ownership") + sys.exit(1) -# a step that produces nothing must say so -if "has no command blocks in this book" not in src: - problems.append("a step resolving to zero blocks is still skipped silently") +# 4.3 hands over the book's directories and NOT /sources +if "sources" in lfs.LFS_HANDOVER_DIRS: + problems.append("/sources is in the 4.3 handover set") +body = open(sys.argv[1]).read() +h = body[body.index("def _chown_tree_to_lfs"):body.index("def _ensure_lfs_home")] +if 'LFS_HANDOVER_DIRS + ("sources",)' in h: + problems.append("the 4.3 chown still adds /sources to the handover") -# book 4.4's ~/.bashrc opens with these; without them the build runs with -# bash's command hash on and root's umask -env = lfs._lfs_env_prefix("/mnt/lfs") -if not env.startswith("set +h\numask 022\n"): - problems.append("the build environment is missing set +h / umask 022") +d = tempfile.mkdtemp() +src = os.path.join(d, "sources"); os.makedirs(src) +tar = os.path.join(src, "bash-5.3.tar.gz"); open(tar, "w").close() +tree = os.path.join(src, "acl-2.3.2"); os.makedirs(tree) +os.chmod(src, 0o775) # what verify --fix used to leave -# ^ anchored to the whole string, so a later line never matched -if not (lfs._TEST_DIAG_RE.flags & 8): - problems.append("_TEST_DIAG_RE has no re.M; ^ only matches the first line") +try: + os.chown(tar, 10753, 10753) # a host uid inside the package range + os.chown(tree, 10753, 10753) +except OSError: + print(" SKIP cannot chown here (needs root)") +else: + lfs._normalize_sources(d, run=True, quiet=True) + st = os.stat(src) + if stat.S_IMODE(st.st_mode) != (stat.S_ISVTX | 0o777): + problems.append("/sources is not 1777 -- the build user cannot unpack " + "(mode %o)" % stat.S_IMODE(st.st_mode)) + if st.st_uid != 0 or st.st_gid != 0: + problems.append("/sources itself is not root:root") + if os.lstat(tar).st_uid != 0: + problems.append("a downloaded file keeps a host uid (book 3.1)") + # an unpacked tree belongs to whoever is building it; taking it away + # mid-build would stop that build. Book 3.1 runs before any unpack. + if os.lstat(tree).st_uid != 10753: + problems.append("an unpacked tree was chowned out from under its build") + # running it twice must change nothing + if lfs._normalize_sources(d, run=True, quiet=True) != (0, False): + problems.append("the rule is not idempotent") -# dead duplicate flagged by pyflakes -if src.count("\ndef _pkg_glob_for") != 1: - problems.append("_pkg_glob_for is defined more than once") +# and it must actually be applied where sources appear or survive +g = body[body.index("def cmd_bs_get_sources"):body.index("def _title_pkgver")] +if "_normalize_sources" not in g: + problems.append("get-sources does not apply book 3.1's chown") +r = body[body.index("def cmd_bs_restart"):body.index("def cmd_bs_set_book")] +if "_normalize_sources" not in r: + problems.append("restart does not apply book 3.1's chown") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS the checks that always passed now actually check") -PY73 +print(" PASS /sources is root:root 1777 with root-owned contents (book 3.1)") +PY65C _count_rc $? -# ---- a staged install must not take the shared install directories ---------- # -# The staging tree is created by the package user, so $stage/usr, $stage/usr/bin -# and so on are owned by that user with mode 755. Stamping that onto the live -# tree handed /usr and /usr/bin to whichever package was building, losing the -# install group and g+w -- the whole install-directory scheme, undone by one -# package. Verified: root:install 775 became gcc:gcc 755. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" - case "$it" in - *"ALREADY THERE -- leave its ownership and mode alone"*) - ok "an existing directory keeps its owner and mode" ;; - *) bad "a staged install still restamps existing directories" ;; - esac - - # behaviour: existing dirs untouched, new ones owned by the package - sd="$T/stageown"; mkdir -p "$sd/usr/bin" "$sd/stage/usr/bin" "$sd/stage/usr/lib/fresh" - chmod 775 "$sd/usr" "$sd/usr/bin" - before_usr="$(stat -c %a "$sd/usr")" - echo p > "$sd/stage/usr/bin/p"; echo l > "$sd/stage/usr/lib/fresh/l" - chmod -R 755 "$sd/stage" - ( SNAP_ROOT="$sd"; say() { :; }; warn() { :; } - eval "$it"; _install_staged_tree "$sd/stage" ) >/dev/null 2>&1 - if [ "$(stat -c %a "$sd/usr")" = "$before_usr" ] \ - && [ -f "$sd/usr/bin/p" ] && [ -f "$sd/usr/lib/fresh/l" ]; then - ok "existing modes survive and the files still install" +# ---- /sources is scratch, not a shared install directory -------------------- # +# Being in install_dirs_list meant verify --fix set it root:install 775 and +# seal-install-dirs set 1775 -- both drop the o+w of book 3.1's `a+wt`, and +# o+w is the only thing that lets the unprivileged build user unpack there. +# It belongs in never_adopt_list (no package may own it) and nowhere else. +if [ -r "$(dirname "$LFS_TOOL")/lfs-helper" ]; then + HELPER="$(dirname "$LFS_TOOL")/lfs-helper" +else + HELPER="" +fi +if [ -n "$HELPER" ]; then + _p="" + # the DEFAULT list, i.e. the heredoc, must not carry it + if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then + _p="$_p;/sources is still a shared install directory" + fi + # but it must still be a directory no package may adopt + if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/sources'; then + _p="$_p;/sources may now be adopted by a package" + fi + # an installdirs.lst from an older run must not bring it back + if ! sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -q 'grep -vx.*/sources'; then + _p="$_p;a stale installdirs.lst can still reintroduce /sources" + fi + # the build scratch is the same kind of thing and needs the same rules + if ! sed -n '/^never_adopt_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then + _p="$_p;/build may be adopted by a package" + fi + if sed -n '/^install_dirs_list()/,/^}/p' "$HELPER" | grep -qx '/build'; then + _p="$_p;/build is a shared install directory" + fi + # every tree scan must skip it -- otherwise each snapshot, ownership pass + # and collector-group map walks every unpacked source tree + if [ "$(grep -c 'not -path "\${\?r}\?/build/\*"' "$HELPER")" -lt 3 ]; then + _p="$_p;a tree scan still descends into the build scratch" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done else - bad "the staged merge changed an existing directory's mode" + ok "/sources and /build are scratch, not shared install directories" fi +else + bad "lfs-helper not found beside $LFS_TOOL" fi -# ---- book 7.6 truncates the user database ----------------------------------- # -# `cat > /etc/passwd` is right the first time and catastrophic the second: it -# wipes every package user, private group, the install group, every collector -# group and every membership. The files on disk then all read as UNKNOWN. -if [ -f "$helper_src" ]; then - grep -q "^_protect_user_db()" "$helper_src" \ - && ok "the user database is protected across 7.6" \ - || bad "re-running init-files would wipe every package user" - grep -q "^_step_rewrites_user_db()" "$helper_src" \ - && ok "steps that rewrite passwd/group are detected" \ - || bad "the protection is not tied to the steps that need it" - # the detector must not fire on scripts that merely READ the files - dt="$(sed -n '/^_step_rewrites_user_db() {/,/^}/p' "$helper_src")" - rd="$T/dbdetect"; mkdir -p "$rd" - printf 'grep root /etc/passwd\n' > "$rd/read.sh" - printf 'cat > /etc/passwd << "EOF"\nroot:x:0:0::/root:/bin/bash\nEOF\n' > "$rd/write.sh" - ( eval "$dt" - _step_rewrites_user_db "$rd/write.sh" && ! _step_rewrites_user_db "$rd/read.sh" ) \ - && ok "it fires on a rewrite, not on a read" \ - || bad "the user-database detector is wrong" - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - case "$cb" in - *_protect_user_db*_restore_user_db*) - ok "entries are saved before the step and put back after" ;; - *) bad "the protection is never applied around the build" ;; +# ---- snapshots must not read live kernel filesystems ------------------------ # +# Excluding "./sys/*" skips the CONTENTS but still archives the ./sys directory +# itself. Reading a live kernel filesystem's metadata gives +# tar: ./sys: file changed as we read it +# and exit 1, so the snapshot failed -- then succeeded on a retry, because the +# timing happened to work out. --one-file-system does not descend into +# anything mounted inside the tree at all. +python3 - "$LFS_TOOL" <<'PY66' +import sys +src = open(sys.argv[1]).read() +body = src[src.index('cmd = ("tar -C'):] +body = body[:body.index("prog = ")] if "prog = " in body else body[:2000] +problems = [] +if "--one-file-system" not in body: + problems.append("tar still descends into mounted filesystems") +# tar's exit 1 is "files differ", a warning on a live tree -- verify, don't die +if "rc == 1" not in body: + problems.append("a tar warning is still treated as a failure") +if "tar -tf" not in body: + problems.append("the archive is not verified before being accepted") +# but an unreadable archive must NOT be kept +if "os.remove(dest)" not in body: + problems.append("a corrupt archive would be left in place") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS snapshots skip live filesystems and verify what they wrote") +PY66 +_count_rc $? + +# ---- an application user must be usable, not just exist --------------------- # +# `user create` made the account and stopped there: you could not become it, +# its home was world-readable, and nothing launched the program. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _setup_application_user" "$pmt" \ + && ok "creating an application user sets up access to it" \ + || bad "user create leaves an account nobody can use" + su_body="$(sed -n '/^def _setup_application_user/,/^def _home_of/p' "$pmt")" + case "$su_body" in + *'"chmod", "0750", home'*) ok "the account's home is private" ;; + *) bad "the application's home stays world-readable" ;; + esac + case "$su_body" in + *'"usermod", "-a", "-G", u, main'*) + ok "your own login joins the account's group" ;; + *) bad "a shared directory would not be reachable" ;; + esac + # without a main user there is nobody to grant access TO + case "$su_body" in + *"No main user is set"*) ok "it says so when no main user is set" ;; + *) bad "it silently does nothing when main_user is unset" ;; + esac + # the PAM file is not part of stock LFS: absence must not be an error + pam="$(sed -n '/^def _allow_su_via_pam/,/^def _write_launcher/p' "$pmt")" + case "$pam" in + *"not present -- \`su - "*) ok "a missing PAM file is handled, not fatal" ;; + *) bad "it assumes the hint's PAM file exists" ;; esac + grep -q "_APPLICATION_USER_HELP" "$pmt" \ + && ok "application users have their own help section" \ + || bad "no help explaining what an application user is" fi -# ---- two accounts sharing one uid ------------------------------------------- # -# Ownership is stored as a NUMBER. Give two names the same uid and every tool -# resolves it to whichever name comes first -- ncurses' binaries reported as -# /usr/bin/tic -rwxr-xr-x xz:xz -# and the ncurses build could not overwrite its own files. It reads exactly -# like a permissions bug and is not one. +# ---- shell completion -------------------------------------------------------- # +comp="$(dirname "$LFS_TOOL")/lfs-completion.bash" +if [ -f "$comp" ]; then + bash -n "$comp" \ + && ok "the completion script parses" \ + || bad "the completion script has a syntax error" + # subcommands must come from the tools themselves, or the list goes stale + grep -q "_lfs_tool_subcommands" "$comp" \ + && ok "completions are read from each tool's own help" \ + || bad "the completion keeps a second list of commands" + for t in lfs blfs packagemanager lfs-helper; do + grep -q "complete -F _lfs_complete $t\b" "$comp" \ + || bad "no completion registered for $t" + done + ok "all four tools have completion registered" +fi + +# ---- a system you can actually log into -------------------------------------- # +# A freshly built system has root with NO password. Depending on the login +# manager that is either "anyone can log in as root" or "nobody can log in at +# all" -- discovered after a reboot, with no way in. +# +# It lives in lfs-helper now, as the built-in init-accounts step. It could not +# move to `packagemanager setup` with the rest of the old final step: setup runs +# on the BOOTED system, and you need to log in to run it. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q "^_check_duplicate_ids()" "$helper_src" \ - && ok "duplicate ids are detected and named" \ - || bad "a uid collision would look like a permissions bug forever" - ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" - case "$ba" in - *_check_duplicate_ids*) ok "the check runs before anything is built" ;; - *) bad "the collision check never runs" ;; - esac - # restoring entries after book 7.6 must not CREATE a collision - rd="$(sed -n '/^_restore_user_db() {/,/^}/p' "$helper_src")" - case "$rd" in - *'is now used by'*) - ok "an entry whose id was reassigned is not restored" ;; - *) bad "restoring after 7.6 could create the collision itself" ;; - esac + _ia="$(_slice_fn "$helper_src" cmd_init_accounts)" + _p="" + case "$_ia" in + *'passwd root'*) ;; + *) _p="$_p;the root password is never set" ;; + esac + case "$_ia" in + *'NO password yet'*) ;; + *) _p="$_p;nothing explains why a root password matters" ;; + esac + # a scripted run has no terminal: it must say what is missing, not hang + case "$_ia" in + *'! -t 0'*) ;; + *) _p="$_p;a non-interactive run would block on a prompt" ;; + esac + case "$_ia" in + *'useradd -m'*) ;; + *) _p="$_p;no login account is ever created" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build ends with a system you can log into" + fi +fi - # behaviour: the detector finds a real collision and ignores a clean file - dd="$T/dupids"; mkdir -p "$dd" - printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\nncurses:x:10005:10005::/n:/bin/bash\n' > "$dd/bad" - printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\n' > "$dd/good" - du="$(sed -n '/^_duplicate_ids() {/,/^}/p' "$helper_src")" - ( eval "$du" - [ -n "$(_duplicate_ids "$dd/bad")" ] && [ -z "$(_duplicate_ids "$dd/good")" ] ) \ - && ok "it finds a real collision and stays quiet on a clean file" \ - || bad "the duplicate-id detector is wrong" + +# password detection must read /etc/shadow correctly +shd="$T/shadowcheck"; mkdir -p "$shd" +_haspw() { + grep -qE "^root:[^:]*:" "$1" && ! grep -qE "^root:(\*|!|)?:" "$1" +} +printf 'root::19000:0:99999:7:::\n' > "$shd/empty" +printf 'root:!:19000:0:99999:7:::\n' > "$shd/locked" +printf 'root:$6$a$b:19000:0:99999:7:::\n' > "$shd/set" +if ! _haspw "$shd/empty" && ! _haspw "$shd/locked" && _haspw "$shd/set"; then + ok "an empty, locked or real root password are told apart" +else + bad "root password detection is wrong" fi -# ---- one reconcile pass instead of eight repair commands -------------------- # -# fix-ownership, adopt-dirs, adopt-existing, fix-orphans, fix-users, -# sort-users, seal-install-dirs and claim-earlier-stages each fixed ONE symptom -# and could not see a problem outside its own remit -- so a staged install -# quietly handing /usr to a package user went unnoticed through four later -# failures. Ownership is derived state: the manifests, the install-dir list -# and the collector map fully determine what the tree should be. +# ---- decisions already made must not be asked again ------------------------- # +# The export copied into the tree said +# dir|/usr/libexec/gcc|nimgnu_gcc +# yet the build still stopped to ask which group should share that directory. +# Two faults: nothing applied the file, and `import-groups --run` read the +# wrong path anyway. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q "^cmd_verify()" "$helper_src" \ - && ok "one command reconciles the tree" \ - || bad "there is no single check of the whole model" - # The commands it replaces must not be SECOND IMPLEMENTATIONS -- but they - # must still be callable. Deleting them outright broke every message that - # tells people to run them: - # !! unknown command 'adopt-existing' - for c in fix-ownership fix-orphans fix-users adopt-existing; do - grep -qE "^\s+.*\b$c[|)]" "$helper_src" \ - || bad "$c is referenced in the hints but not dispatched" - done - ok "the familiar repair names still work" - # ... and route to verify rather than repeating its logic - disp="$(sed -n '/fix-ownership|fix-orphans/,/cmd_verify/p' "$helper_src")" - case "$disp" in - *cmd_verify*) ok "they are aliases for verify, not second copies" ;; - *) bad "a repair command has its own implementation again" ;; + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *"cmd_import_groups --run"*) + ok "a copied-in groups export is applied automatically" ;; + *) bad "the export sits unapplied until run by hand" ;; esac + grep -q "^_groups_already_imported()" "$helper_src" \ + && ok "it is applied once, not on every build" \ + || bad "the import would run on every build-all" - # every rule the model needs - for r in _vfy_manifest_ownership _vfy_install_dirs _vfy_orphans \ - _vfy_duplicate_ids _vfy_user_order; do - grep -q "^$r()" "$helper_src" || bad "verify has no $r rule" - done - ok "verify checks ownership, install dirs, orphans, ids and order" + # `--run` used to be taken as the FILENAME, then overwrite the path + ig="$(sed -n '/^cmd_import_groups() {/,/^}/p' "$helper_src")" + case "$ig" in + *'in="$STATE/collector-groups.export"; }'*) + bad "import-groups --run still overwrites the file it found" ;; + *) ok "import-groups --run reads the file it chose" ;; + esac + case "$ig" in + *'local in="" run=0 a'*) ok "the arguments are parsed in one place" ;; + *) bad "the run flag can still be wiped by a later declaration" ;; + esac - # reporting by default, repairing only when asked - cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" - case "$cv" in - *'"${1:-}" = "--fix" ] && fix=1'*|*"--fix|--run) fix=1"*) - ok "verify reports unless asked to repair" ;; - *) bad "verify repairs without being asked" ;; + # and the prompt itself must consult the export, imported or not + cg="$(sed -n '/^choose_collector_group() {/,/^}/p' "$helper_src")" + case "$cg" in + *'collector-groups.import'*) + ok "the prompt checks the copied-in decisions first" ;; + *) bad "a decision in the tree would still be asked about" ;; esac - # manifest ownership must settle before orphans, or a file whose uid no - # longer resolves is handed to 'lfs' instead of to its package - i_own="$(printf '%s' "$cv" | grep -n '_vfy_manifest_ownership' | head -1 | cut -d: -f1)" - i_orp="$(printf '%s' "$cv" | grep -n '_vfy_orphans' | head -1 | cut -d: -f1)" - if [ -n "$i_own" ] && [ -n "$i_orp" ] && [ "$i_own" -lt "$i_orp" ]; then - ok "manifest ownership is settled before orphans are claimed" - else - bad "an orphan pass could take a file that belongs to a package" - fi +fi - # A1 end to end: the install dirs are checked, so a staged install taking - # /usr is caught rather than surfacing four failures later - vd="$T/verifydirs"; mkdir -p "$vd/usr/bin" "$vd/.lfs-pkgusr/manifests" - chmod 755 "$vd/usr" "$vd/usr/bin" - id_out="$( ( SNAP_ROOT="$vd"; ETC="$vd/etc"; INSTALLDIRS="" - MANIFESTS="$vd/.lfs-pkgusr/manifests" - say() { echo "$*"; }; warn() { echo "$*"; } - _vfy_n_checked=0; _vfy_n_wrong=0; _vfy_n_fixed=0 - INSTALL_GID=9999 - for _f in install_dirs_list _install_dirs_are_sealed _vfy_report \ - set_install_dir_owner install_dir_owner_ok _vfy_install_dirs; do - eval "$(_slice_fn "$helper_src" "$_f")" - done - # the pass is gated on the ownership epoch now: before book 7.6 there - # is no user database, so `install` is not a name and reporting 27 - # wrong directories would be noise. This fixture is past that. - ownership_established() { return 0; } - ownership_possible() { return 0; } - _vfy_install_dirs 0 ) 2>&1 )" - case "$id_out" in - *"root:install"*) ok "a wrongly-owned install directory is reported" ;; - *) bad "verify does not notice install dirs losing the install group" ;; - esac +# end to end: the answer is in the file, so nothing is asked +igd="$T/importdec"; mkdir -p "$igd/.lfs-pkgusr" "$igd/etc" "$igd/usr/src" +printf 'root:x:0:\ninstall:x:9999:\n' > "$igd/etc/group" +printf 'root:x:0:0:root:/root:/bin/bash\n' > "$igd/etc/passwd" +printf '# prefix|nimgnu\ndir|/usr/libexec/gcc|nimgnu_gcc\n' \ + > "$igd/.lfs-pkgusr/collector-groups.import" +got="$(STATE="$igd/.lfs-pkgusr" awk -F'|' -v d=/usr/libexec/gcc \ + '$1 == "dir" && $2 == d { print $3; exit }' \ + "$igd/.lfs-pkgusr/collector-groups.import")" +[ "$got" = "nimgnu_gcc" ] \ + && ok "a recorded decision resolves without asking" \ + || bad "the recorded decision was not found (got '$got')" + +# ---- 7.2 must actually hand the tree to root -------------------------------- # +# The handover used `chown --from lfs`, which changes only files owned by +# exactly that user. A tree owned by a uid with no matching account -- or by +# an `lfs` whose uid changed since -- was silently left alone, so `prepare` +# reported "give the tree to root", changed nothing, and `enter` then refused. +# The loop in `run` caught it as a step that ran but stayed pending. +python3 - "$LFS_TOOL" <<'PY68' +import sys +src = open(sys.argv[1]).read() +i = src.index('"give the tree to root"') +cmd = src[i:i + 400] +if "--from lfs" in cmd: + print(" FAIL the handover still only matches files owned by 'lfs'") + sys.exit(1) +if "chown -h -R root:root" not in cmd: + print(" FAIL the handover does not chown recursively"); sys.exit(1) +# and the refusal should say who DOES own it +if "_owner_name_of" not in src: + print(" FAIL the refusal does not name the actual owner"); sys.exit(1) +print(" PASS 7.2 hands the tree to root whoever owned it") +PY68 +_count_rc $? + +# behaviour: a tree owned by an unknown uid must still be handed over +hod="$T/handover"; mkdir -p "$hod/usr/bin" "$hod/etc" +chown -R 4242 "$hod/usr" "$hod/etc" 2>/dev/null || true +if [ "$(stat -c %u "$hod/usr")" = "4242" ]; then + ( for d in usr etc; do [ -e "$hod/$d" ] && chown -h -R root:root "$hod/$d"; done; true ) + [ "$(stat -c %u "$hod/usr")" = "0" ] \ + && ok "a tree owned by an unknown uid is handed to root" \ + || bad "the handover left the tree owned by $(stat -c %u "$hod/usr")" +else + ok "(cannot test unknown-uid handover as this user)" fi -# ---- ownership is derived state, so it can be checked -------------------------# -# The manifests say who installed what, the install-dir list says what is -# shared, the collector map says which groups govern which directories. -# Together they determine what the tree SHOULD look like -- so a discrepancy -# can be found before it becomes a failed build, instead of being chased one -# symptom at a time through fix-ownership, adopt-dirs, fix-perms, fix-orphans -# and claim-earlier-stages. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^cmd_verify()" "$helper_src" \ - && ok "the tree can be checked against the manifests" \ - || bad "there is no way to ask whether ownership is correct" - cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" - # the A1 case: an install directory taken by a package user - # The wording lives in the _vfy_* helpers now -- cmd_verify calls them - # rather than carrying its own copy of each rule. Assert the check runs. - case "$cv" in - *_vfy_install_dirs*) - ok "it catches an install directory taken by a package" ;; - *) bad "the install-directory scheme is not checked" ;; +# ---- a shared user reaches the session -------------------------------------- # +# A "shared user" is one that belongs to the group on the main user's XDG +# runtime directory: +# drwxrws--- n76310 u_xdg_runtime /tmp/xdg-n76310 +# The display socket, the session bus and audio live there, so without that +# group a graphical program starts and then cannot draw anything. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _join_xdg_runtime_group" "$pmt" \ + && ok "a shared user joins the session group" \ + || bad "a shared user cannot reach the display or audio" + jx="$(sed -n '/^def _join_xdg_runtime_group/,/^def _setup_application_user/p' "$pmt")" + # the group must be READ from the directory, never assumed by name + case "$jx" in + *"getgrgid(os.stat(rt).st_gid)"*) + ok "the group is read from the runtime directory" ;; + *) bad "the session group name is hardcoded" ;; esac - _vfy_install_dirs_body="$(_slice_fn "$helper_src" _vfy_install_dirs)" - case "$_vfy_install_dirs_body" in - *'"root:install $want_mode"'*) ;; - *) bad "the install-dir check no longer says what it wants" ;; + case "$jx" in + *'u_xdg_runtime"'*) bad "a specific group name is hardcoded" ;; + *) ok "no group name is baked in" ;; esac - case "$cv" in - *_vfy_manifest_ownership*) - ok "it catches files owned by the wrong package" ;; - *) bad "mis-owned files are not detected" ;; + # a personal group is not a shared group + case "$jx" in + *"which is not a shared "*) + ok "it refuses to add the user to a personal group" ;; + *) bad "an application user could join the person's own group" ;; esac - case "$cv" in - *_check_duplicate_ids*) ok "it checks for id collisions too" ;; - *) bad "a uid collision would not be reported" ;; + # both /run/user/<uid> and /tmp/xdg-<user> layouts + xd="$(sed -n '/^def _xdg_runtime_dir_of/,/^def _join_xdg/p' "$pmt")" + for p in "/run/user" "/tmp/xdg-"; do + case "$xd" in + *"$p"*) ;; + *) bad "the runtime directory search misses $p" ;; + esac + done + ok "both runtime-directory layouts are searched" + # --shared and --share-dir are different things + grep -q '"--share-dir"' "$pmt" \ + && ok "the session group and a shared directory are separate options" \ + || bad "--shared conflates the session group with a directory" +fi + +# ---- the setup must be applicable after the fact ---------------------------- # +# An account made earlier -- or by hand -- should be able to reach the same +# state without deleting and recreating it. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def cmd_user_setup" "$pmt" \ + && ok "an existing user can be set up after the fact" \ + || bad "the setup only happens at creation time" + grep -q '"setup"' "$pmt" \ + && ok "`user setup` is registered" \ + || bad "the setup command is not reachable" + # it must refuse on a user that does not exist, rather than half-creating + cs="$(sed -n '/^def cmd_user_setup/,/^def _write_desktop_entry/p' "$pmt")" + case "$cs" in + *"no such user"*) ok "setup refuses on a user that does not exist" ;; + *) bad "setup would act on a missing user" ;; esac - # report by default, repair only when asked - case "$cv" in - *'[ "${1:-}" = "--fix" ] && fix=1'*) - ok "it reports by default and repairs only with --fix" ;; - *) bad "verify changes things without being asked" ;; + # the same code path as create, or the two drift + case "$cs" in + *_setup_application_user*) ok "create and setup share one code path" ;; + *) bad "setup reimplements what create does" ;; esac - # a collision is NOT auto-repairable -- which name owns what is a decision - case "$cv" in - *"verify cannot repair this"*) - ok "it does not guess at an id collision" ;; - *) bad "verify would silently reassign ids" ;; + + # menu entry + grep -q "^def _write_desktop_entry" "$pmt" \ + && ok "a menu entry is written" \ + || bad "no desktop entry is created" + de="$(sed -n '/^def _write_desktop_entry/,/^def _write_launcher/p' "$pmt")" + for field in "Desktop Entry" "Exec=" "StartupWMClass=" "Icon="; do + case "$de" in + *"$field"*) ;; + *) bad "the desktop entry has no $field" ;; + esac + done + ok "the desktop entry has the fields a menu needs" + # a launcher for a program that is not installed is a broken command + sa="$(sed -n '/# 6. a launcher and a menu entry/,/_write_desktop_entry(main, u, app)/p' "$pmt")" + case "$sa" in + *"no launcher or menu entry"*) + ok "nothing is written when the program is not installed" ;; + *) bad "a launcher could point at a program that does not exist" ;; + esac + # and neither file is clobbered if it is already there + case "$de" in + *"already exists -- left alone"*) ok "an existing menu entry is kept" ;; + *) bad "the desktop entry would be overwritten" ;; esac fi -# ---- durable records and scratch must be distinguishable ---------------------# -# $STATE mixes records that ARE the ownership (manifests, collector map) with -# pure scratch (staging trees, logs). A snapshot or a restart had to guess. +# ---- changing LFS_TGT invalidates the toolchain ----------------------------- # +# gcc-pass2 installs its C++ headers under $LFS_TGT. Change LFS_TGT afterwards +# and the compiler looks under the new name while the headers sit under the +# old one -- present, readable, and invisible. It surfaces forty packages +# later as "bits/c++config.h: No such file or directory", which reads like a +# missing file rather than a mismatch. +python3 - "$LFS_TOOL" <<'PY69' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] +if not hasattr(lfs, "_crosschain_tgt_mismatch"): + problems.append("nothing detects a changed target triplet") +# the triplet must be RECORDED with the build, or there is nothing to compare +mk = src[src.index("def _mark_crosschain_done"):] +mk = mk[:mk.index("def lfs_tgt")] +if '"lfs_tgt"' not in mk: + problems.append("the target is not recorded with the build progress") +# and checked before the chroot, not after +if "this tree was built for a different target" not in src: + problems.append("a changed target is not reported") +i_check = src.find("this tree was built for a different target") +i_prep = src.find('if action == "prepare":') +if i_check < i_prep: + problems.append("the check is not inside chroot prepare") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a changed target triplet is caught before the chroot") +PY69 +_count_rc $? + +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q "^STATE_DURABLE=" "$helper_src" \ - && ok "the durable records are named" \ - || bad "nothing says which state must survive" - grep -q "^STATE_SCRATCH=" "$helper_src" \ - && ok "the scratch is named" \ - || bad "nothing says which state is disposable" - grep -q "manifests" <<< "$(sed -n '/^STATE_DURABLE=/,/^STATE_SCRATCH=/p' "$helper_src")" \ - && ok "the manifests count as durable" \ - || bad "the manifests could be cleaned away" - grep -q "^STATE_FORMAT=" "$helper_src" \ - && ok "the state format is versioned" \ - || bad "an incompatible state directory would be misread" - cs="$(sed -n '/^cmd_clean_state() {/,/^}/p' "$helper_src")" - case "$cs" in - *'rm -rf "${STATE:?}/$d"'*) - ok "cleaning scratch cannot expand to rm -rf /" ;; - *) bad "an unset STATE would delete the root" ;; + grep -q "^_warn_on_triplet_mismatch()" "$helper_src" \ + && ok "the mismatch is also reported inside the chroot" \ + || bad "inside the chroot the mismatch is only a missing-file error" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *_warn_on_triplet_mismatch*) + ok "it is checked before building, not after failing" ;; + *) bad "the mismatch check never runs" ;; esac fi -# ---- the suite's own count must be trustworthy ------------------------------ # -# ok/bad used shell variables, and several checks call them from inside a -# subshell -- `( ... ) && ok || bad`, a pipeline, a `while read` fed by process -# substitution. An increment there dies with the subshell, so the suite -# printed 446 PASS lines and reported 433. A wrong count is worse than none: -# it hides a regression behind a number that still looks healthy. -cnt="$T/countcheck"; mkdir -p "$cnt" -( _COUNTDIR="$cnt"; : > "$cnt/pass"; : > "$cnt/fail" - ok() { echo x >> "$_COUNTDIR/pass"; } - # the case that used to be lost: incremented inside a subshell - ( ok ) ; ( true && ok ) ; echo | while read -r _; do ok; done ) -n="$(wc -l < "$cnt/pass" | tr -d ' ')" -[ "$n" = 3 ] \ - && ok "counts survive subshells, pipelines and process substitution" \ - || bad "the counter loses $((3 - n)) of 3 increments from subshells" - -# and every assertion must be counted exactly once -_asserts="$(grep -cE '^\s*(ok|bad) "' "$0" 2>/dev/null || echo 0)" -[ "$_asserts" -gt 0 ] \ - && ok "the suite counts assertions, not printed lines" \ - || bad "the suite has no assertions to count" - -# ---- host layout and built-system layout are separate ------------------------ # -# Configuring where package users live on THIS machine must not change where -# they live on every system built from it. The whole point of the build is -# that the new system is not this one. -python3 - "$LFS_TOOL" <<'PY74' -import sys, os, json, tempfile, importlib.machinery as m -store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store -os.makedirs(os.path.join(store, "books"), exist_ok=True) -json.dump({"default": "12.4", - "pkgusr_home": "/usr/src", - "target_pkgusr_home": "/usr/src/package_users", - "target_appuser_home": "/home/shared_users"}, - open(os.path.join(store, "config.json"), "w")) -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -problems = [] -if not hasattr(lfs, "layout"): - print(" FAIL the layout is not configurable"); sys.exit(1) -if lfs.layout("pkgusr_home") != "/usr/src": - problems.append("the host layout was changed by a target setting") -if lfs.layout("pkgusr_home", target=True) != "/usr/src/package_users": - problems.append("the built system does not use its own layout") -if lfs.layout("appuser_home", target=True) != "/home/shared_users": - problems.append("application users do not get their own base") -# The default moved deliberately: accounts are grouped by kind, so a -# directory listing says what a thing IS. Existing trees are NOT migrated -- -# a fresh build is required, and migrate-layout is still unwritten. Pin the -# new values so the next move is also a decision and not an accident. -d = lfs.LAYOUT_DEFAULTS -if d["target_pkgusr_home"] != "/usr/src/pkgusr" \ - or d["pkgusr_home"] != "/usr/src/pkgusr": - problems.append("packages no longer default to /usr/src/pkgusr") -if d["target_cfguser_home"] != "/usr/src/cfg" \ - or d["cfguser_home"] != "/usr/src/cfg": - problems.append("config steps no longer default to /usr/src/cfg") -# and the target value must reach the chroot and the built system's config -src = open(sys.argv[1]).read() -if 'LFS_SRC_ROOT="{layout("pkgusr_home", target=True)}"' not in src: - problems.append("the chroot build environment does not carry the layout") -if 'pkgusr_home=%s' not in src: - problems.append("the built system's config does not record the layout") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS host and built-system layouts are configured independently") -PY74 -_count_rc $? - -pmt="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pmt" ]; then - grep -q "^def appuser_home" "$pmt" \ - && ok "application users have their own base directory" \ - || bad "application users still share the package-user base" - # an unconfigured system must keep the old layout - lyd="$T/layout"; mkdir -p "$lyd" - printf 'collector_prefix=n\n' > "$lyd/pm.conf" - got="$(PKGUSR_CONFIG="$lyd/pm.conf" python3 -c " -import importlib.machinery as m, os -pm = m.SourceFileLoader('pm', '$pmt').load_module() -print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" - [ "$got" = "/usr/src /usr/src" ] \ - && ok "an unconfigured system keeps the existing layout" \ - || bad "the layout changed without being asked (got '$got')" - printf 'collector_prefix=n\npkgusr_home=/usr/src/package_users\nappuser_home=/home/shared_users\n' \ - > "$lyd/pm2.conf" - got2="$(PKGUSR_CONFIG="$lyd/pm2.conf" python3 -c " -import importlib.machinery as m, os -pm = m.SourceFileLoader('pm', '$pmt').load_module() -print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" - [ "$got2" = "/usr/src/package_users /home/shared_users" ] \ - && ok "a configured system uses the layout it was built with" \ - || bad "the configured layout is ignored (got '$got2')" -fi - -# ---- /root must never carry the install group -------------------------------- # -# Two different ideas got conflated. An install directory is SHARED: it gets -# the install group and g+w so any package user can add files. /root, /home -# and the scratch dirs are something else -- they must never be adopted by a -# package, but they are not shared. Listing them as install dirs produced -# drwxrwx--- 1 root install /root -# handing every package user access to root's home. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^never_adopt_list()" "$helper_src" \ - && ok "protected-from-adoption is separate from shared" \ - || bad "the two ideas are still conflated" - ( SNAP_ROOT=/; INSTALLDIRS="" - eval "$(sed -n '/^never_adopt_list() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^is_never_adopt() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^install_dirs_list() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^install_dir_patterns() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^matches_install_pattern() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^is_install_dir() {/,/^}/p' "$helper_src")" - # never group-writable - for d in /root /home /tmp /var/tmp; do - install_dirs_list | grep -qx "$d" && exit 1 - done - # but still never adopted - for d in /root /home /tmp /var/tmp; do is_install_dir "$d" || exit 1; done - # and the real install dirs are untouched - install_dirs_list | grep -qx /usr/bin || exit 1 - exit 0 ) \ - && ok "/root and /home are protected but not group-writable" \ - || bad "/root or /home would get the install group" -fi - -# ---- nothing owned by the build user survives into the system --------------- # -# Book 7.2 hands over the directories 4.2 created, but the tree ROOT and the -# mount points are not in that list, so /, /dev, /proc, /sys, /run, /media, -# /opt and /tools stayed owned by a user that does not exist in the chroot: -# drwxr-xr-x 1 lfs lfs 178 / -python3 - "$LFS_TOOL" <<'PY75' -import sys -src = open(sys.argv[1]).read() -if "give the top level to root" not in src: - print(" FAIL the tree root and mount points keep the build user") - sys.exit(1) -i = src.index("give the top level to root") -seg = src[i:i + 500] -problems = [] -if "/sources" not in seg: - problems.append("/sources is not excluded; the book keeps it with lfs") -if "-R " in seg: - problems.append("the handover is recursive; it must only touch the top level") -if 'chown -h root:root' not in seg: - problems.append("symlinks at the top level are not handed over") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the tree root and mount points are handed back to root") -PY75 -_count_rc $? - -# ---- the package-user model must be usable from outside ---------------------- # -# `build` only works for steps that already exist in the step list with a -# generated script, so a BLFS package or a local script had no way to say -# "install this as its own package user and track it" -- it would have to -# reimplement user creation, the wrappers, staging, tracking and the permission -# grants. That is exactly how the two halves of this toolchain drifted apart. +# ---- three causes, one error message ---------------------------------------- # +# "bits/c++config.h: No such file or directory" is produced by three different +# situations needing three different fixes. The compiler's own configure line +# tells them apart: +# no --host and no --target -> a NATIVE gcc, which only chapter 8 builds: +# its install is half-finished +# both are cross names -> LFS_TGT changed since chapter 6 +# no headers at all -> chapter 6 never finished helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q "^cmd_install_as()" "$helper_src" \ - && ok "anything can be installed as its own package user" \ - || bad "third-party installs cannot use the model" - ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" - # both shapes a caller might have - case "$ia" in - *"--script)"*) ok "an existing install script can be used" ;; - *) bad "only inline commands are accepted" ;; - esac - case "$ia" in - *"--) shift; cmdv=("*) ok "a bare command can be used" ;; - *) bad "only a script file is accepted" ;; + grep -q "^_explain_triplet_mismatch()" "$helper_src" \ + && ok "the cause of a mismatch is named, not just the symptom" \ + || bad "all three causes give the same message" + ex="$(sed -n '/^_explain_triplet_mismatch() {/,/^}/p' "$helper_src")" + case "$ex" in + *"it is a NATIVE build"*) + ok "a native compiler is recognised as chapter 8's" ;; + *) bad "a half-installed chapter-8 gcc is not identified" ;; esac - # it must REUSE cmd_build, not reimplement it -- one code path or they drift - case "$ia" in - *'cmd_build "$name" --force'*) - ok "it reuses the build path rather than duplicating it" ;; - *) bad "install-as reimplements what build already does" ;; + case "$ex" in + *"LFS_TGT changed between"*) + ok "two cross names are recognised as a changed LFS_TGT" ;; + *) bad "a changed LFS_TGT is not identified" ;; esac - # the result must be a normal package, visible to every query - case "$ia" in - *'"$STATE_PROGRESS/steporder"'*) - ok "the package becomes visible to status, list and manifests" ;; - *) bad "an install-as package would be invisible to the other commands" ;; + # the two cases must give DIFFERENT fixes -- that is the whole point + # the two causes need different fixes -- that is the whole point + case "$ex" in + *"lfs config lfs_tgt"*"--phase all --force"*|*"--phase all --force"*"lfs config lfs_tgt"*) + ok "each cause gets the fix that applies to it" ;; + *) bad "the same fix is offered for different causes" ;; esac - # reserved names must be refused - case "$ia" in - *"_is_not_a_package"*) ok "reserved step names are refused" ;; - *) bad "install-as could create a user for last-step or cfg_*" ;; + # and it must say nothing when the names agree + case "$ex" in + *'[ "$have" = "$mine" ] && return 0'*) + ok "it is silent when there is no mismatch" ;; + *) bad "it would explain a mismatch that does not exist" ;; esac - # a command with spaces or quotes must survive being written to a script - grep -q "^_quote_cmd()" "$helper_src" \ - && ok "arguments are quoted, so paths with spaces survive" \ - || bad "a command containing spaces would be mangled" fi -# ---- a capitalised step must find its lowercase earlier stage --------------- # -# Book titles are capitalised ("Python", "Util-linux") but the chapter-7 stage -# is named from the tarball ("python-tmp"). Matching only the step's own -# spelling meant Python never claimed python-tmp's files, and chapter 7's -# root-owned tree blocked the chapter-8 install: -# install: cannot remove '/usr/lib/python3.13/idlelib/Icons/README.txt' +# ---- installs are staged, not written straight into the system -------------- # +# `make install` writes into the live system one file at a time, in Makefile +# order. Interrupt it and the system is neither the old version nor the new +# one -- and for gcc that is unrecoverable: it installs its driver early and +# its C++ headers late, so a cancel in between leaves a compiler that cannot +# compile C++, which is what you need to rebuild it. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src")" - case "$ce" in - *"tr 'A-Z' 'a-z'"*) ok "a capitalised step finds its lowercase stage" ;; - *) bad "Python would never claim python-tmp's files" ;; - esac - case "$ce" in - *'for stem in "$name" "$lname" "$owner" "$owname"'*) - ok "the owner's spelling is tried as well as the step's" ;; - *) bad "only the step name is tried" ;; + grep -q "^_merge_stage()" "$helper_src" \ + && ok "installs go to a staging tree first" \ + || bad "installs still write straight into the live system" + grep -q "DESTDIR='\$_stage'" "$helper_src" \ + && ok "DESTDIR points the install at the staging tree" \ + || bad "the staging tree is created but never used" + + # a FAILED install must leave the system untouched -- the whole point + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *'if [ "${rc:-0}" = 0 ]; then'*'_merge_stage'*) + ok "the staged tree is moved in only on success" ;; + *) bad "a failed install could still be merged" ;; esac - case "$ce" in - *"de-duplicate"*) ok "a manifest is not walked twice" ;; - *) bad "the same manifest could be walked four times" ;; + case "$cb" in + *"the system was not touched"*) + ok "a failed install says the system is unchanged" ;; + *) bad "a failed install does not say what state things are in" ;; esac -fi -# ---- a package user should record where it came from ------------------------ # -# An LFS base package, a BLFS package, a pip module and a local script are -# maintained differently and updated from different places. Nothing in a -# package user's home said which it was. -if [ -f "$helper_src" ]; then - grep -q "^_write_pkgusr_info()" "$helper_src" \ - && ok "each package user records its source" \ - || bad "there is no way to tell an LFS package from a BLFS one" - grep -q "^cmd_pkgusr_info()" "$helper_src" \ - && ok "it can be read back" \ - || bad "the record is written but never readable" - # both completion paths must write it, not just one - cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" - n="$(printf '%s' "$cb" | grep -c '_write_pkgusr_info')" - [ "${n:-0}" -ge 2 ] \ - && ok "every path that completes a build records it" \ - || bad "only one of the two done paths records the source" - # the helper must be defined before cmd_build uses it - i_def="$(grep -n '^_write_pkgusr_info()' "$helper_src" | head -1 | cut -d: -f1)" - i_use="$(grep -n '^cmd_build() {' "$helper_src" | head -1 | cut -d: -f1)" - [ -n "$i_def" ] && [ -n "$i_use" ] && [ "$i_def" -lt "$i_use" ] \ - && ok "it is defined before the build uses it" \ - || bad "the branding helper is defined after its first use" - ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" - case "$ia" in - *"LFS_PKG_SOURCE"*) ok "install-as records what kind of install it was" ;; - *) bad "a BLFS or local install would be recorded as lfs" ;; + # a package that ignores DESTDIR writes to the system anyway: say so + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + case "$ms" in + *"return 2"*) ok "an empty staging tree is detected" ;; + *) bad "a package ignoring DESTDIR would look like a clean install" ;; esac -fi - -# ---- adopt-existing must still exist ----------------------------------------- # -# The pass moved into verify, but the command was removed while messages -# elsewhere still told people to run it: -# !! unknown command 'adopt-existing' -# A command that has been deleted is worse than one that is redundant. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^cmd_adopt_existing()" "$helper_src" \ - && ok "adopt-existing is still a command" \ - || bad "a command the messages tell people to run does not exist" - grep -qE "^\s+adopt-existing\|adopt-dirs\)" "$helper_src" \ - && ok "it is reachable from the command line" \ - || bad "adopt-existing is defined but not dispatched" - # every command named in a hint must actually dispatch - for c in fix-ownership fix-orphans grant-dir verify which-package \ - check-toolchain seal-install-dirs sort-users install-as; do - # a command may be dispatched on its own or as part of an alternation - grep -qE "^\s+[a-z|-]*\b$c[|)]" "$helper_src" \ - || bad "$c is referenced but not dispatched" - done - ok "every command referenced in the hints is dispatched" - - # ---- and the users must be created in BUILD order --------------------- # - # This pass creates the package users it finds missing, and a uid is just - # the order of creation. Alphabetical order gave man-pages (first - # chapter-8 package) 10000 and binutils -- built first, in chapter 5 -- a - # much higher one, so the passwd file no longer reads as the build order. - vm="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" - case "$vm" in - *'$(_manifests_in_build_order)'*) - ok "the users are created in build order" ;; - *) bad "manifests are still walked alphabetically" ;; + case "$cb" in + *"ignores DESTDIR"*) ok "and reported, not silently trusted" ;; + *) bad "ignoring DESTDIR is not reported" ;; esac - mo="$(sed -n '/^_manifests_in_build_order() {/,/^}/p' "$helper_src")" - case "$mo" in - *'[ "${1:-}" = dirs ]'*) - ok "the directory manifests are ordered the same way" ;; - *) bad "the .dirs manifests are still alphabetical" ;; + # merging must target the tracked root, not a hardcoded / + # the installer, not the merge wrapper, is what writes + it2="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it2" in + *'root="${SNAP_ROOT:-/}"'*) ok "the merge targets the tracked root" ;; + *) bad "the merge writes to a hardcoded /" ;; esac -fi -# ---- package users cannot exist before book 7.6 ----------------------------- # -# A crosschain snapshot is taken BEFORE chapter 7, so /etc/passwd does not -# exist -- 7.6 creates it. Running the adoption pass then printed -# user binutils (could not create) -> package user -# forty times, once per package, with no hint why. The give-away is the -# prompt: "I have no name!" means even root has no passwd entry. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_user_db_ready()" "$helper_src" \ - && ok "the user database is checked before creating users" \ - || bad "adoption would fail once per package with no explanation" - ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" - case "$ae" in - *_explain_no_user_db*) - ok "it explains that the tree is simply too early" ;; - *) bad "the error does not say what is actually wrong" ;; - esac - # and the work must still happen, later, without being asked again - # The late pass is the ownership epoch now: build-all calls - # _ownership_checkpoint after every step, and the marker makes it fire once. - ba="$(_slice_fn "$helper_src" cmd_build_all)" + # the workarounds this replaces must be gone + grep -q "cmd_fix_toolchain" "$helper_src" \ + && bad "the symlink workaround is still present" \ + || ok "the triplet symlink workaround is gone" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" case "$ba" in - *_ownership_checkpoint*) - ok "the packages are adopted once chapter 7 has created it" ;; - *) bad "a chapter 5-6 tree would never adopt its packages" ;; - esac - case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in - *ownership_established*) ok "the late pass runs once, not per step" ;; - *) bad "the late adoption would repeat on every step" ;; - esac - ud="$(sed -n '/^_user_db_ready() {/,/^}/p' "$helper_src")" - case "$ud" in - *"grep -q '^root:'"*) - ok "an empty or rootless passwd counts as not ready" ;; - *) bad "a truncated passwd would look usable" ;; + *"Finishing its install before going on"*) + bad "the auto-finish workaround is still present" ;; + *) ok "the auto-finish workaround is gone" ;; esac fi -# ---- "did not change owner" vs "no package claims it" ----------------------- # -# The ownership pass can only act on what a manifest lists. A file installed -# before tracking existed, or whose manifest was lost, stays root-owned and -# never appears in the report at all -- so a skipped file and an unclaimed one -# look identical from the outside, and they need different fixes. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - grep -q "^_vfy_unclaimed()" "$helper_src" \ - && ok "files no manifest claims are reported" \ - || bad "an unclaimed file is indistinguishable from a skipped one" - vu="$(sed -n '/^_vfy_unclaimed() {/,/^}/p' "$helper_src")" - # /etc is root's by right; reporting it would bury the real cases - case "$vu" in - *'"${SNAP_ROOT%/}/etc"'*) ok "configuration under /etc is not reported" ;; - *) bad "every file in /etc would be listed as unclaimed" ;; +# ---- a shared user must know where the session is --------------------------- # +# Being in the session group is necessary but not sufficient: XDG_RUNTIME_DIR, +# WAYLAND_DISPLAY and the package-user PATH all come from +# /etc/pkgusr/skel-u_xdg/.bash_profile. Without it the account can reach the +# socket and still not find it. +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def _link_xdg_profile" "$pmt" \ + && ok "a shared user gets the XDG profile" \ + || bad "a shared user has no XDG_RUNTIME_DIR" + lp="$(sed -n '/^def _link_xdg_profile/,/^def _setup_application_user/p' "$pmt")" + # a symlink, so editing the file once reaches every shared account + case "$lp" in + *"os.symlink(XDG_SKEL, target)"*) ok "it is linked, not copied" ;; + *) bad "each account gets its own copy, which then drift" ;; esac - # the manifest paths carry the host prefix; comparison must strip it - case "$vu" in - *strip_host_prefix*) ok "manifest paths are compared without the prefix" ;; - *) bad "every claimed file would look unclaimed inside the chroot" ;; + # the runtime directory belongs to whoever is logged in: fill it in, do + # not ship someone else's path + case "$lp" in + *"@XDG_RUNTIME_DIR@"*) ok "the runtime directory is filled in, not shipped" ;; + *) bad "a hardcoded runtime directory would be installed" ;; esac - case "$vu" in - *"lfs-helper which-package"*) - ok "it says how to find out which package should own them" ;; - *) bad "the report gives no way forward" ;; + case "$lp" in + *"kept the old one as"*) ok "an existing profile is kept, not destroyed" ;; + *) bad "the user's own .bash_profile would be overwritten" ;; esac - cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" - case "$cv" in - *_vfy_unclaimed*) ok "verify runs the check" ;; - *) bad "the unclaimed check is never called" ;; + # and it must be wired into the shared path + sa="$(sed -n '/# 4. the session/,/# 5. a shared directory/p' "$pmt")" + case "$sa" in + *_link_xdg_profile*) ok "--shared links the profile as well as the group" ;; + *) bad "the profile is never linked" ;; esac fi - -# ---- a versioned chapter-9 "configuration" section is a PACKAGE ------------- # -# Book 9.2 is titled LFS-Bootscripts-20250827: a versioned tarball that unpacks -# and installs programs into /etc/rc.d/init.d and /lib/services. `lfs` gets -# this right -- _title_pkgver finds a version, so it generates a full phased -# build and deliberately keeps the step OUT of rootsteps, meaning "build this -# as a package user". -# -# lfs-helper then blanket-matched cfg_* as "not a package", refused to create -# the user, and the non-root path staged the script as that user anyway: -# 'cfg_bootscripts' is a build step, not a package -- no user created. -# install: invalid user 'cfg_bootscripts' -# !! could not stage the install script -# The build stopped at 88/99. Two tools, two answers, no shared authority. -# -# The script itself is the authority: a package has pkg_glob, prose does not. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _cd="$T/cfgpkg"; mkdir -p "$_cd" - printf 'pkg_glob="lfs-bootscripts-*.tar.*"\n' > "$_cd/cfg_bootscripts.sh" - printf '#!/bin/bash\necho x > /etc/hostname\n' > "$_cd/cfg_hostname.sh" - _r="$( SCRIPTS="$_cd"; PKGUSR_PREFIX="p_" - # _is_not_a_package strips the package-user prefix first, so the - # helper it calls has to be here too -- without it every name came - # back empty and matched nothing, which reads as "is a package". - eval "$(sed -n '/^unprefix_pkg_user() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^_is_not_a_package() {/,/^}/p' "$helper_src")" - # a versioned cfg section installs files and needs an owner - _is_not_a_package cfg_bootscripts && echo "bootscripts-has-no-user" - # a prose one writes root's config and must NOT get a user - _is_not_a_package cfg_hostname || echo "hostname-got-a-user" - # a cfg step with no script at all is prose until proven otherwise - _is_not_a_package cfg_nothing || echo "unknown-cfg-step-got-a-user" - # the genuinely-not-packages must stay that way - _is_not_a_package last-step || echo "last-step-got-a-user" - _is_not_a_package init-dirs || echo "init-dirs-got-a-user" - _is_not_a_package refind || echo "refind-got-a-user" - # and a normal package is still a package - _is_not_a_package gcc && echo "gcc-lost-its-user" - # callers pass the OWNER name too, already prefixed - _is_not_a_package p_cfg_bootscripts && echo "prefixed-form-misread" )" - if [ -z "$_r" ]; then - ok "a versioned cfg_* section is built as its own package user" - else - printf '%s\n' "$_r" | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - fi - # and lfs must agree: a versioned cfg section is kept out of rootsteps - grep -q 'pkg_like_cfg.append(cname)' "$LFS_TOOL" \ - && ok "lfs keeps a versioned cfg section out of the root steps" \ - || bad "lfs no longer treats a versioned cfg section as a package" +# the file must ship, and the Makefile must install it without clobbering. +# Beside the tool in a checkout; at /etc/pkgusr on an installed system -- +# that is where the Makefile puts it, so /usr/bin/lfs having no skel-u_xdg/ +# next to it is not a missing file. +skel="$(dirname "$LFS_TOOL")/skel-u_xdg/.bash_profile" +[ -f "$skel" ] || skel="/etc/pkgusr/skel-u_xdg/.bash_profile" +[ -f "$skel" ] \ + && ok "the XDG profile ships with the tools" \ + || bad "the XDG profile is not shipped" +mkf="$(dirname "$LFS_TOOL")/Makefile" +if [ -f "$mkf" ]; then + grep -q "skel-u_xdg" "$mkf" \ + && ok "make install installs it" \ + || bad "make install does not install the XDG profile" + grep -q "kept your existing .bash_profile" "$mkf" \ + && ok "reinstalling does not overwrite your edits" \ + || bad "a reinstall would clobber the XDG profile" fi -# ---- the triplet warning must not fire on a healthy toolchain --------------- # -# A finished build has TWO C++ header directories: chapter 6 installs libstdc++ -# under $LFS_TGT (x86_64-lfs-linux-gnu, or whatever vendor string is set) and -# chapter 8's native gcc installs its own under x86_64-pc-linux-gnu. Both are -# present and that is correct -- the chapter-6 one is simply left over. -# -# The check took the FIRST glob match and stopped, so it reported a mismatch -# whenever the stale directory sorted first -- which is alphabetical, i.e. luck: -# "nimgnu" < "pc", so a perfectly good toolchain warned on every single package. -# What matters is only whether a directory matching the compiler exists. -if [ -f "$helper_src" ]; then - _tp="$T/triplet"; mkdir -p "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits" \ - "$_tp/15.2.0/x86_64-pc-linux-gnu/bits" - touch "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits/c++config.h" \ - "$_tp/15.2.0/x86_64-pc-linux-gnu/bits/c++config.h" - # the fixed shape: scan every match, stay silent if one is the compiler's - _fn="$(sed -n '/^_warn_on_triplet_mismatch() {/,/^}/p' "$helper_src")" - _p="" - case "$_fn" in - *'[ "$have" = "$mine" ] && return 0'*) ;; - *) _p="$_p;the check does not stop at the compiler's own headers" ;; - esac - case "$_fn" in - *"break"*) _p="$_p;the check still stops at the first directory found" ;; - esac - case "$_fn" in - *'[ -n "$found" ] || return 0'*) ;; - *) _p="$_p;a tree with no C++ headers yet would warn" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the triplet warning ignores leftover chapter-6 headers" - fi +# ---- the sync carries the XDG profile too (1.11.29) ------------------------ # +# `make install` placed the profile, but a system whose tools arrive through +# install-tools / sync-tools -- which is how every chroot gets them -- never +# received it, and each --shared user creation warned about the missing file. +# The sync already carried skel-package verbatim; this one was forgotten. +python3 - "$LFS_TOOL" <<'PYXDG' +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "_install_xdg_skel"): + problems.append("the sync has nothing to place the XDG profile with") +else: + # an edited copy in the tree is never clobbered + t = tempfile.mkdtemp() + dst = os.path.join(t, "etc/pkgusr/skel-u_xdg/.bash_profile") + os.makedirs(os.path.dirname(dst)) + open(dst, "w").write("MINE\n") + lfs._install_xdg_skel(t, verbose=False) + if open(dst).read() != "MINE\n": + problems.append("the sync clobbered an edited XDG profile") + # a fresh tree receives it, whenever any source exists at all + src_host = "/etc/pkgusr/skel-u_xdg/.bash_profile" + src_repo = os.path.join(os.path.dirname(os.path.abspath(sys.argv[1])), + "skel-u_xdg", ".bash_profile") + if os.path.isfile(src_host) or os.path.isfile(src_repo): + t2 = tempfile.mkdtemp() + lfs._install_xdg_skel(t2, verbose=False) + if not os.path.isfile(os.path.join( + t2, "etc/pkgusr/skel-u_xdg/.bash_profile")): + problems.append("a fresh tree does not receive the XDG profile") +src = open(sys.argv[1]).read() +for site in ("_install_pkgusr_skel(lfs)\n _install_xdg_skel(lfs)", + "_install_xdg_skel(lfs, verbose=False)"): + if site not in src: + problems.append("a sync path skips the XDG profile") + break +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS install-tools and sync-tools carry the XDG profile") +PYXDG +_count_rc $? + +# ---- the completion travels with the sync too (1.11.31) -------------------- # +# Same forgotten-file story as the XDG profile: the Makefile installed it, the +# sync -- which is how every chroot gets its tools -- did not, so no synced +# system ever had tab completion. +python3 - "$LFS_TOOL" <<'PYCOMP' +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "_install_completion"): + problems.append("the sync has nothing to place the completion with") +else: + src_repo = os.path.join(os.path.dirname(os.path.abspath(sys.argv[1])), + "lfs-completion.bash") + if os.path.isfile(src_repo) or os.path.isfile( + "/usr/share/bash-completion/completions/lfs"): + t = tempfile.mkdtemp() + lfs._install_completion(t, verbose=False) + d = os.path.join(t, "usr/share/bash-completion/completions") + if not os.path.isfile(os.path.join(d, "lfs")): + problems.append("a fresh tree does not receive the completion") + for tool in ("blfs", "packagemanager", "lfs-helper"): + if os.readlink(os.path.join(d, tool)) != "lfs" \ + if os.path.islink(os.path.join(d, tool)) else True: + problems.append("%s has no completion symlink" % tool) + break +src = open(sys.argv[1]).read() +if "_install_completion(lfs)" not in src \ + or "_install_completion(lfs, verbose=False)" not in src: + problems.append("a sync path skips the completion") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS install-tools and sync-tools carry the bash completion") +PYCOMP +_count_rc $? + +# ---- update-all says what it will do, and what it will NOT ------------------ # +# `packagemanager update` with no names is the whole-system update. Three +# things its plan owed and did not deliver (1.11.31): +# * a package with local edits to its installed version's script would have +# them silently left behind by the fresh new-version script (the make-ca +# systemctl edit was this) -- now collected at the END of the dry run, +# and --run STOPS before the first one instead of discarding; +# * a package not in the BLFS book simply vanished from the plan -- now +# named, with `lfs update` pointed at for the LFS-book side; +# * both self-clear: a merged install_<new-version> in the home resolves +# the conflict, --regenerate declares the discard on purpose. +_pmt2="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$_pmt2" ]; then + python3 - "$_pmt2" <<'PYUPD' +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +problems = [] +if not hasattr(pm, "_pending_edit_conflict"): + problems.append("nothing detects edits an update would leave behind") +else: + t = tempfile.mkdtemp() + def w(n, body): + open(os.path.join(t, n), "w").write(body) + # edited: canonical differs from the as-run record -> conflict + w("install_foo-1.0", "make\n# my fix\n") + w("install_last", "make\n") + if not pm._pending_edit_conflict(t, "foo-1.0", "foo-1.1"): + problems.append("a pending edit is not seen as a conflict") + # unedited: identical -> no conflict + w("install_foo-1.0", "make\n") + if pm._pending_edit_conflict(t, "foo-1.0", "foo-1.1"): + problems.append("a pristine script is flagged as a conflict") + # edited but MERGED into the new version's script -> resolved + w("install_foo-1.0", "make\n# my fix\n") + w("install_foo-1.1", "make\n# my fix\n") + if pm._pending_edit_conflict(t, "foo-1.0", "foo-1.1"): + problems.append("a merged new-version script does not clear the conflict") + # a reinstall row and missing records judge nothing + if pm._pending_edit_conflict(t, "(reinstall)", "foo-1.1") \ + or pm._pending_edit_conflict(tempfile.mkdtemp(), "a-1", "a-2"): + problems.append("the conflict check judges what it cannot know") +src = open(sys.argv[1]).read() +if "LOCAL EDITS" not in src: + problems.append("conflicts are not collected at the end of the plan") +if "stopping BEFORE" not in src: + problems.append("--run would discard edits instead of stopping") +if "not in the BLFS book" not in src: + problems.append("packages outside the book vanish from the plan silently") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS update-all surfaces conflicts, stops for them, and names the rest") +PYUPD + _count_rc $? + + # ---- a status line is cleared by whoever set it (1.11.32) ---------------- # + # run_blfs announced "reading versions from the book" and left it standing; + # the caller's next print landed on the same line: + # reading versions from the bookChecking 97 package(s)... + _rb="$(_slice_py_fn() { sed -n "/^def $2(/,/^def /p" "$1"; }; _slice_py_fn "$_pmt2" run_blfs)" + _p="" + printf '%s' "$_rb" | grep -q 'finally:' \ + || _p="$_p;run_blfs can return with its status line still standing" + printf '%s' "$_rb" | grep -q 'clear_status()' \ + || _p="$_p;run_blfs never clears the status it set" + # the plan and its verdict name the book they compared against + grep -q '_current_blfs_book' "$_pmt2" \ + || _p="$_p;there is no one place that names the current book" + _cu="$(sed -n '/^def cmd_update(/,/^def /p' "$_pmt2")" + printf '%s' "$_cu" | grep -q '_current_blfs_book' \ + || _p="$_p;the update messages do not name the book" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the status line is cleared, and update names the book it read" + fi fi -# ---- the adoption gate must actually close --------------------------------- # -# Chapters 5-6 build before /etc/passwd exists, so their files are recorded in -# manifests with no owner. Once book 7.6 creates the user database, the -# adoption pass hands each package its files, and adopted.list records -# "name size" so it does not run again. -# -# It ran again on EVERY build-all, and the cause was that the gate and the -# recorder tested opposite conditions: -# _needs_adoption: user_exists "$owner" || return 0 # no user -> NEEDS -# _mark_all_adopted: user_exists "$owner" || continue # no user -> SKIP -# So a single manifest without a user could never be recorded and kept -# _has_unadopted_packages at yes forever. One rule now, used by both. -# -# The write also swallowed every error (2>/dev/null on the redirect AND -# `|| rm -f` on the move), so a missing or read-only state directory looked -# exactly like a successful record. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - # the state directory is sorted now: adopted.list lives under progress/ - _ad="$T/adopt"; mkdir -p "$_ad/manifests" "$_ad/progress" - for f in haveuser nouser; do echo "/usr/bin/$f" > "$_ad/manifests/$f.files"; done - _out="$( STATE="$_ad"; MANIFESTS="$_ad/manifests"; SCRIPTS="$_ad/scripts" - STATE_PROGRESS="$_ad/progress" - warn(){ :; } - user_exists(){ [ "$1" = haveuser ]; } # only one package has a user - pkg_owner_name(){ echo "$1"; } - is_done(){ return 1; } # none built by lfs-helper here - PKGUSR_PREFIX="p_" - # _is_not_a_package strips the prefix first, so its helper is needed too - for fn in _ADOPTED_LIST _manifest_size _adopted_size _mark_adopted \ - _mark_all_adopted _needs_adoption _has_unadopted_packages \ - unprefix_pkg_user _is_not_a_package; do - eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" +# the set-default hint about build_book is gone -- it answered a question +# nobody asked, on every invocation (1.11.32) +grep -q "the book the SYSTEM is built from is 'build_book'" "$LFS_TOOL" \ + && bad "set-default still lectures about build_book" \ + || ok "set-default says what it did and stops" + +# ---- a chosen book is a decision, and a decision is not silently overridden - # +# `blfs set-default 13.0-systemd` saved the choice; `packagemanager update` +# read 12.4. Three holes in one flow (1.11.33): set-default validated +# nothing, so an uncached default sent every lookup to the closest-cached +# fallback; blfs DID say so ("note: no book matched..."), but run_blfs +# captures stderr and threw the note away; and the fallback's book was shown +# by its filename, so even the wrong name was hard to read. +_bt2="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt2" ] && [ -f "$_pmt2" ]; then + _p="" + # set-default caches what it saves, or says plainly that it could not + _sd="$(sed -n '/^def cmd_set_default(/,/^def /p' "$_bt2")" + printf '%s' "$_sd" | grep -q 'ensure_book' \ + || _p="$_p;set-default saves a default it never validated" + printf '%s' "$_sd" | grep -q 'blfs fetch' \ + || _p="$_p;an uncachable default gives no way forward" + # the notes blfs writes reach the person running packagemanager + _rb2="$(sed -n '/^def run_blfs(/,/^def /p' "$_pmt2")" + printf '%s' "$_rb2" | grep -q 'note:' \ + || _p="$_p;blfs notes are still swallowed by the wrapper" + # books discovers by itself, and offline says cached, not current + _cb="$(sed -n '/^def cmd_books(/,/^def /p' "$_bt2")" + printf '%s' "$_cb" | grep -qv 'discover' \ + && : ; printf '%s' "$_cb" | grep -q 'offline' \ + || _p="$_p;an offline books listing poses as a live one" + _n=0; _n="$(printf '%s' "$_cb" | grep -c 'discover_all()')" + [ "${_n:-0}" -ge 2 ] \ + || _p="$_p;books does not discover without being asked" + # an unreachable index is a failure, not a tiny successful discovery + _da="$(sed -n '/^def discover_all(/,/^def /p' "$_bt2")" + printf '%s' "$_da" | grep -q 'raise' \ + || _p="$_p;an offline discovery still poses as a result" + # the person reads a selector ('12.4'), never a scraped filename. (This + # first asserted the filename-folding regex -- the implementation -- and + # broke one release later when the parse moved to the Default line. The + # property: the name never comes from a [bracketed] listing filename.) + _cbb="$(sed -n '/^def _current_blfs_book(/,/^def /p' "$_pmt2")" + printf '%s' "$_cbb" | grep -q 'split("\[", 1)' \ + && _p="$_p;the book is still named from a bracket-scraped filename" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" done - _has_unadopted_packages || echo "gate-was-closed-before-the-pass" - _mark_all_adopted - # the package WITHOUT a user must be recorded too -- that is the fix - grep -q '^nouser ' "$_ad/progress/adopted.list" 2>/dev/null \ - || echo "a package without a user is never recorded" - # and the gate must now be shut - _has_unadopted_packages && echo "the gate stays open after the pass" - # but a manifest that GROWS has new files to hand over -- reopen - echo /usr/bin/extra >> "$_ad/manifests/haveuser.files" - _has_unadopted_packages || echo "a grown manifest is not re-adopted" )" - if [ -z "$_out" ]; then - ok "the adoption gate closes once the pass has run" else - printf '%s\n' "$_out" | while IFS= read -r m; do + ok "a chosen book is cached, its notes are heard, and its name is read" + fi + + # ---- the wrapper asks for the default; discovery stays with the person ---- # + # Three at once from one real run (1.11.34): the book name came from the + # FIRST [bracketed] filename of the listing -- alphabetical, so 12.4 beat + # 13.0 and update reported the wrong book while using the right default was + # impossible to see; books' auto-discovery ran through the wrapper, so + # every packagemanager invocation probed the site twice; and enumeration + # fell into the resolve-for-use cached fallback, turning stale index + # directories ('oldsvn') into notes plus the cached book under a false name. + _p="" + _cb2="$(sed -n '/^def cmd_books(/,/^def /p' "$_bt2")" + printf '%s' "$_cb2" | grep -q '"brief"' \ + || _p="$_p;there is no discovery-free books listing for scripts" + grep -q '"--brief"' "$_bt2" \ + || _p="$_p;--brief is not registered" + grep -q 'def discover_dir(selector, listing' "$_bt2" \ + || _p="$_p;enumeration cannot be told apart from resolution" + grep -q 'discover_dir(d, listing=True)' "$_bt2" \ + || _p="$_p;the listing still resolves stale directories to the cached book" + _cbb2="$(sed -n '/^def _current_blfs_book(/,/^def /p' "$_pmt2")" + printf '%s' "$_cbb2" | grep -q '"--brief"' \ + || _p="$_p;the wrapper still triggers discovery to learn the default" + printf '%s' "$_cbb2" | grep -q 'Default:' \ + || _p="$_p;the book name is still scraped from a filename, not the default" + grep -q '_FORWARDED_NOTES' "$_pmt2" \ + || _p="$_p;the same note would repeat once per blfs call" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done + else + ok "the wrapper reads the default without probing, and notes say each thing once" fi - # a failed write must be reported, not swallowed: an unrecorded adoption - # means the pass runs forever and nothing on disk says why - _w="$( STATE=/dev/null/nope - warn(){ echo "reported"; } - for fn in _ADOPTED_LIST _manifest_size _mark_adopted; do - eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" + + # ---- an empty BLFS plan is not a clean bill for the whole system ---------- # + # `lfs update glibc` said 2.42 -> 2.43 while `packagemanager update` said + # "Nothing to update" -- true for the BLFS-matched subset, read as the whole + # system. The not-in-book report (1.11.31) printed only when the plan was + # non-empty: the early return skipped it in exactly the state that needs it + # most. Now (1.11.35) both paths share one report, and the dry run actually + # CONSULTS the LFS book through `lfs update` -- one door, lfs owns that + # comparison. + _p="" + _n=0; _n="$(grep -c '_report_no_book(no_book' "$_pmt2")" + [ "${_n:-0}" -ge 2 ] \ + || _p="$_p;the empty-plan path still hides the not-in-book packages" + grep -q 'def _lfs_update_report' "$_pmt2" \ + || _p="$_p;the LFS book is never consulted for the rest" + _lr="$(sed -n '/^def _lfs_update_report(/,/^def /p' "$_pmt2")" + printf '%s' "$_lr" | grep -q '"update"' \ + || _p="$_p;the LFS comparison is reimplemented instead of asked for" + # 'fetch' gets the script and runs nothing, in every tool + grep -q '"fetch"' "$_pmt2" \ + || _p="$_p;packagemanager script has no fetch" + _cs="$(sed -n '/^def cmd_script(/,/^def /p' "$_pmt2")" + printf '%s' "$_cs" | grep -q 'resolve_script' \ + || _p="$_p;script fetch does not go through the one script resolver" + grep -q 'def cmd_script_fetch' "$LFS_TOOL" \ + || _p="$_p;lfs script fetch does not exist" + _n2=0; _n2="$(grep -c '_ensure_lfs_script(' "$LFS_TOOL")" + [ "${_n2:-0}" -ge 3 ] \ + || _p="$_p;install and fetch do not share one script generator" + grep -q 'def cmd_search' "$LFS_TOOL" \ + || _p="$_p;there is no lfs search" + grep -q '"search", help="find a package or section' "$LFS_TOOL" \ + || _p="$_p;lfs search is not reachable" + _bs="$(sed -n '/^def cmd_script(/,/^def /p' "$_bt2")" + printf '%s' "$_bs" | grep -q '"fetch"' \ + || _p="$_p;blfs script does not read the shared fetch verb" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" done - _mark_adopted p 12 2>/dev/null && echo "claimed-success" )" - case "$_w" in - *reported*) ok "a failed adoption record is reported, not swallowed" ;; - *) bad "a failed adoption record still looks like success" ;; - esac -fi + else + ok "the whole system is answered for, and every tool fetches scripts alike" + fi -# ---- chapter-8 configuration must be generated, and must not rot ----------- # -# nsswitch.conf, the timezone and the dynamic-loader config were never -# generated: the step table only harvested chapter 9. Without /etc/nsswitch.conf -# glibc falls back to compiled-in defaults; without a timezone the system runs -# UTC forever. -# -# Two of those sections carry ids the BOOK TOOLCHAIN generated -- -# idm139921653990176 8.5.2.1. Adding nsswitch.conf -# -- which change every time the book is rebuilt. Naming one in the table -# works exactly once and then resolves to nothing, silently. Silently is the -# real danger: /etc/shells went missing from finished systems because -# cfg_shells resolved to zero blocks and was skipped without a word. So a step -# may name an id OR a title fragment, and this test fails if any chapter-8 -# entry stops resolving. -python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYCFG8' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() + # ---- every package lands in a named bucket, and unknown is not current ---- # + # 97 checked, "nothing to update", 1 reported as outside the book -- while + # `lfs update glibc` said 2.42 -> 2.43. The classification loop had two + # silent exits (1.11.36): a package whose state is not "installed" was + # dropped unless it had been NAMED (everything lfs-helper built has no + # install_last, so it reads that way), and `u.version and bv != u.version` + # made an UNKNOWN installed version falsy -- filing it as up to date. Not + # knowing is not the same as being current. + python3 - "$_pmt2" <<'PYBUCKET' +import sys, io, contextlib, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() problems = [] +src = open(sys.argv[1]).read() +seg = src[src.index("def cmd_update("):] +seg = seg[:seg.index("\ndef ", 10)] +# the predicate that read unknown as current is gone +if "u.version and bv != u.version" in seg: + problems.append("an unknown installed version is still read as up to date") +for bucket in ("no_record", "unjudged"): + if bucket not in seg: + problems.append("packages exit the loop into no bucket (%s)" % bucket) +# the report accounts for all of them, and says so +pm._lfs_update_report = lambda names: ("lines", [" glibc Glibc-2.42 -> Glibc-2.43"]) +buf = io.StringIO() try: - from bs4 import BeautifulSoup -except ImportError: - print(" SKIP beautifulsoup4 not installed"); sys.exit(0) - -if not hasattr(lfs, "CHROOT_CONFIG_STEPS_8"): - print(" FAIL chapter-8 configuration is not generated at all"); sys.exit(1) - -soup = BeautifulSoup(open(sys.argv[2], encoding='utf-8', errors='replace').read(), - 'html.parser') -secs = {sid: (t, sec) for sid, t, sec in lfs.iter_sections(soup)} - -for name, sid in lfs.CHROOT_CONFIG_STEPS_8: - e = lfs._resolve_section(secs, sid) - if not e: - problems.append("%s: '%s' resolves to no section" % (name, sid)) - elif not e[1]["commands"]: - problems.append("%s: resolved but has no command blocks" % name) - -# the resolver itself: id first, then title, and never a silent guess -if lfs._resolve_section(secs, "ch-config-hostname") is None: - problems.append("a stable section id no longer resolves") -if lfs._resolve_section(secs, "Adding nsswitch.conf") is None: - problems.append("a section cannot be found by its title") -if lfs._resolve_section(secs, "zzz-no-such-section") is not None: - problems.append("a nonexistent section resolves to something") -# an ambiguous fragment must NOT be resolved by picking the first match -if lfs._resolve_section(secs, "Configuring") is not None: - problems.append("an ambiguous title fragment is silently resolved") - -# /tools deletion and stripping are opt-in: neither should ever be a default -if lfs.remove_tools(): - problems.append("/tools is deleted by default") -if lfs.strip_binaries(): - problems.append("binaries are stripped by default") + with contextlib.redirect_stdout(buf): + pm._report_no_book(no_book=["p1"], consult_lfs=True, no_record=["glibc"], + unjudged=["bash"], uptodate_n=94, total=97) +except TypeError as e: + print(" FAIL the report cannot account for unjudged packages (%s)" % e) + sys.exit(1) +out = buf.getvalue() +if "97" not in out or "94" not in out: + problems.append("the report does not account for what was checked") +if "glibc" not in out: + problems.append("a package with no recorded version is not named") +if "Glibc-2.43" not in out: + problems.append("the LFS book's answer is not relayed") +# and it stays quiet when there is genuinely nothing unjudged +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + pm._report_no_book(no_book=[], consult_lfs=True, no_record=[], unjudged=[]) +if buf.getvalue().strip(): + problems.append("the report invents findings when everything was judged") +# ---- one update covers the system: BLFS preferred, LFS for the rest ------ +# The plan held only what the BLFS book carries -- two packages -- while +# twenty-nine LFS-book packages were merely REPORTED with a command to paste. +# "update all, preferring BLFS" was the request from the start. (1.11.50) +if not hasattr(pm, "_lfs_stale_list"): + problems.append("the LFS book's packages cannot enter the plan") +else: + import types as _t + class _R: + stdout = "glibc\tGlibc-2.42\tGlibc-2.43\nless\tLess-679\tLess-692\njunk\n" + returncode = 0 + _sp, _wh = pm.subprocess, pm.shutil.which + pm.subprocess = _t.SimpleNamespace(run=lambda *a, **k: _R(), + SubprocessError=Exception) + pm.shutil.which = lambda x: "/usr/bin/lfs" + try: + _got = pm._lfs_stale_list() + if _got != [("glibc", "Glibc-2.42", "Glibc-2.43"), + ("less", "Less-679", "Less-692")]: + problems.append("the machine-readable LFS list is misread") + finally: + pm.subprocess, pm.shutil.which = _sp, _wh +_cu2 = src[src.index("def cmd_update("):] +_cu2 = _cu2[:_cu2.index("\ndef ", 10)] +# the property is that the plan CONSULTS the LFS book -- not how the call is +# spelled. This first matched "_lfs_stale_list()" exactly and broke the +# moment the call took an argument. +if "_lfs_stale_list(" not in _cu2: + problems.append("the plan still stops at what BLFS carries") +if "_blfs_planned" not in _cu2: + problems.append("BLFS is no longer preferred over the LFS book") +if "_LFS_STEP" not in _cu2: + problems.append("an LFS-book entry cannot be told from a BLFS one") +if "--toolchain" not in src or "_TOOLCHAIN_RISK" not in src: + problems.append("rebuilding glibc needs no explicit consent") +# ---- rebuilding what is already "current" (1.11.53) ---- +# After glibc changes, "up to date" and "built against the current system" +# stop being the same statement. --reinstall must therefore reach the LFS +# book's up-to-date packages too, not just BLFS's. +import types as _ty +_seen = {} +class _RR: + returncode = 0 + stdout = "glibc\tGlibc-2.42\tGlibc-2.43\nbash\tBash-5.3\tBash-5.3\n" +_sp2, _wh2 = pm.subprocess, pm.shutil.which +pm.subprocess = _ty.SimpleNamespace( + run=lambda cmd, **k: (_seen.__setitem__("cmd", cmd), _RR())[1], + SubprocessError=Exception) +pm.shutil.which = lambda x: "/usr/bin/lfs" +try: + pm._lfs_stale_list(include_current=False) + if "--include-current" in _seen.get("cmd", []): + problems.append("a plain update would rebuild up-to-date packages") + pm._lfs_stale_list(include_current=True) + if "--include-current" not in _seen.get("cmd", []): + problems.append("--reinstall cannot reach the LFS book's current packages") +finally: + pm.subprocess, pm.shutil.which = _sp2, _wh2 +# the plan and the "not covered" report must not print the same package twice +if "_planned_now" not in _cu2: + problems.append("a planned package is also reported as unplanned") +# a systemd book on a SysV system: say which scripts call systemctl (1.11.51) +import tempfile as _tf, os as _os +if not hasattr(pm, "_systemctl_users"): + problems.append("nothing says which scripts need systemctl") +else: + _sd = _tf.mkdtemp() + _s1 = _os.path.join(_sd, "install_foo-1.0") + open(_s1, "w").write("make install\nsystemctl enable foo\n") + _s2 = _os.path.join(_sd, "install_bar-1.0") + open(_s2, "w").write("make install\n") + _steps = [("foo", "foo", "1.0", "1.1", _s1), ("bar", "bar", "1.0", "1.1", _s2)] + _isdir, _which, _home = pm.os.path.isdir, pm.shutil.which, pm.pkgusr_home + pm.os.path.isdir = lambda p: False if "systemd" in p else _isdir(p) + pm.shutil.which = lambda x: None + pm.pkgusr_home = lambda n: _sd + try: + _got = pm._systemctl_users(_steps) + if [r[0] for r in _got] != ["foo"]: + problems.append("the systemctl scan misreads which scripts need it") + if not _got or len(_got[0]) < 3 or not str(_got[0][2]).endswith("install_foo-1.0"): + problems.append("the systemctl hint does not say which file to edit") + pm.shutil.which = lambda x: "/usr/bin/systemctl" + if pm._systemctl_users(_steps): + problems.append("it warns about systemctl on a system that has it") + finally: + pm.os.path.isdir, pm.shutil.which, pm.pkgusr_home = _isdir, _which, _home if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS chapter-8 configuration resolves by id or title, and cannot rot") -PYCFG8 -_count_rc $? - -# cfg_rm-tools deletes a root-owned tree, so it must be a root step -- never a -# package user's job, and never adoptable -grep -q 'root_steps.append("cfg_rm-tools")' "$LFS_TOOL" \ - && ok "removing /tools is a root step" \ - || bad "removing /tools is not a root step" +print(" PASS every checked package is accounted for, and unknown is not current") +PYBUCKET + _count_rc $? -# ---- every build decision is asked UP FRONT --------------------------------- # -# The build used to discover decisions halfway through, or at the very end: -# last_build_step.sh blocked on `passwd root` after a six-hour build, and its -# own comment admits the non-interactive path leaves the system -# "unbootable-but-known". Collecting the answers in `session` is what lets -# `lfs run` and `build-all` finish without a person sitting there. -# -# They must also REACH the chroot: the config file is on the host and the -# chroot environment is reset, so anything not written into the tree's env -# file is simply not there when it is needed. -python3 - "$LFS_TOOL" <<'PYASK' -import sys, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() + # ---- one fact, two files: read whichever the build wrote (1.11.37) ------- # + # "97 checked, 0 current with the BLFS book, 97 it cannot judge" -- on a + # system where `lfs update` compares all 97 without trouble. Both build + # paths record what they installed; packagemanager read only its own + # (install_last) and was blind to lfs-helper's (VERSION). The comparison + # has to be case-insensitive with it: BLFS anchors are lowercase + # ('which-2.23'), the LFS book titles its packages ('Which-2.23'), and an + # exact compare would invent an update for every package both books carry. + python3 - "$_pmt2" <<'PYVER' +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() problems = [] -keys = [k for k, _, _ in lfs._SESSION_KEYS] -for k in ("timezone", "remove_tools", "strip", "main_user"): - if k not in keys: - problems.append("'%s' is not asked during session" % k) - -# Destructive or slow options must never default to on. Deleting the -# cross-toolchain and rewriting every binary are both things a person should -# have chosen, not discovered. -if lfs.remove_tools(): - problems.append("/tools is deleted by default") -if lfs.strip_binaries(): - problems.append("binaries are stripped by default") - -# each answer has to be written into the tree, or the chroot never sees it -body = open(sys.argv[1]).read() -i = body.index("def write_chroot_env") -j = body.find("\ndef ", body.index("_make_world_readable(path)", i)) -w = body[i:j if j > i else len(body)] -for var in ("LFS_TIMEZONE", "LFS_MAIN_USER", "LFS_STRIP", "LFS_REMOVE_TOOLS"): - if var not in w: - problems.append("%s never reaches the chroot" % var) +t = tempfile.mkdtemp() +h1 = os.path.join(t, "glibc"); os.makedirs(h1) +open(os.path.join(h1, "VERSION"), "w").write("# from: LFS 13.0\nGlibc-2.42\n") +if pm.read_install_last(h1).get("name_version") != "Glibc-2.42": + problems.append("a package built by lfs-helper still has no version") +# install_last stays authoritative where it exists +h2 = os.path.join(t, "which"); os.makedirs(h2) +open(os.path.join(h2, "VERSION"), "w").write("Which-1.0\n") +open(os.path.join(h2, "install_last"), "w").write('name_version="which-2.23"\n') +if pm.read_install_last(h2).get("name_version") != "which-2.23": + problems.append("VERSION overrides the install record it should defer to") +h3 = os.path.join(t, "empty"); os.makedirs(h3) +if pm.read_install_last(h3): + problems.append("a package with no record at all invents one") +if not hasattr(pm, "_same_version"): + problems.append("versions are compared without allowing for book spelling") +else: + if not pm._same_version("which-2.23", "Which-2.23"): + problems.append("the two books' capitalisation reads as an update") + if pm._same_version("Glibc-2.42", "Glibc-2.43"): + problems.append("a real version difference is missed") + if pm._same_version(None, "Glibc-2.43") or pm._same_version("", ""): + problems.append("a missing version compares equal to something") +seg = open(sys.argv[1]).read() +seg2 = seg[seg.index("def cmd_update("):] +seg2 = seg2[:seg2.index("\ndef ", 10)] +if "_same_version" not in seg2: + problems.append("the update loop does not use the tolerant comparison") + +# ---- the reader must ask for the ACCOUNT, and prefer the right record ---- +# (1.11.38) Two bugs one after the other, both silent: +# * gather_user looked the PACKAGE name up in the user database ('glibc'), +# but the account is 'p_glibc'. KeyError -> has_user false -> state +# 'broken' -> every package the LFS build made landed in update's +# "cannot judge" bucket with its home, VERSION and manifest right there. +# * the install script in that home records the version the BOOK has (it is +# regenerated when the book moves on), so reading it before VERSION would +# compare the book with itself and call everything current, forever. +gu = seg[seg.index("def gather_user("):] +gu = gu[:gu.index("\ndef ", 10)] +if "getpwnam(pkgusr_name(" not in gu: + problems.append("the reader looks up a package name in the user database") +ril = seg[seg.index("def read_install_last("):] +ril = ril[:ril.index("\ndef ", 10)] +if ril.index("VERSION") > ril.index('"install_" + os.path.basename'): + problems.append("the book's own version is read before the installed one") +# functional: a home shaped exactly like an lfs-helper-built package +h4 = os.path.join(t, "p_shaped"); os.makedirs(h4) +open(os.path.join(h4, "VERSION"), "w").write("# from: LFS 13.0\nGlibc-2.42\n") +open(os.path.join(h4, "install_p_shaped"), "w").write('name_version="Glibc-2.43"\n') +if pm.read_install_last(h4).get("name_version") != "Glibc-2.42": + problems.append("the installed version loses to the script's book version") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS every build decision is collected before the build starts") -PYASK -_count_rc $? - -# and the accounts step must USE the configured name rather than stopping to -# ask for it -- the whole point of asking early -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - case "$(_slice_fn "$helper_src" cmd_init_accounts)" in - *main_user*) ok "the final step uses the login account chosen up front" ;; - *) bad "the final step stops to ask for a name already configured" ;; - esac +print(" PASS both build paths' version records are read, and compared fairly") +PYVER + _count_rc $? fi -# ---- a section that ships a FILE instead of commands ----------------------- # -# Book 9.6.8 (The rc.site File) has no <kbd class="command"> at all. It prints -# /etc/sysconfig/rc.site as a <pre class="auto"> block with every line -# commented out, for the reader to copy and uncomment. The parser harvests -# command blocks only, so cfg_rcsite resolved to zero blocks and the file was -# never created -- the same silent-nothing failure that once lost /etc/shells. -# -# An all-commented file is still worth installing: it documents every knob the -# boot scripts have, in the place the boot scripts look for it. -python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYRCSITE' -import sys, subprocess, tempfile, os, importlib.machinery as m +# ---- a snapshot must say what state it captured ----------------------------- # +# Restoring a snapshot named "chrosscompile_done" brought back a tree whose +# toolchain was ALREADY broken -- the snapshot had been taken after chapter 8 +# damaged it. Nothing said so, at save or at restore, so it looked like the +# restore had failed rather than the snapshot being bad. +python3 - "$LFS_TOOL" <<'PY70' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(store, "config.json"), "w")) lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -problems = [] -if not hasattr(lfs, "_auto_file_block"): - print(" FAIL a section that ships a file instead of commands is ignored") - sys.exit(1) - -raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() -blk = lfs._auto_file_block(raw, "ch-config-site", "/etc/sysconfig/rc.site") -if not blk: - problems.append("rc.site is not extracted from the book") -else: - d = tempfile.mkdtemp() - sh = blk.replace("/etc/sysconfig", d + "/etc/sysconfig") - open(d + "/t.sh", "w").write("set -e\n" + sh + "\n") - r = subprocess.run(["bash", d + "/t.sh"], capture_output=True, text=True) - out = d + "/etc/sysconfig/rc.site" - if r.returncode != 0: - problems.append("the generated rc.site block is not valid bash: %s" - % r.stderr.strip()[:80]) - elif not os.path.isfile(out): - problems.append("the block runs but writes no file") - else: - body = open(out).read() - if len(body.splitlines()) < 50: - problems.append("rc.site came out truncated (%d lines)" - % len(body.splitlines())) - # a quoted heredoc: the body is full of # and shell metacharacters and - # must land byte-for-byte as the book prints it, not expanded - if "$" in body and "EOF" not in blk.split("\n")[1]: - problems.append("the heredoc is not quoted -- the body would expand") +if not hasattr(lfs, "_toolchain_state"): + print(" FAIL a snapshot does not record the toolchain state"); sys.exit(1) -# a section with neither commands nor a file body must still be reported, not -# silently skipped -if lfs._auto_file_block(raw, "ch-config-sysklogd", "/etc/x") is not None: - problems.append("a prose section is treated as if it shipped a file") +def tree(headers=None, native=False): + t = tempfile.mkdtemp() + if headers: + p = os.path.join(t, "usr", "include", "c++", "15.2.0", headers, "bits") + os.makedirs(p) + open(os.path.join(p, "c++config.h"), "w").close() + if native: + os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) + return t +problems = [] +if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu")).startswith("ok"): + problems.append("a clean chapter-6 tree is not recognised") +if not lfs._toolchain_state(tree("x86_64-nimgnu-linux-gnu", native=True)) \ + .startswith("BROKEN"): + problems.append("a native gcc over the cross one is not detected") +if "unfinished" not in lfs._toolchain_state(tree()): + problems.append("a tree with no C++ headers is not detected") +# headers under a different target than configured +st = lfs._toolchain_state(tree("x86_64-lfs-linux-gnu")) +if not st.startswith("BROKEN"): + problems.append("headers under the wrong target are not detected") +src = open(sys.argv[1]).read() +if "Restoring it later will restore the breakage too" not in src: + problems.append("saving a broken tree gives no warning") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS a section that ships a file body is installed, not skipped") -PYRCSITE +print(" PASS a snapshot records and reports the state it captured") +PY70 _count_rc $? -# ---- the package-user prefix must be one decision, not three --------------- # -# lfs-helper has ALWAYS read a prefix and defaulted to `p`: -# PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" -# but nothing ever wrote the setting -- `lfs` did not put it in the chroot env -# and `packagemanager` did not implement it at all. So the default applied by -# accident, and the built system would look for `gcc` while the build had -# created `p_gcc`. Same species as cfg_bootscripts and adopted.list: one -# decision held in several places that could disagree. -# -# Prefixing MUST be idempotent. It is applied to names typed by a person -# (gcc), to directory names read back from disk (p_gcc), and to values that -# already went through it -- lfs-helper's own comment names p_p_gcc as the -# thing to avoid. -python3 - "$LFS_TOOL" <<'PYPFX' -import sys, os, tempfile, importlib.machinery as m -problems = [] +# ---- a snapshot label must not hide what it captured ------------------------ # +# "chapters 5-6: 22/22 steps" was all a snapshot recorded, so a tree where +# chapter 8 had already run -- and already broken the toolchain -- looked +# pristine. Restoring it restored the damage, and it read as a failed restore +# rather than a bad snapshot. +python3 - "$LFS_TOOL" <<'PY71' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(store, "config.json"), "w")) lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") -if not os.path.exists(pmp): - print(" SKIP packagemanager not found next to the tool"); sys.exit(0) -pm = m.SourceFileLoader('pm', pmp).load_module() - -if not hasattr(lfs, "pkgusr_prefix"): - problems.append("lfs cannot say what the package-user prefix is") -if not hasattr(pm, "pkgusr_name"): - problems.append("packagemanager does not implement the prefix") - -if not problems: - # the two tools must agree, or the build and the built system diverge - if lfs.pkgusr_prefix() != pm.pkgusr_prefix(): - problems.append("lfs and packagemanager disagree: %r vs %r" - % (lfs.pkgusr_prefix(), pm.pkgusr_prefix())) - if pm.pkgusr_prefix() != "p_": - problems.append("the default prefix is not p_ (got %r)" - % pm.pkgusr_prefix()) - if pm.pkgusr_name("gcc") != "p_gcc": - problems.append("a package name is not prefixed") - # idempotent -- the whole reason every call site can apply it blindly - if pm.pkgusr_name(pm.pkgusr_name("gcc")) != "p_gcc": - problems.append("prefixing twice gives p_p_gcc") - if pm.unprefix_pkgusr("p_gcc") != "gcc": - problems.append("the prefix cannot be stripped for display") - # the OTHER kinds of account must not be touched: an application user and - # a collector group are not packages - if pm.pkgusr_name("u_firefox") != "u_firefox": - problems.append("an application user was given the package prefix") - if pm.pkgusr_name(pm.collector_prefix() + "python") \ - != pm.collector_prefix() + "python": - problems.append("a collector group was given the package prefix") - # the home is the ACCOUNT's, whichever kind of name it is asked with - if pm.pkgusr_home("gcc") != pm.pkgusr_home("p_gcc"): - problems.append("a package name and its user name give different homes") - -# and it has to REACH the chroot, or lfs-helper falls back to a guess -body = open(sys.argv[1]).read() -if "LFS_PKGUSR_PREFIX" not in body: - problems.append("the prefix never reaches the chroot") -if "pkgusr_prefix" not in [k for k, _, _ in lfs._SESSION_KEYS]: - problems.append("the prefix is never asked for") +problems = [] +if not hasattr(lfs, "_tree_state"): + problems.append("nothing reports the state of a tree") +if not hasattr(lfs, "cmd_bs_verify"): + problems.append("there is no way to ask what state a tree is in") +t = tempfile.mkdtemp() +os.makedirs(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress")) +os.makedirs(os.path.join(t, "usr", "src", "zlib")) +os.makedirs(os.path.join(t, "usr", "lib", "gcc", "x86_64-pc-linux-gnu")) +p = os.path.join(t, "usr", "include", "c++", "15.2.0", + "x86_64-nimgnu-linux-gnu", "bits") +os.makedirs(p); open(os.path.join(p, "c++config.h"), "w").close() +names = [n for n, _s, _t in lfs.CROSSCHAIN_STEPS] +json.dump({"book": "12.4", "done": names, "lfs_tgt": "x86_64-nimgnu-linux-gnu"}, + open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "crosschain-progress.json"), "w")) +open(os.path.join(t, "usr", "src", "lfs-pkgusr", "progress", "steps-built"), "w").write("zlib\n") +state = dict(lfs._tree_state(t)) +# chapters 5-6 alone is not the whole story +if "chapters 7-9" not in state: + problems.append("chapter 7-9 progress is not reported") +if "WARNING" not in state: + problems.append("a native gcc in the tree is not flagged") +if not state.get("toolchain", "").startswith("BROKEN"): + problems.append("the toolchain state is not reported") +# and the snapshot must carry it +src = open(sys.argv[1]).read() +if '"state":' not in src: + problems.append("the snapshot metadata does not record the tree state") if problems: - for p in problems: print(" FAIL %s" % p) + for p2 in problems: print(" FAIL %s" % p2) sys.exit(1) -print(" PASS lfs, lfs-helper and packagemanager share one package-user prefix") -PYPFX +print(" PASS a tree's real state is reported and recorded with snapshots") +PY71 _count_rc $? -# lfs-helper's side: it must take the value from the environment lfs wrote, -# and an explicitly EMPTY value must mean "no prefix" -- not fall through to -# the default and rename every user out from under an existing tree +# ---- staged installs must cope with the merged-/usr symlinks ---------------- # +# Book 4.2 makes /bin -> usr/bin, /sbin -> usr/sbin, /lib -> usr/lib. A +# package installing into /bin produces a REAL bin/ directory in the staging +# tree, and copying that over the symlink fails: +# cp: cannot overwrite non-directory '/bin' with directory '.../bin' +# That broke every package installing into those paths -- most of chapter 7 -- +# on a completely fresh build. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - grep -q 'PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-' "$helper_src" \ - && ok "an empty prefix means none, and is not the same as unset" \ - || bad "an empty prefix falls through to the default" -fi - -# ---- accounts live in subdirectories by kind ------------------------------- # -# Flat under /usr/src, a hundred package users and a handful of config steps -# were one undifferentiated list, and nothing about a directory said what kind -# of thing it was. Now: -# /usr/src/pkgusr/p_gcc a package -# /usr/src/cfg/p_cfg_bootscripts a config step that installs files -# All three tools have to agree, or the build creates a home the built system -# never looks in. -python3 - "$LFS_TOOL" <<'PYSPLITHOME' -import sys, os, tempfile, importlib.machinery as m -problems = [] -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") -if not os.path.exists(pmp): - print(" SKIP packagemanager not found next to the tool"); sys.exit(0) -d = tempfile.mkdtemp(); os.environ["PKGUSR_BASE"] = d -pm = m.SourceFileLoader('pm', pmp).load_module() + grep -q "^_resolve_staged_symlink_dirs()" "$helper_src" \ + && ok "staged directories are folded onto the tree's symlinks" \ + || bad "a package installing into /bin cannot be merged" + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + i_fold="$(printf '%s' "$ms" | grep -n "_resolve_staged_symlink_dirs" | head -1 | cut -d: -f1)" + i_cp="$(printf '%s' "$ms" | grep -n '_install_staged_tree' | head -1 | cut -d: -f1)" + if [ -n "$i_fold" ] && [ -n "$i_cp" ] && [ "$i_fold" -lt "$i_cp" ]; then + ok "the folding happens before the copy" + else + bad "the copy runs first and still fails on the symlinks" + fi + # driven by the live tree, not a hardcoded list of names + rs="$(sed -n '/^_resolve_staged_symlink_dirs() {/,/^}/p' "$helper_src")" + case "$rs" in + *'[ -L "$live" ] || continue'*) + ok "it folds whatever the tree actually has as a symlink" ;; + *) bad "the symlink names are hardcoded" ;; + esac -for k in ("target_pkgusr_home", "target_cfguser_home", - "pkgusr_home", "cfguser_home"): - if k not in lfs.LAYOUT_DEFAULTS: - problems.append("layout has no '%s'" % k) -if lfs.LAYOUT_DEFAULTS.get("pkgusr_home") == lfs.LAYOUT_DEFAULTS.get("cfguser_home"): - problems.append("packages and config steps share one directory") -if not lfs.LAYOUT_DEFAULTS.get("pkgusr_home", "").endswith("/pkgusr"): - problems.append("packages do not default to /usr/src/pkgusr") -if not lfs.LAYOUT_DEFAULTS.get("cfguser_home", "").endswith("/cfg"): - problems.append("config steps do not default to /usr/src/cfg") + # behaviour: bin/ and sbin/ fold, usr/bin/ is left alone + sd="$T/stagedirs"; mkdir -p "$sd/usr/bin" "$sd/usr/sbin" + ln -sf usr/bin "$sd/bin"; ln -sf usr/sbin "$sd/sbin" + mkdir -p "$sd/stage/bin" "$sd/stage/usr/bin" + echo m > "$sd/stage/bin/mount"; echo w > "$sd/stage/usr/bin/wall" + ( SNAP_ROOT="$sd"; say() { :; } + eval "$rs" + _resolve_staged_symlink_dirs "$sd/stage" ) >/dev/null 2>&1 + if [ -f "$sd/stage/usr/bin/mount" ] && [ ! -d "$sd/stage/bin" ] \ + && [ -f "$sd/stage/usr/bin/wall" ]; then + ok "bin/ is folded into usr/bin/ and nothing else is disturbed" + else + bad "the folding did not produce the expected staging tree" + fi +fi -# a package and a config step must land in DIFFERENT roots, whichever form of -# the name they are asked with -if pm.pkgusr_home("gcc") == pm.pkgusr_home("cfg_bootscripts"): - problems.append("a package and a config step share a root") -if not pm.pkgusr_home("gcc").startswith(os.path.join(d, "pkgusr")): - problems.append("a package is not under the package root") -if not pm.pkgusr_home("cfg_bootscripts").startswith(os.path.join(d, "cfg")): - problems.append("a config step is not under the config root") -# prefixed form must resolve identically -- callers pass both -if pm.pkgusr_home("p_cfg_bootscripts") != pm.pkgusr_home("cfg_bootscripts"): - problems.append("the prefixed name lands somewhere else") - -# listing must walk BOTH roots, or half the accounts vanish -os.makedirs(os.path.join(d, "pkgusr", pm.pkgusr_name("gcc"))) -os.makedirs(os.path.join(d, "cfg", pm.pkgusr_name("cfg_bootscripts"))) -got = pm.list_all_users() -for want in ("gcc", "cfg_bootscripts"): - if want not in got: - problems.append("'%s' is missing from the account list %r" % (want, got)) - -# and both roots must reach the chroot -body = open(sys.argv[1]).read() -for var in ("LFS_PKGUSR_ROOT", "LFS_CFGUSR_ROOT"): - if var not in body: - problems.append("%s never reaches the chroot" % var) - -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS packages and config steps live in separate roots") -PYSPLITHOME -_count_rc $? - -# lfs-helper's side: one chokepoint that routes by kind, and the passes that -# walk every account must walk both roots -- not just the package one +# ---- replacing a library a running system is using -------------------------- # +# `cp -a` opens the destination with O_TRUNC and writes. For a shared library +# that is currently mapped that is fatal: every running process sees the file +# shrink under it. Merging glibc that way killed the shell mid-install -- +# /usr/bin/env: error while loading shared libraries: +# /usr/lib/libc.so.6: file too short +# -- and the chroot could not start any program afterwards. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _o="$( SRCROOT=/tmp/_hr; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" - PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" - # pkg_owner_name too: pkgusr_home_for calls it, and without it every - # home came back as the bare root with no account on the end - for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ - unprefix_pkg_user pkg_owner_name pkgusr_home_for pkgusr_roots; do - eval "$(_slice_fn "$helper_src" "$fn")" - done - # a package gets the package prefix and the package root - [ "$(pkgusr_home_for gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "package-in-wrong-root" - [ "$(pkgusr_home_for p_gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "prefixed-package-in-wrong-root" - # a config step gets the CONFIG prefix and the config root -- not the - # package prefix stacked on top of its own - [ "$(pkgusr_home_for cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "cfg-in-wrong-root" - # and a legacy p_cfg_ name is repaired rather than carried forward - [ "$(pkgusr_home_for p_cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "stacked-cfg-not-repaired" - [ "$(pkgusr_roots | wc -l)" = 2 ] || echo "not-both-roots-walked" )" - if [ -z "$_o" ]; then - ok "lfs-helper routes each account to the root for its kind" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi - # no home may be built by hand any more -- that is how the two roots drift - grep -q '"\$SRCROOT/\$name"' "$helper_src" \ - && bad "a home is still built from SRCROOT directly" \ - || ok "every home goes through the one chokepoint" + grep -q "^_install_staged_tree()" "$helper_src" \ + && ok "staged files are installed one at a time" \ + || bad "the merge still overwrites files in place" + it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it" in + *'mv -f "$tmp" "$dest"'*) + ok "each file is renamed into place, never truncated" ;; + *) bad "files are still written over in place" ;; + esac + # the temp file must be in the SAME directory, or mv is a copy, not a rename + case "$it" in + *'tmp="$dest.pkgusr-new.$$"'*) + ok "the temporary is beside its destination, so mv is atomic" ;; + *) bad "the temporary is elsewhere and mv would copy" ;; + esac + ms="$(sed -n '/^_merge_stage() {/,/^}/p' "$helper_src")" + case "$ms" in + *'cp -a "$stage"/. '*) bad "the wholesale cp -a is still there" ;; + *) ok "the wholesale copy is gone" ;; + esac + + # behaviour: replacing a file a process holds open must not disturb it + ad="$T/atomic"; mkdir -p "$ad/usr/lib" "$ad/stage/usr/lib" + printf 'OLDOLDOLD' > "$ad/usr/lib/lib.so" + printf 'NEWNEWNEWNEW' > "$ad/stage/usr/lib/lib.so" + before_ino="$(stat -c %i "$ad/usr/lib/lib.so")" + ( SNAP_ROOT="$ad"; say() { :; }; warn() { :; } + eval "$it" + _install_staged_tree "$ad/stage" ) >/dev/null 2>&1 + after_ino="$(stat -c %i "$ad/usr/lib/lib.so")" + if [ "$before_ino" != "$after_ino" ] \ + && [ "$(cat "$ad/usr/lib/lib.so")" = "NEWNEWNEWNEW" ]; then + ok "the replacement is a new inode, so open files keep the old one" + else + bad "the file was overwritten in place (inode unchanged)" + fi fi -# ---- the home a build cd's into is the home the account has ---------------- # -# The build creates the account with useradd -d <home>, then cd's into that -# home to run the staged script. Those two must be the same path, and they -# are reached from DIFFERENT names: cmd_add_user has the owner -# ("p_man-pages"), build_pkg has the step name ("man-pages"). -# -# pkgusr_home_for routed by kind but never applied the prefix, so: -# # created package user p_man-pages ... -# line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory -# Chapter 7 did not catch it -- root steps mkdir -p their own directory first, -# so it surfaced at the first real package user, nine steps in. +# ---- staging is not worth paying for on every package ----------------------- # +# Staging costs roughly an extra copy of everything installed -- measurably +# slower on a big package. It is worth it only where an interrupted install +# cannot be recovered from: interrupt man-pages and you rebuild man-pages; +# interrupt gcc or glibc and there is no compiler and no libc left to rebuild +# them WITH. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _o="$( SRCROOT=/usr/src; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" - PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" - for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ - pkg_owner_name unprefix_pkg_user pkgusr_home_for; do - eval "$(_slice_fn "$helper_src" "$fn")" - done - # THE bug: a step name and its owner must give one home - [ "$(pkgusr_home_for man-pages)" = "$(pkgusr_home_for p_man-pages)" ] \ - || echo "step name and owner give different homes" - # and it must be the account's, i.e. prefixed - [ "$(pkgusr_home_for man-pages)" = "/usr/src/pkgusr/p_man-pages" ] \ - || echo "the home is not the account's: $(pkgusr_home_for man-pages)" - # book titles are capitalised, tarballs are not - [ "$(pkgusr_home_for Man-Pages)" = "$(pkgusr_home_for man-pages)" ] \ - || echo "a capitalised book title gives a different home" - # earlier stages fold onto the real package -- util-linux-tmp is - # util-linux's, which is where its files end up - [ "$(pkgusr_home_for util-linux-tmp)" = "$(pkgusr_home_for util-linux)" ] \ - || echo "an earlier stage gets its own home" - [ "$(pkgusr_home_for gcc-pass1)" = "$(pkgusr_home_for gcc)" ] \ - || echo "a pass1 build gets its own home" - # config steps still route to their own root, and wear ONE prefix -- - # their own. This used to read p_cfg_bootscripts: the package prefix - # stacked on the step name's own, so the account claimed to be a package - # while living in the config root. - [ "$(pkgusr_home_for cfg_bootscripts)" = "/usr/src/cfg/cfg_bootscripts" ] \ - || echo "a config step is not under the config root" )" - if [ -z "$_o" ]; then - ok "a step name and its owner resolve to the same account home" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi - - # the rule must be reachable from shell, or every other script re-derives it - # -- and a third copy is how the roots drift apart - grep -q 'pkgusr-home)' "$helper_src" \ - && ok "the home rule is available to scripts as 'lfs-helper pkgusr-home'" \ - || bad "shell scripts have to re-derive where an account lives" -fi + grep -q "^STAGE_PACKAGES_DEFAULT=" "$helper_src" \ + && ok "only the packages that cannot be recovered are staged" \ + || bad "every package pays the staging cost" + for p in gcc glibc binutils bash coreutils; do + grep -q "^STAGE_PACKAGES_DEFAULT=.*$p" "$helper_src" \ + || bad "$p is not staged, but an interrupted install would be fatal" + done + ok "the toolchain packages are all in the default list" -# nothing outside the chokepoints may build an account path by hand -for _f in packagemanager_install lfs-completion.bash; do - _p="$(dirname "$LFS_TOOL")/$_f" - [ -f "$_p" ] || continue - if grep -qE '"?/usr/src/\$[a-z_]+' "$_p"; then - bad "$_f builds an account path by hand" - else - ok "$_f asks for the account home instead of building it" - fi -done + ss="$(sed -n '/^_should_stage() {/,/^}/p' "$helper_src")" + # -tmp and -passN are the same package at an earlier stage + case "$ss" in + *'base="${name%-tmp}"'*) ok "the -tmp and -passN variants count too" ;; + *) bad "gcc-pass1 would not be staged" ;; + esac + case "$ss" in + *"LFS_STAGE_PACKAGES"*) ok "the list is configurable" ;; + *) bad "the list is hardcoded" ;; + esac + case "$ss" in + *"all) return 0"*"none) return 1"*) ok "all / none are accepted" ;; + *) bad "there is no way to turn staging fully on or off" ;; + esac + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *"--stage)"*"--no-stage)"*) ok "a single build can override it" ;; + *) bad "staging cannot be forced or skipped per build" ;; + esac -# ---- chapter-8 config steps must not stop the build ------------------------ # -# Two ways they did, both from lifting a sub-section out of its parent package. -# -# cfg_timezone (8.5.2.2) ends with an INTERACTIVE menu whose only output is -# the name to put in the placeholder: -# tzselect -# ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime -# `session` already asks for the timezone, so both are answered. The same -# section also unpacks tzdata with a path relative to GLIBC's build directory -# (`tar -xf ../../tzdata2025b.tar.gz`), which resolves nowhere once the step -# stands on its own. -python3 - "$LFS_TOOL" <<'PYCFG8RUN' -import sys, re, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -problems = [] -body = ("tar -xf ../../tzdata2025b.tar.gz\n" - "zic -d $ZONEINFO -p America/New_York\n" - "tzselect\n" - "ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime\n") -out, left = lfs._fill_placeholders(body, "cfg_timezone") -if left: - problems.append("the timezone step still asks for %s" % left) -if "<xxx>" in out: - problems.append("the timezone placeholder is not filled in") -if re.search(r'^\s*tzselect\s*$', out, flags=re.M): - problems.append("the interactive tzselect menu still runs") -if "../../tzdata" in out: - problems.append("tzdata is still read relative to glibc's build directory") -if "SOURCES_DIR" not in out: - problems.append("tzdata is not taken from the sources directory") -if "/usr/share/zoneinfo//etc/localtime" in out.replace(" ", ""): - problems.append("the localtime symlink lost its zone name") -if problems: - for p in problems: print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the timezone step is answered from config, not from a prompt") -PYCFG8RUN -_count_rc $? + # behaviour + ( STAGE_PACKAGES_DEFAULT="$(sed -n 's/^STAGE_PACKAGES_DEFAULT=\"\(.*\)\"$/\1/p' \ + "$helper_src" | head -1)" + eval "$ss" + _should_stage gcc && _should_stage glibc && _should_stage gcc-pass2 \ + && ! _should_stage man-pages && ! _should_stage zlib ) \ + && ok "gcc and glibc stage; man-pages and zlib do not" \ + || bad "the wrong packages are being staged" +fi -# cfg_ld rewrites /etc/ld.so.conf -- which glibc created and still owns -- and -# makes one directory. Nothing NEW appears, so the manifest count is 0 and the -# step was failed as a silent no-op: -# mkdir: created directory '/etc/ld.so.conf.d' -# !! cfg_ld reported success but installed NO files. -# That is the normal shape of a configuration step. Packages must still fail. +# ---- the staging directory must belong to whoever builds -------------------- # +# It is created by root, but the build runs as the PACKAGE USER, so +# `make install DESTDIR=...` could not write into it: +# mkdir: cannot create directory '/.lfs-pkgusr/stage/gcc': Permission denied helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - # to the end of the branch, not a fixed window: the message grew a - # breakdown of where the count came from, and a +6 window stopped - # reaching the `return 1` it is asserting - _blk="$(sed -n '/reported success but installed NO files/,/^ fi$/p' "$helper_src")" - # the whole function, not a fixed window above the message: the branch - # gained a third case (a re-run over its own work) and a -B 14 window - # stopped reaching the config-step test it is asserting - _pre="$(_slice_fn "$helper_src" cmd_build)" - case "$_pre" in - *'_is_not_a_package "$name"'*) - ok "a config step may install no new files; a package may not" ;; - *) bad "installing no new files still fails a configuration step" ;; + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *'chown "$owner:$owner" "$_stage"'*) + ok "the staging directory is given to the package user" ;; + *) bad "the package user cannot write into its own staging directory" ;; esac - case "$_blk" in - *'return 1'*) ok "a package that installs nothing still fails" ;; - *) bad "the no-files check no longer fails a package" ;; + # and the parent must be traversable, or the chown alone is not enough + case "$cb" in + *'chmod 755 "$(dirname "$_stage")"'*) + ok "the staging parent is traversable" ;; + *) bad "the package user cannot reach into the staging directory" ;; + esac + # a root step needs no chown -- there is no package user to give it to + case "$cb" in + *'[ "$as_root" != 1 ] && user_exists "$owner"'*) + ok "a root step is not given away to a user that may not exist" ;; + *) bad "the chown would fail on a root step" ;; esac + # the owner must be known BEFORE the stage is made, or it chowns to nothing + i_owner="$(printf '%s' "$cb" | grep -n 'owner="\$(pkg_owner_name' | head -1 | cut -d: -f1)" + i_stage="$(printf '%s' "$cb" | grep -n '_stage="\$(_stage_dir_for' | head -1 | cut -d: -f1)" + if [ -n "$i_owner" ] && [ -n "$i_stage" ] && [ "$i_owner" -lt "$i_stage" ]; then + ok "the owner is resolved before the staging directory is made" + else + bad "the staging directory is created before the owner is known" + fi fi -# ---- a symlinked install directory is checked at its target ---------------- # -# Book 7.x makes /bin, /sbin, /lib and /lib64 symlinks into /usr. `test -d` -# FOLLOWS a symlink but `stat` does not, so verify compared the LINK's own -# mode and owner -- always lrwxrwxrwx root:root -- against what a directory -# should look like, and reported all of them on every run: -# /bin root:root lrwxrwxrwx (want root:install, group-writable) -# Chowning a symlink changes nothing that matters, and the target (/usr/bin) -# is in the same list and checked on its own. Both the report and the --fix -# path need the guard: without it, --fix "repairs" a symlink forever. +# ---- a package must own the files its earlier stages installed -------------- # +# The same package is built more than once: ncurses in chapter 6 (as the lfs +# user), perl-tmp in chapter 7 (as root), then the real one in chapter 8 as +# its package user. The earlier files keep the earlier owner, and the package +# user cannot overwrite or re-mode them: +# cp: cannot create regular file '/usr/bin/tic': Permission denied +# Couldn't chmod 644 /usr/lib/perl5/.../perl.pod: Operation not permitted +# One cause, two chapters -- so one mechanism, applied to every package. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - # There used to be two copies of the install-dir rule and so two guards to - # count. cmd_verify calls _vfy_install_dirs now, so there is one -- and - # the check and the fix share it, which is the property this was after. - _vid="$(_slice_fn "$helper_src" _vfy_install_dirs)" - if printf '%s\n' "$_vid" | grep -q '\[ -L "\$live" \] && continue'; then - ok "a symlinked install dir is skipped by both the check and the fix" + grep -q "^_claim_earlier_stages()" "$helper_src" \ + && ok "a package claims the files its earlier stages installed" \ + || bad "a rebuild cannot overwrite its own earlier files" + # the owner filter moved into _claim_candidates, which does one stat for + # the whole manifest instead of one per path (1.14.2) -- read both + ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src" + sed -n '/^_claim_candidates() {/,/^}/p' "$helper_src")" + # every naming shape the same package takes across the book + # the stages are built from a loop now, so check the suffixes it appends + case "$ce" in + *'"$stem-tmp" "$stem-pass1" "$stem-pass2"'*) + ok "the -tmp and -passN stages are all claimed" ;; + *) bad "an earlier stage suffix is missing" ;; + esac + # ONLY what the build left behind -- never another package's file + case "$ce" in + *'root|lfs|UNKNOWN) ;;'*|*'u != "root" && u != "lfs" && u != "UNKNOWN"'*) + ok "only root- and lfs-owned paths are claimed" ;; + *) bad "it could take a file belonging to another package" ;; + esac + # directories too, or the package cannot create files in them + case "$ce" in + *'.dirs'*) ok "the directories are claimed as well as the files" ;; + *) bad "the package could not create files in its own directories" ;; + esac + # shared install dirs stay shared + case "$ce" in + *"is_install_dir"*) ok "shared install directories are left alone" ;; + *) bad "a shared directory could be taken by one package" ;; + esac + # and it must run BEFORE the build, not after it fails + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + i_claim="$(printf '%s' "$cb" | grep -n '_claim_earlier_stages' | head -1 | cut -d: -f1)" + i_build="$(printf '%s' "$cb" | grep -n 'building \$name' | head -1 | cut -d: -f1)" + if [ -n "$i_claim" ] && [ -n "$i_build" ] && [ "$i_claim" -lt "$i_build" ]; then + ok "the claim happens before the build starts" else - bad "a symlinked install dir is still reported" + bad "the claim runs too late to help" fi - # and the guard must come BEFORE the stat, or it reports then skips - if sed -n '/install directories not in the expected state/,-12p' \ - "$helper_src" >/dev/null 2>&1; then :; fi - _blk="$(grep -A 4 '\[ -d "\$live" \] || continue' "$helper_src" | head -20)" - case "$_blk" in - *'-L "$live"'*) ok "the symlink guard runs before stat" ;; - *) bad "stat runs before the symlink guard" ;; - esac fi -# ---- verify must write every finding to a file ----------------------------- # -# The console prints the first eight of each kind and then -# ... and 44265 more -# which is a number and nothing you can act on: it cannot be grepped, diffed -# against the previous run, or fed to which-package. Every finding goes to a -# log, tagged by kind, while the console stays short. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - grep -q '^VERIFY_LOG=' "$helper_src" || _p="$_p;verify has no log file" - grep -q '_vfy_log_open' "$helper_src" || _p="$_p;the log is never opened" - # every truncated list must also be written in full - for _k in wrong-owner unclaimed installdir; do - grep -q "$_k" "$helper_src" || _p="$_p;nothing logs '$_k' findings" - done - # and the summary must say where it is, or nobody finds it - grep -q 'full report: \$VERIFY_LOG' "$helper_src" \ - || _p="$_p;the summary never names the log" - # a read-only state dir must not abort verify -- the report still matters - grep -q 'VERIFY_LOG=""; return 0' "$helper_src" \ - || _p="$_p;verify dies instead of running when the log cannot be written" +# ---- checks that silently always passed --------------------------------------# +# From an independent review of `lfs` against the book. Each of these hid a +# real failure rather than reporting it. +python3 - "$LFS_TOOL" <<'PY73' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +src = open(sys.argv[1]).read() +problems = [] - _o="$( STATE=/tmp/_vst; SNAP_ROOT=/; VERIFY_LOG=/tmp/_vst/verify.log - rm -rf /tmp/_vst - eval "$(sed -n '/^_vfy_log_open() {/,/^}/p' "$helper_src")" - eval "$(sed -n '/^_vfy_log() {/,/^}/p' "$helper_src")" - _vfy_log_open - _vfy_log wrong-owner "/usr/bin/python3 is root want p_python" - _vfy_log unclaimed "/usr/bin/lfs-helper" - [ -s /tmp/_vst/verify.log ] || echo "the log is empty after findings" - grep -q '^wrong-owner' /tmp/_vst/verify.log || echo "wrong-owner not tagged" - grep -q '^unclaimed' /tmp/_vst/verify.log || echo "unclaimed not tagged" - # a fresh run must not append to the last one's findings - _vfy_log_open - grep -q '^wrong-owner' /tmp/_vst/verify.log && echo "the log is not reset per run" - # an unwritable location must be survivable - VERIFY_LOG=/dev/null/nope/verify.log - _vfy_log_open >/dev/null 2>&1 || echo "an unwritable log aborts verify" )" - [ -n "$_o" ] && _p="$_p;$(printf '%s' "$_o" | tr '\n' ';')" +# md5 verification: `md5sum -c … | grep -v ": OK"` returned 0 on a CORRUPTED +# tree -- grep inverts the verdict and the pipe discards md5sum's status. +i = src.index("verifying md5sums") +seg = src[i:i + 900] +# ignore comment lines: the fix is documented with the broken command quoted +code = "\n".join(l for l in seg.splitlines() if not l.lstrip().startswith("#")) +if 'grep -v ": OK"' in code: + problems.append("md5 verification still inverts its own verdict") +if "md5sum -c md5sums --quiet" not in seg or "sys.exit(1)" not in seg: + problems.append("a bad md5sum does not fail the download") - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "verify writes every finding to a log, tagged by kind" - fi -fi +# The book's toolchain gate must stay a gate. These five greps MUST produce +# output -- no match means the build is linked against the host -- and the +# blanket `|| true` turned the only real check into a guaranteed pass. +gates = [ + 'grep -E -o "/usr/lib.*/S?crt[1in].*succeeded" dummy.log', + 'grep -B3 "^ /usr/include" dummy.log', + "grep 'SEARCH.*/usr/lib' dummy.log", + 'grep "/lib.*/libc.so.6 " dummy.log', + 'grep found dummy.log', +] +out = lfs._soften_diagnostics("\n".join(gates)) +for line in out.splitlines(): + if "|| true" in line: + problems.append("a toolchain gate is still softened: %s" % line[:40]) +# but an ordinary diagnostic grep must STILL be softened +if "|| true" not in lfs._soften_diagnostics("grep something optional.txt"): + problems.append("ordinary diagnostic greps are no longer softened") -# ---- the built system gets the whole toolchain, and it has an owner -------- # -# Only lfs-helper was copied into the tree, because only lfs-helper runs during -# the build. But the built system needs the rest -- packagemanager cannot be -# installed BY packagemanager -- and a system that boots without them has no -# way to install anything at all. They are a few files and they refresh on -# every `lfs run`. -# -# They are also software installed into /usr/bin, so they get a package user -# like anything else. Without one they are reported forever as -# 1516 root-owned file(s) that NO manifest claims: -# /usr/bin/lfs-helper -# which is true, unhelpful, and buries the cases that matter. -python3 - "$LFS_TOOL" <<'PYTOOLS' -import sys, os, tempfile, importlib.machinery as m -lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() -problems = [] +# 9.9's `cat > /etc/shells` block is <pre class="root">, not "userinput", so +# cfg_shells resolved to zero commands and /etc/shells was never written. +pi = src.index("classes = tag.get(\"class\") or []") +if '"root", "install"' not in src[pi:pi + 700]: + problems.append("only 'userinput' blocks are parsed; /etc/shells is missed") +# the accepted-class tuple itself must not include "screen" (output samples) +accept = src[src.index("any(c in classes for c in ("):][:120] +if "screen" in accept: + problems.append("'screen' blocks (output samples) would be run as commands") -for t in ("lfs-helper", "packagemanager", "packagemanager_install", "blfs"): - if t not in getattr(lfs, "TOOLCHAIN_SCRIPTS", ()): - problems.append("%s is not copied into the tree" % t) +# a step that produces nothing must say so +if "has no command blocks in this book" not in src: + problems.append("a step resolving to zero blocks is still skipped silently") -# the owner is a package user like any other, so it carries the prefix -if lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER != "p_pkgusr": - problems.append("the tools' owner is %r, not p_pkgusr" - % (lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER)) +# book 4.4's ~/.bashrc opens with these; without them the build runs with +# bash's command hash on and root's umask +env = lfs._lfs_env_prefix("/mnt/lfs") +if not env.startswith("set +h\numask 022\n"): + problems.append("the build environment is missing set +h / umask 022") -# the uid must be read from the TREE's passwd, never this machine's -- a host -# uid means nothing inside the chroot, the same mistake book 3.1 warns about -d = tempfile.mkdtemp() -os.makedirs(os.path.join(d, "etc")) -if lfs._uid_in_tree(d, "p_pkgusr") is not None: - problems.append("a uid is invented when the tree has no passwd file") -with open(os.path.join(d, "etc", "passwd"), "w") as f: - f.write("root:x:0:0::/root:/bin/bash\np_pkgusr:x:10042:10042::/usr/src:/bin/bash\n") -if lfs._uid_in_tree(d, "p_pkgusr") != 10042: - problems.append("the tree's own passwd file is not used") -if lfs._uid_in_tree(d, "nosuchuser") is not None: - problems.append("an unknown user resolves to something") +# ^ anchored to the whole string, so a later line never matched +if not (lfs._TEST_DIAG_RE.flags & 8): + problems.append("_TEST_DIAG_RE has no re.M; ^ only matches the first line") + +# dead duplicate flagged by pyflakes +if src.count("\ndef _pkg_glob_for") != 1: + problems.append("_pkg_glob_for is defined more than once") if problems: for p in problems: print(" FAIL %s" % p) sys.exit(1) -print(" PASS the whole toolchain is installed and owned by p_pkgusr") -PYTOOLS +print(" PASS the checks that always passed now actually check") +PY73 _count_rc $? -# lfs-helper claims them once /etc/passwd is real -- lfs cannot, because on a -# fresh tree the account does not exist until book 7.6 +# ---- a staged install must not take the shared install directories ---------- # +# The staging tree is created by the package user, so $stage/usr, $stage/usr/bin +# and so on are owned by that user with mode 755. Stamping that onto the live +# tree handed /usr and /usr/bin to whichever package was building, losing the +# install group and g+w -- the whole install-directory scheme, undone by one +# package. Verified: root:install 775 became gcc:gcc 755. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _p="" - grep -q '_adopt_pkgusr_tools' "$helper_src" || _p="$_p;the tools are never claimed" - grep -q 'PKGUSR_TOOLS=' "$helper_src" || _p="$_p;there is no list of tools to claim" - # it must run in build-all, beside the other adoption pass - case "$(_slice_fn "$helper_src" establish_ownership)" in - *_adopt_pkgusr_tools*) ;; - *) _p="$_p;the tools are not claimed when ownership is established" ;; - esac - # and it must never take another package's file - sed -n '/^_adopt_pkgusr_tools() {/,/^}/p' "$helper_src" \ - | grep -q 'user_exists "\$cur" && continue' \ - || _p="$_p;claiming the tools could steal another package's file" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done + it="$(sed -n '/^_install_staged_tree() {/,/^}/p' "$helper_src")" + case "$it" in + *"ALREADY THERE -- leave its ownership and mode alone"*) + ok "an existing directory keeps its owner and mode" ;; + *) bad "a staged install still restamps existing directories" ;; + esac + + # behaviour: existing dirs untouched, new ones owned by the package + sd="$T/stageown"; mkdir -p "$sd/usr/bin" "$sd/stage/usr/bin" "$sd/stage/usr/lib/fresh" + chmod 775 "$sd/usr" "$sd/usr/bin" + before_usr="$(stat -c %a "$sd/usr")" + echo p > "$sd/stage/usr/bin/p"; echo l > "$sd/stage/usr/lib/fresh/l" + chmod -R 755 "$sd/stage" + ( SNAP_ROOT="$sd"; say() { :; }; warn() { :; } + eval "$it"; _install_staged_tree "$sd/stage" ) >/dev/null 2>&1 + if [ "$(stat -c %a "$sd/usr")" = "$before_usr" ] \ + && [ -f "$sd/usr/bin/p" ] && [ -f "$sd/usr/lib/fresh/l" ]; then + ok "existing modes survive and the files still install" else - ok "the tools are claimed once the user database exists" + bad "the staged merge changed an existing directory's mode" fi fi -# ---- an ADOPTED package gets a real home, like a built one ----------------- # -# Packages built in a run owned their files and had full homes; packages -# ADOPTED from chapter 5-6 manifests in the same run did not: -# drwxr-xr-x 1 p_bison p_bison 148 p_bison <- built -# drwxr-xr-x 1 root root 86 p_gcc <- adopted -# 86 vs 148 bytes: the adopted homes were missing the profile links and had -# never been chowned. cmd_add_user was called with BOTH streams suppressed, -# so whatever went wrong said nothing. Adoption now initialises the home -# explicitly (init_package_user_home is idempotent) and stderr is let through. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +# ---- book 7.6 truncates the user database ----------------------------------- # +# `cat > /etc/passwd` is right the first time and catastrophic the second: it +# wipes every package user, private group, the install group, every collector +# group and every membership. The files on disk then all read as UNKNOWN. if [ -f "$helper_src" ]; then - _blk="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" - _p="" - case "$_blk" in - *'init_package_user_home "$owner"'*) ;; - *) _p="$_p;an adopted package's home is never initialised" ;; - esac - case "$_blk" in - *'cmd_add_user "$owner" >/dev/null 2>&1'*) - _p="$_p;adoption still hides why a user could not be created" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "an adopted package gets an initialised, owned home" - fi + grep -q "^_protect_user_db()" "$helper_src" \ + && ok "the user database is protected across 7.6" \ + || bad "re-running init-files would wipe every package user" + grep -q "^_step_rewrites_user_db()" "$helper_src" \ + && ok "steps that rewrite passwd/group are detected" \ + || bad "the protection is not tied to the steps that need it" + # the detector must not fire on scripts that merely READ the files + dt="$(sed -n '/^_step_rewrites_user_db() {/,/^}/p' "$helper_src")" + rd="$T/dbdetect"; mkdir -p "$rd" + printf 'grep root /etc/passwd\n' > "$rd/read.sh" + printf 'cat > /etc/passwd << "EOF"\nroot:x:0:0::/root:/bin/bash\nEOF\n' > "$rd/write.sh" + ( eval "$dt" + _step_rewrites_user_db "$rd/write.sh" && ! _step_rewrites_user_db "$rd/read.sh" ) \ + && ok "it fires on a rewrite, not on a read" \ + || bad "the user-database detector is wrong" + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + case "$cb" in + *_protect_user_db*_restore_user_db*) + ok "entries are saved before the step and put back after" ;; + *) bad "the protection is never applied around the build" ;; + esac fi -# ---- one account, one prefix ----------------------------------------------- # -# A config step used to be `p_cfg_bootscripts`: pkg_owner_name applied the -# PACKAGE prefix to every name regardless of kind, so the account wore its own -# `cfg_` and `p_` on top of it. Neither prefix then identified the account -- -# `p_` said "package" while it lived in /usr/src/cfg -- and every rule that -# reads a prefix had to special-case the stacked form. -# -# The fix is that kind is decided ONCE (pkgusr_kind) and the prefix and the -# root are both answered from it, so a name cannot be given one kind's prefix -# and placed in another kind's root. +# ---- two accounts sharing one uid ------------------------------------------- # +# Ownership is stored as a NUMBER. Give two names the same uid and every tool +# resolves it to whichever name comes first -- ncurses' binaries reported as +# /usr/bin/tic -rwxr-xr-x xz:xz +# and the ncurses build could not overwrite its own files. It reads exactly +# like a permissions bug and is not one. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _o="$( PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" - PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg - for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ - unprefix_pkg_user pkg_owner_name pkgusr_home_for; do - eval "$(_slice_fn "$helper_src" "$fn")" - done - [ "$(pkg_owner_name cfg_bootscripts)" = "cfg_bootscripts" ] \ - || echo "a config step still gets the package prefix: $(pkg_owner_name cfg_bootscripts)" - [ "$(pkg_owner_name gcc)" = "p_gcc" ] \ - || echo "a package lost its prefix: $(pkg_owner_name gcc)" - # kind decides both answers, so they cannot disagree - [ "$(pkgusr_kind cfg_nsswitch)" = "cfg" ] || echo "a cfg step is not kind cfg" - [ "$(pkgusr_kind gcc)" = "pkg" ] || echo "a package is not kind pkg" - # a tree or a config written before the split holds stacked names; they - # are repaired on the way through rather than carried forward - [ "$(pkg_owner_name p_cfg_bootscripts)" = "cfg_bootscripts" ] \ - || echo "a stacked legacy name survives: $(pkg_owner_name p_cfg_bootscripts)" - # and the whole thing is still idempotent -- that property is what lets - # every call site apply it without knowing which kind of name it holds - for _n in gcc p_gcc cfg_bootscripts p_cfg_bootscripts; do - _a="$(pkg_owner_name "$_n")"; _b="$(pkg_owner_name "$_a")" - [ "$_a" = "$_b" ] || echo "not idempotent: $_n -> $_a -> $_b" - done )" - if [ -z "$_o" ]; then - ok "each kind of account wears exactly one prefix, its own" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi -fi + grep -q "^_check_duplicate_ids()" "$helper_src" \ + && ok "duplicate ids are detected and named" \ + || bad "a uid collision would look like a permissions bug forever" + ba="$(sed -n '/^cmd_build_all() {/,/^}/p' "$helper_src")" + case "$ba" in + *_check_duplicate_ids*) ok "the check runs before anything is built" ;; + *) bad "the collision check never runs" ;; + esac + # restoring entries after book 7.6 must not CREATE a collision + rd="$(sed -n '/^_restore_user_db() {/,/^}/p' "$helper_src")" + case "$rd" in + *'is now used by'*) + ok "an entry whose id was reassigned is not restored" ;; + *) bad "restoring after 7.6 could create the collision itself" ;; + esac -# packagemanager must reach the same answer, or the built system looks for -# accounts the build never created -pm_src="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pm_src" ]; then - python3 - "$pm_src" <<'PYONEPFX' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -ns = {"os": __import__("os"), "BASE_DIR": "/usr/src", - "PKGUSR_SUBDIR": "pkgusr", "CFGUSR_SUBDIR": "cfg", - "_DEFAULT_CONFIG": {"collector_prefix": "sysgroup", "user_prefix": "u", - "pkgusr_prefix": "p", "cfguser_prefix": "cfg", - "main_user": ""}} -ns["load_config"] = lambda: dict(ns["_DEFAULT_CONFIG"]) -for fn in ("collector_prefix", "user_prefix", "pkgusr_prefix", "cfguser_prefix", - "pkgusr_kind", "pkgusr_name", "unprefix_pkgusr", "pkgusr_root", - "pkgusr_home"): - m = re.search(r"^def %s\(.*?(?=\n\ndef |\n\n# |\nPKGUSR_SUBDIR)" % fn, src, re.S | re.M) - if not m: - problems.append("packagemanager has no %s" % fn) - continue - exec(m.group(0), ns) -if not problems: - want = { - # name -> (account name, home) - "gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), - "p_gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), - "cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), - # the stacked legacy form is repaired, not carried forward - "p_cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), - "u_firefox": ("u_firefox", "/usr/src/u_firefox"), - } - for given, (name, home) in want.items(): - got = ns["pkgusr_name"](given) - if got != name: - problems.append("packagemanager names %s as %s, want %s" % (given, got, name)) - got = ns["pkgusr_home"](given) - if got != home: - problems.append("packagemanager puts %s at %s, want %s" % (given, got, home)) - again = ns["pkgusr_name"](ns["pkgusr_name"](given)) - if again != ns["pkgusr_name"](given): - problems.append("packagemanager is not idempotent on %s" % given) -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS packagemanager and lfs-helper name each kind identically") -PYONEPFX - _count_rc $? + # behaviour: the detector finds a real collision and ignores a clean file + dd="$T/dupids"; mkdir -p "$dd" + printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\nncurses:x:10005:10005::/n:/bin/bash\n' > "$dd/bad" + printf 'root:x:0:0::/root:/bin/bash\nxz:x:10005:10005::/x:/bin/bash\n' > "$dd/good" + du="$(sed -n '/^_duplicate_ids() {/,/^}/p' "$helper_src")" + ( eval "$du" + [ -n "$(_duplicate_ids "$dd/bad")" ] && [ -z "$(_duplicate_ids "$dd/good")" ] ) \ + && ok "it finds a real collision and stays quiet on a clean file" \ + || bad "the duplicate-id detector is wrong" fi -# ---- a step that is not a package leaves no home behind --------------------- # -# cmd_build did `mkdir -p "$(pkgusr_home_for "$name")"` for EVERY root step, -# including the ones _is_not_a_package will never give an account. A sanity run -# then found /usr/src/pkgusr/p_init-dirs and p_init-files: directories with no -# matching account, which every ownership pass has to report as strays. -# -# init-*, last-step and refind own no files, so they are staged and run out of -# the state directory instead -- beside the scripts they came from. +# ---- one reconcile pass instead of eight repair commands -------------------- # +# fix-ownership, adopt-dirs, adopt-existing, fix-orphans, fix-users, +# sort-users, seal-install-dirs and claim-earlier-stages each fixed ONE symptom +# and could not see a problem outside its own remit -- so a staged install +# quietly handing /usr to a package user went unnoticed through four later +# failures. Ownership is derived state: the manifests, the install-dir list +# and the collector map fully determine what the tree should be. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _o="$( STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" - PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" - PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg - for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ - pkg_owner_name pkgusr_home_for _is_not_a_package step_stage_dir; do - eval "$(_slice_fn "$helper_src" "$fn")" - done - # an empty answer must not read as "not under an account root" - command -v step_stage_dir >/dev/null 2>&1 \ - || echo "there is no step_stage_dir -- every root step still gets a home" - for _s in init-dirs init-files last-step refind; do - _d="$(step_stage_dir "$_s" 2>/dev/null)" - [ -n "$_d" ] || { echo "$_s resolves to no directory at all"; continue; } - case "$_d" in - "$PKGUSR_ROOT"/*|"$CFGUSR_ROOT"/*) - echo "$_s still gets a home under an account root" ;; - esac - done - # a real package is unaffected: it still runs from its user's home - [ "$(step_stage_dir gcc)" = "$PKGUSR_ROOT/p_gcc" ] \ - || echo "a package no longer runs from its own home: $(step_stage_dir gcc)" )" - if [ -z "$_o" ]; then - ok "a step with no account leaves no directory under the account roots" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi - - # the staged path, the mkdir, the cd and the failure banner must be ONE - # value. They were four separate calls to pkgusr_home_for, free to disagree. - _cb="$(_slice_fn "$helper_src" cmd_build)" - case "$_cb" in - *'cd "$(pkgusr_home_for'*) - bad "the build still re-derives the directory it cd's into" ;; - *) ok "the staged script, the cd and the banner share one path" ;; - esac -fi - -# ---- the account roots are a decision, not a side effect -------------------- # -# /usr/src/pkgusr and /usr/src/cfg came out root:root 0755 because `mkdir -p` -# on the first home created them with root's umask. Nothing said so, and -# nothing would have noticed it changing. -# -# They carry the same root:install 1775 as /usr/src above them -- one rule for -# the subtree instead of three directories with two rules. The STICKY BIT is -# what makes group-writable safe and is not optional: without it any member of -# `install` could delete or rename another package's home. -# -# They are still NOT install dirs. Being in never_adopt_list is the other -# half: the mode is shared, but no package may ever take one as its own. + grep -q "^cmd_verify()" "$helper_src" \ + && ok "one command reconciles the tree" \ + || bad "there is no single check of the whole model" + # The commands it replaces must not be SECOND IMPLEMENTATIONS -- but they + # must still be callable. Deleting them outright broke every message that + # tells people to run them: + # !! unknown command 'adopt-existing' + for c in fix-ownership fix-orphans fix-users adopt-existing; do + grep -qE "^\s+.*\b$c[|)]" "$helper_src" \ + || bad "$c is referenced in the hints but not dispatched" + done + ok "the familiar repair names still work" + # ... and route to verify rather than repeating its logic + disp="$(sed -n '/fix-ownership|fix-orphans/,/cmd_verify/p' "$helper_src")" + case "$disp" in + *cmd_verify*) ok "they are aliases for verify, not second copies" ;; + *) bad "a repair command has its own implementation again" ;; + esac + + # every rule the model needs + for r in _vfy_manifest_ownership _vfy_install_dirs _vfy_orphans \ + _vfy_duplicate_ids _vfy_user_order; do + grep -q "^$r()" "$helper_src" || bad "verify has no $r rule" + done + ok "verify checks ownership, install dirs, orphans, ids and order" + + # reporting by default, repairing only when asked + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + case "$cv" in + *'"${1:-}" = "--fix" ] && fix=1'*|*"--fix|--run) fix=1"*) + ok "verify reports unless asked to repair" ;; + *) bad "verify repairs without being asked" ;; + esac + # manifest ownership must settle before orphans, or a file whose uid no + # longer resolves is handed to 'lfs' instead of to its package + i_own="$(printf '%s' "$cv" | grep -n '_vfy_manifest_ownership' | head -1 | cut -d: -f1)" + i_orp="$(printf '%s' "$cv" | grep -n '_vfy_orphans' | head -1 | cut -d: -f1)" + if [ -n "$i_own" ] && [ -n "$i_orp" ] && [ "$i_own" -lt "$i_orp" ]; then + ok "manifest ownership is settled before orphans are claimed" + else + bad "an orphan pass could take a file that belongs to a package" + fi + + # A1 end to end: the install dirs are checked, so a staged install taking + # /usr is caught rather than surfacing four failures later + vd="$T/verifydirs"; mkdir -p "$vd/usr/bin" "$vd/.lfs-pkgusr/manifests" + chmod 755 "$vd/usr" "$vd/usr/bin" + id_out="$( ( SNAP_ROOT="$vd"; ETC="$vd/etc"; INSTALLDIRS="" + MANIFESTS="$vd/.lfs-pkgusr/manifests" + say() { echo "$*"; }; warn() { echo "$*"; } + _vfy_n_checked=0; _vfy_n_wrong=0; _vfy_n_fixed=0 + INSTALL_GID=9999 + for _f in install_dirs_list _install_dirs_are_sealed _vfy_report \ + set_install_dir_owner install_dir_owner_ok _vfy_install_dirs; do + eval "$(_slice_fn "$helper_src" "$_f")" + done + # the pass is gated on the ownership epoch now: before book 7.6 there + # is no user database, so `install` is not a name and reporting 27 + # wrong directories would be noise. This fixture is past that. + ownership_established() { return 0; } + ownership_possible() { return 0; } + _vfy_install_dirs 0 ) 2>&1 )" + case "$id_out" in + *"root:install"*) ok "a wrongly-owned install directory is reported" ;; + *) bad "verify does not notice install dirs losing the install group" ;; + esac +fi + +# ---- ownership is derived state, so it can be checked -------------------------# +# The manifests say who installed what, the install-dir list says what is +# shared, the collector map says which groups govern which directories. +# Together they determine what the tree SHOULD look like -- so a discrepancy +# can be found before it becomes a failed build, instead of being chased one +# symptom at a time through fix-ownership, adopt-dirs, fix-perms, fix-orphans +# and claim-earlier-stages. helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then - _p="" - _ep="$(_slice_fn "$helper_src" ensure_pkgusr_roots)" - case "$_ep" in - *'chmod 1775'*) ;; - *) _p="$_p;the account roots have no stated mode" ;; - esac - case "$_ep" in - *set_install_dir_owner*) ;; - *) _p="$_p;the account roots do not get the install group" ;; - esac - # sticky, or group-write means any package can delete another's home - case "$_ep" in - *17[0-9][0-9]*) ;; - *) _p="$_p;the account roots are group-writable WITHOUT the sticky bit" ;; - esac - case "$(_slice_fn "$helper_src" init_package_user_home)" in - *ensure_pkgusr_roots*) ;; - *) _p="$_p;the account roots are still created as a side effect" ;; - esac - # never_adopt_list must reach them through pkgusr_roots -- a second literal - # list is one more place to forget when a root is added - case "$(_slice_fn "$helper_src" never_adopt_list)" in - *pkgusr_roots*) ;; - *) _p="$_p;never_adopt_list does not cover the account roots" ;; - esac - # and they must NOT be install dirs - if _slice_fn "$helper_src" install_dirs_list | grep -qx '/usr/src/pkgusr'; then - _p="$_p;the package root is an install dir -- any package user could delete a home" - fi - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the account roots are created deliberately and never adopted" - fi + grep -q "^cmd_verify()" "$helper_src" \ + && ok "the tree can be checked against the manifests" \ + || bad "there is no way to ask whether ownership is correct" + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + # the A1 case: an install directory taken by a package user + # The wording lives in the _vfy_* helpers now -- cmd_verify calls them + # rather than carrying its own copy of each rule. Assert the check runs. + case "$cv" in + *_vfy_install_dirs*) + ok "it catches an install directory taken by a package" ;; + *) bad "the install-directory scheme is not checked" ;; + esac + _vfy_install_dirs_body="$(_slice_fn "$helper_src" _vfy_install_dirs)" + case "$_vfy_install_dirs_body" in + *'"root:install $want_mode"'*) ;; + *) bad "the install-dir check no longer says what it wants" ;; + esac + case "$cv" in + *_vfy_manifest_ownership*) + ok "it catches files owned by the wrong package" ;; + *) bad "mis-owned files are not detected" ;; + esac + case "$cv" in + *_check_duplicate_ids*) ok "it checks for id collisions too" ;; + *) bad "a uid collision would not be reported" ;; + esac + # report by default, repair only when asked. (1.13.30 parses all the + # arguments so a package name can be given too, so check the BEHAVIOUR: + # fix starts at 0 and only --fix/--run turns it on.) + case "$cv" in + *'local fix=0'*'--fix|--run) fix=1'*) + ok "it reports by default and repairs only with --fix" ;; + *) bad "verify changes things without being asked" ;; + esac + # a collision is NOT auto-repairable -- which name owns what is a decision + case "$cv" in + *"verify cannot repair this"*) + ok "it does not guess at an id collision" ;; + *) bad "verify would silently reassign ids" ;; + esac +fi + +# ---- durable records and scratch must be distinguishable ---------------------# +# $STATE mixes records that ARE the ownership (manifests, collector map) with +# pure scratch (staging trees, logs). A snapshot or a restart had to guess. +if [ -f "$helper_src" ]; then + grep -q "^STATE_DURABLE=" "$helper_src" \ + && ok "the durable records are named" \ + || bad "nothing says which state must survive" + grep -q "^STATE_SCRATCH=" "$helper_src" \ + && ok "the scratch is named" \ + || bad "nothing says which state is disposable" + grep -q "manifests" <<< "$(sed -n '/^STATE_DURABLE=/,/^STATE_SCRATCH=/p' "$helper_src")" \ + && ok "the manifests count as durable" \ + || bad "the manifests could be cleaned away" + grep -q "^STATE_FORMAT=" "$helper_src" \ + && ok "the state format is versioned" \ + || bad "an incompatible state directory would be misread" + cs="$(sed -n '/^cmd_clean_state() {/,/^}/p' "$helper_src")" + case "$cs" in + *'rm -rf "${STATE:?}/$d"'*) + ok "cleaning scratch cannot expand to rm -rf /" ;; + *) bad "an unset STATE would delete the root" ;; + esac +fi + +# ---- the suite's own count must be trustworthy ------------------------------ # +# ok/bad used shell variables, and several checks call them from inside a +# subshell -- `( ... ) && ok || bad`, a pipeline, a `while read` fed by process +# substitution. An increment there dies with the subshell, so the suite +# printed 446 PASS lines and reported 433. A wrong count is worse than none: +# it hides a regression behind a number that still looks healthy. +cnt="$T/countcheck"; mkdir -p "$cnt" +( _COUNTDIR="$cnt"; : > "$cnt/pass"; : > "$cnt/fail" + ok() { echo x >> "$_COUNTDIR/pass"; } + # the case that used to be lost: incremented inside a subshell + ( ok ) ; ( true && ok ) ; echo | while read -r _; do ok; done ) +n="$(wc -l < "$cnt/pass" | tr -d ' ')" +[ "$n" = 3 ] \ + && ok "counts survive subshells, pipelines and process substitution" \ + || bad "the counter loses $((3 - n)) of 3 increments from subshells" + +# and every assertion must be counted exactly once +_asserts="$(grep -cE '^\s*(ok|bad) "' "$0" 2>/dev/null || echo 0)" +[ "$_asserts" -gt 0 ] \ + && ok "the suite counts assertions, not printed lines" \ + || bad "the suite has no assertions to count" + +# ---- host layout and built-system layout are separate ------------------------ # +# Configuring where package users live on THIS machine must not change where +# they live on every system built from it. The whole point of the build is +# that the new system is not this one. +python3 - "$LFS_TOOL" <<'PY74' +import sys, os, json, tempfile, importlib.machinery as m +store = tempfile.mkdtemp(); os.environ["LFS_STORE"] = store +os.makedirs(os.path.join(store, "books"), exist_ok=True) +json.dump({"default": "12.4", + "pkgusr_home": "/usr/src", + "target_pkgusr_home": "/usr/src/package_users", + "target_appuser_home": "/home/shared_users"}, + open(os.path.join(store, "config.json"), "w")) +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "layout"): + print(" FAIL the layout is not configurable"); sys.exit(1) +if lfs.layout("pkgusr_home") != "/usr/src": + problems.append("the host layout was changed by a target setting") +if lfs.layout("pkgusr_home", target=True) != "/usr/src/package_users": + problems.append("the built system does not use its own layout") +if lfs.layout("appuser_home", target=True) != "/home/shared_users": + problems.append("application users do not get their own base") +# The default moved deliberately: accounts are grouped by kind, so a +# directory listing says what a thing IS. Existing trees are NOT migrated -- +# a fresh build is required, and migrate-layout is still unwritten. Pin the +# new values so the next move is also a decision and not an accident. +d = lfs.LAYOUT_DEFAULTS +if d["target_pkgusr_home"] != "/usr/src/pkgusr" \ + or d["pkgusr_home"] != "/usr/src/pkgusr": + problems.append("packages no longer default to /usr/src/pkgusr") +if d["target_cfguser_home"] != "/usr/src/cfg" \ + or d["cfguser_home"] != "/usr/src/cfg": + problems.append("config steps no longer default to /usr/src/cfg") +# and the target value must reach the chroot and the built system's config +src = open(sys.argv[1]).read() +if 'LFS_SRC_ROOT="{layout("pkgusr_home", target=True)}"' not in src: + problems.append("the chroot build environment does not carry the layout") +if 'pkgusr_home=%s' not in src: + problems.append("the built system's config does not record the layout") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS host and built-system layouts are configured independently") +PY74 +_count_rc $? + +pmt="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pmt" ]; then + grep -q "^def appuser_home" "$pmt" \ + && ok "application users have their own base directory" \ + || bad "application users still share the package-user base" + # an unconfigured system must keep the old layout + lyd="$T/layout"; mkdir -p "$lyd" + printf 'collector_prefix=n\n' > "$lyd/pm.conf" + got="$(PKGUSR_CONFIG="$lyd/pm.conf" python3 -c " +import importlib.machinery as m, os +pm = m.SourceFileLoader('pm', '$pmt').load_module() +print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" + [ "$got" = "/usr/src /usr/src" ] \ + && ok "an unconfigured system keeps the existing layout" \ + || bad "the layout changed without being asked (got '$got')" + printf 'collector_prefix=n\npkgusr_home=/usr/src/package_users\nappuser_home=/home/shared_users\n' \ + > "$lyd/pm2.conf" + got2="$(PKGUSR_CONFIG="$lyd/pm2.conf" python3 -c " +import importlib.machinery as m, os +pm = m.SourceFileLoader('pm', '$pmt').load_module() +print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" + [ "$got2" = "/usr/src/package_users /home/shared_users" ] \ + && ok "a configured system uses the layout it was built with" \ + || bad "the configured layout is ignored (got '$got2')" +fi + +# ---- /root must never carry the install group -------------------------------- # +# Two different ideas got conflated. An install directory is SHARED: it gets +# the install group and g+w so any package user can add files. /root, /home +# and the scratch dirs are something else -- they must never be adopted by a +# package, but they are not shared. Listing them as install dirs produced +# drwxrwx--- 1 root install /root +# handing every package user access to root's home. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^never_adopt_list()" "$helper_src" \ + && ok "protected-from-adoption is separate from shared" \ + || bad "the two ideas are still conflated" + ( SNAP_ROOT=/; INSTALLDIRS="" + eval "$(sed -n '/^never_adopt_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_never_adopt() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^install_dirs_list() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^install_dir_patterns() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^matches_install_pattern() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^is_install_dir() {/,/^}/p' "$helper_src")" + # never group-writable + for d in /root /home /tmp /var/tmp; do + install_dirs_list | grep -qx "$d" && exit 1 + done + # but still never adopted + for d in /root /home /tmp /var/tmp; do is_install_dir "$d" || exit 1; done + # and the real install dirs are untouched + install_dirs_list | grep -qx /usr/bin || exit 1 + exit 0 ) \ + && ok "/root and /home are protected but not group-writable" \ + || bad "/root or /home would get the install group" +fi + +# ---- nothing owned by the build user survives into the system --------------- # +# Book 7.2 hands over the directories 4.2 created, but the tree ROOT and the +# mount points are not in that list, so /, /dev, /proc, /sys, /run, /media, +# /opt and /tools stayed owned by a user that does not exist in the chroot: +# drwxr-xr-x 1 lfs lfs 178 / +python3 - "$LFS_TOOL" <<'PY75' +import sys +src = open(sys.argv[1]).read() +if "give the top level to root" not in src: + print(" FAIL the tree root and mount points keep the build user") + sys.exit(1) +i = src.index("give the top level to root") +seg = src[i:i + 500] +problems = [] +if "/sources" not in seg: + problems.append("/sources is not excluded; the book keeps it with lfs") +if "-R " in seg: + problems.append("the handover is recursive; it must only touch the top level") +if 'chown -h root:root' not in seg: + problems.append("symlinks at the top level are not handed over") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the tree root and mount points are handed back to root") +PY75 +_count_rc $? + +# ---- the package-user model must be usable from outside ---------------------- # +# `build` only works for steps that already exist in the step list with a +# generated script, so a BLFS package or a local script had no way to say +# "install this as its own package user and track it" -- it would have to +# reimplement user creation, the wrappers, staging, tracking and the permission +# grants. That is exactly how the two halves of this toolchain drifted apart. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_install_as()" "$helper_src" \ + && ok "anything can be installed as its own package user" \ + || bad "third-party installs cannot use the model" + ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" + # both shapes a caller might have + case "$ia" in + *"--script)"*) ok "an existing install script can be used" ;; + *) bad "only inline commands are accepted" ;; + esac + case "$ia" in + *"--) shift; cmdv=("*) ok "a bare command can be used" ;; + *) bad "only a script file is accepted" ;; + esac + # it must REUSE cmd_build, not reimplement it -- one code path or they drift + case "$ia" in + *'cmd_build "$name" --force'*) + ok "it reuses the build path rather than duplicating it" ;; + *) bad "install-as reimplements what build already does" ;; + esac + # the result must be a normal package, visible to every query + case "$ia" in + *'"$STATE_PROGRESS/steporder"'*) + ok "the package becomes visible to status, list and manifests" ;; + *) bad "an install-as package would be invisible to the other commands" ;; + esac + # reserved names must be refused + case "$ia" in + *"_is_not_a_package"*) ok "reserved step names are refused" ;; + *) bad "install-as could create a user for last-step or cfg_*" ;; + esac + # a command with spaces or quotes must survive being written to a script + grep -q "^_quote_cmd()" "$helper_src" \ + && ok "arguments are quoted, so paths with spaces survive" \ + || bad "a command containing spaces would be mangled" +fi + +# ---- a capitalised step must find its lowercase earlier stage --------------- # +# Book titles are capitalised ("Python", "Util-linux") but the chapter-7 stage +# is named from the tarball ("python-tmp"). Matching only the step's own +# spelling meant Python never claimed python-tmp's files, and chapter 7's +# root-owned tree blocked the chapter-8 install: +# install: cannot remove '/usr/lib/python3.13/idlelib/Icons/README.txt' +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # the owner filter moved into _claim_candidates, which does one stat for + # the whole manifest instead of one per path (1.14.2) -- read both + ce="$(sed -n '/^_claim_earlier_stages() {/,/^}/p' "$helper_src" + sed -n '/^_claim_candidates() {/,/^}/p' "$helper_src")" + case "$ce" in + *"tr 'A-Z' 'a-z'"*) ok "a capitalised step finds its lowercase stage" ;; + *) bad "Python would never claim python-tmp's files" ;; + esac + case "$ce" in + *'for stem in "$name" "$lname" "$owner" "$owname"'*) + ok "the owner's spelling is tried as well as the step's" ;; + *) bad "only the step name is tried" ;; + esac + case "$ce" in + *"de-duplicate"*) ok "a manifest is not walked twice" ;; + *) bad "the same manifest could be walked four times" ;; + esac +fi + +# ---- a package user should record where it came from ------------------------ # +# An LFS base package, a BLFS package, a pip module and a local script are +# maintained differently and updated from different places. Nothing in a +# package user's home said which it was. +if [ -f "$helper_src" ]; then + grep -q "^_write_pkgusr_info()" "$helper_src" \ + && ok "each package user records its source" \ + || bad "there is no way to tell an LFS package from a BLFS one" + grep -q "^cmd_pkgusr_info()" "$helper_src" \ + && ok "it can be read back" \ + || bad "the record is written but never readable" + # both completion paths must write it, not just one + cb="$(sed -n '/^cmd_build() {/,/^}/p' "$helper_src")" + n="$(printf '%s' "$cb" | grep -c '_write_pkgusr_info')" + [ "${n:-0}" -ge 2 ] \ + && ok "every path that completes a build records it" \ + || bad "only one of the two done paths records the source" + # the helper must be defined before cmd_build uses it + i_def="$(grep -n '^_write_pkgusr_info()' "$helper_src" | head -1 | cut -d: -f1)" + i_use="$(grep -n '^cmd_build() {' "$helper_src" | head -1 | cut -d: -f1)" + [ -n "$i_def" ] && [ -n "$i_use" ] && [ "$i_def" -lt "$i_use" ] \ + && ok "it is defined before the build uses it" \ + || bad "the branding helper is defined after its first use" + ia="$(sed -n '/^cmd_install_as() {/,/^}/p' "$helper_src")" + case "$ia" in + *"LFS_PKG_SOURCE"*) ok "install-as records what kind of install it was" ;; + *) bad "a BLFS or local install would be recorded as lfs" ;; + esac +fi + +# ---- adopt-existing must still exist ----------------------------------------- # +# The pass moved into verify, but the command was removed while messages +# elsewhere still told people to run it: +# !! unknown command 'adopt-existing' +# A command that has been deleted is worse than one that is redundant. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^cmd_adopt_existing()" "$helper_src" \ + && ok "adopt-existing is still a command" \ + || bad "a command the messages tell people to run does not exist" + grep -qE "^\s+adopt-existing\|adopt-dirs\)" "$helper_src" \ + && ok "it is reachable from the command line" \ + || bad "adopt-existing is defined but not dispatched" + # every command named in a hint must actually dispatch + for c in fix-ownership fix-orphans grant-dir verify which-package \ + check-toolchain seal-install-dirs sort-users install-as; do + # a command may be dispatched on its own or as part of an alternation + grep -qE "^\s+[a-z|-]*\b$c[|)]" "$helper_src" \ + || bad "$c is referenced but not dispatched" + done + ok "every command referenced in the hints is dispatched" + + # ---- and the users must be created in BUILD order --------------------- # + # This pass creates the package users it finds missing, and a uid is just + # the order of creation. Alphabetical order gave man-pages (first + # chapter-8 package) 10000 and binutils -- built first, in chapter 5 -- a + # much higher one, so the passwd file no longer reads as the build order. + vm="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" + case "$vm" in + *'$(_manifests_in_build_order)'*) + ok "the users are created in build order" ;; + *) bad "manifests are still walked alphabetically" ;; + esac + mo="$(sed -n '/^_manifests_in_build_order() {/,/^}/p' "$helper_src")" + case "$mo" in + *'[ "${1:-}" = dirs ]'*) + ok "the directory manifests are ordered the same way" ;; + *) bad "the .dirs manifests are still alphabetical" ;; + esac +fi + +# ---- package users cannot exist before book 7.6 ----------------------------- # +# A crosschain snapshot is taken BEFORE chapter 7, so /etc/passwd does not +# exist -- 7.6 creates it. Running the adoption pass then printed +# user binutils (could not create) -> package user +# forty times, once per package, with no hint why. The give-away is the +# prompt: "I have no name!" means even root has no passwd entry. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_user_db_ready()" "$helper_src" \ + && ok "the user database is checked before creating users" \ + || bad "adoption would fail once per package with no explanation" + ae="$(sed -n '/^cmd_adopt_existing() {/,/^}/p' "$helper_src")" + case "$ae" in + *_explain_no_user_db*) + ok "it explains that the tree is simply too early" ;; + *) bad "the error does not say what is actually wrong" ;; + esac + # and the work must still happen, later, without being asked again + # The late pass is the ownership epoch now: build-all calls + # _ownership_checkpoint after every step, and the marker makes it fire once. + ba="$(_slice_fn "$helper_src" cmd_build_all)" + case "$ba" in + *_ownership_checkpoint*) + ok "the packages are adopted once chapter 7 has created it" ;; + *) bad "a chapter 5-6 tree would never adopt its packages" ;; + esac + case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in + *ownership_established*) ok "the late pass runs once, not per step" ;; + *) bad "the late adoption would repeat on every step" ;; + esac + ud="$(sed -n '/^_user_db_ready() {/,/^}/p' "$helper_src")" + case "$ud" in + *"grep -q '^root:'"*) + ok "an empty or rootless passwd counts as not ready" ;; + *) bad "a truncated passwd would look usable" ;; + esac +fi + +# ---- "did not change owner" vs "no package claims it" ----------------------- # +# The ownership pass can only act on what a manifest lists. A file installed +# before tracking existed, or whose manifest was lost, stays root-owned and +# never appears in the report at all -- so a skipped file and an unclaimed one +# look identical from the outside, and they need different fixes. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + grep -q "^_vfy_unclaimed()" "$helper_src" \ + && ok "files no manifest claims are reported" \ + || bad "an unclaimed file is indistinguishable from a skipped one" + vu="$(sed -n '/^_vfy_unclaimed() {/,/^}/p' "$helper_src")" + # /etc is root's by right; reporting it would bury the real cases + case "$vu" in + *'"${SNAP_ROOT%/}/etc"'*) ok "configuration under /etc is not reported" ;; + *) bad "every file in /etc would be listed as unclaimed" ;; + esac + # the manifest paths carry the host prefix; comparison must strip it + case "$vu" in + *strip_host_prefix*) ok "manifest paths are compared without the prefix" ;; + *) bad "every claimed file would look unclaimed inside the chroot" ;; + esac + case "$vu" in + *"lfs-helper which-package"*) + ok "it says how to find out which package should own them" ;; + *) bad "the report gives no way forward" ;; + esac + cv="$(sed -n '/^cmd_verify() {/,/^}/p' "$helper_src")" + case "$cv" in + *_vfy_unclaimed*) ok "verify runs the check" ;; + *) bad "the unclaimed check is never called" ;; + esac +fi + + +# ---- a versioned chapter-9 "configuration" section is a PACKAGE ------------- # +# Book 9.2 is titled LFS-Bootscripts-20250827: a versioned tarball that unpacks +# and installs programs into /etc/rc.d/init.d and /lib/services. `lfs` gets +# this right -- _title_pkgver finds a version, so it generates a full phased +# build and deliberately keeps the step OUT of rootsteps, meaning "build this +# as a package user". +# +# lfs-helper then blanket-matched cfg_* as "not a package", refused to create +# the user, and the non-root path staged the script as that user anyway: +# 'cfg_bootscripts' is a build step, not a package -- no user created. +# install: invalid user 'cfg_bootscripts' +# !! could not stage the install script +# The build stopped at 88/99. Two tools, two answers, no shared authority. +# +# The script itself is the authority: a package has pkg_glob, prose does not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cd="$T/cfgpkg"; mkdir -p "$_cd" + printf 'pkg_glob="lfs-bootscripts-*.tar.*"\n' > "$_cd/cfg_bootscripts.sh" + printf '#!/bin/bash\necho x > /etc/hostname\n' > "$_cd/cfg_hostname.sh" + _r="$( SCRIPTS="$_cd"; PKGUSR_PREFIX="p_" + # _is_not_a_package strips the package-user prefix first, so the + # helper it calls has to be here too -- without it every name came + # back empty and matched nothing, which reads as "is a package". + eval "$(sed -n '/^unprefix_pkg_user() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^_is_not_a_package() {/,/^}/p' "$helper_src")" + # a versioned cfg section installs files and needs an owner + _is_not_a_package cfg_bootscripts && echo "bootscripts-has-no-user" + # a prose one writes root's config and must NOT get a user + _is_not_a_package cfg_hostname || echo "hostname-got-a-user" + # a cfg step with no script at all is prose until proven otherwise + _is_not_a_package cfg_nothing || echo "unknown-cfg-step-got-a-user" + # the genuinely-not-packages must stay that way + _is_not_a_package last-step || echo "last-step-got-a-user" + _is_not_a_package init-dirs || echo "init-dirs-got-a-user" + _is_not_a_package refind || echo "refind-got-a-user" + # and a normal package is still a package + _is_not_a_package gcc && echo "gcc-lost-its-user" + # callers pass the OWNER name too, already prefixed + _is_not_a_package p_cfg_bootscripts && echo "prefixed-form-misread" )" + if [ -z "$_r" ]; then + ok "a versioned cfg_* section is built as its own package user" + else + printf '%s\n' "$_r" | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + fi + # and lfs must agree: a versioned cfg section is kept out of rootsteps + grep -q 'pkg_like_cfg.append(cname)' "$LFS_TOOL" \ + && ok "lfs keeps a versioned cfg section out of the root steps" \ + || bad "lfs no longer treats a versioned cfg section as a package" +fi + +# ---- the triplet warning must not fire on a healthy toolchain --------------- # +# A finished build has TWO C++ header directories: chapter 6 installs libstdc++ +# under $LFS_TGT (x86_64-lfs-linux-gnu, or whatever vendor string is set) and +# chapter 8's native gcc installs its own under x86_64-pc-linux-gnu. Both are +# present and that is correct -- the chapter-6 one is simply left over. +# +# The check took the FIRST glob match and stopped, so it reported a mismatch +# whenever the stale directory sorted first -- which is alphabetical, i.e. luck: +# "nimgnu" < "pc", so a perfectly good toolchain warned on every single package. +# What matters is only whether a directory matching the compiler exists. +if [ -f "$helper_src" ]; then + _tp="$T/triplet"; mkdir -p "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits" \ + "$_tp/15.2.0/x86_64-pc-linux-gnu/bits" + touch "$_tp/15.2.0/x86_64-nimgnu-linux-gnu/bits/c++config.h" \ + "$_tp/15.2.0/x86_64-pc-linux-gnu/bits/c++config.h" + # the fixed shape: scan every match, stay silent if one is the compiler's + _fn="$(sed -n '/^_warn_on_triplet_mismatch() {/,/^}/p' "$helper_src")" + _p="" + case "$_fn" in + *'[ "$have" = "$mine" ] && return 0'*) ;; + *) _p="$_p;the check does not stop at the compiler's own headers" ;; + esac + case "$_fn" in + *"break"*) _p="$_p;the check still stops at the first directory found" ;; + esac + case "$_fn" in + *'[ -n "$found" ] || return 0'*) ;; + *) _p="$_p;a tree with no C++ headers yet would warn" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the triplet warning ignores leftover chapter-6 headers" + fi +fi + +# ---- the adoption gate must actually close --------------------------------- # +# Chapters 5-6 build before /etc/passwd exists, so their files are recorded in +# manifests with no owner. Once book 7.6 creates the user database, the +# adoption pass hands each package its files, and adopted.list records +# "name size" so it does not run again. +# +# It ran again on EVERY build-all, and the cause was that the gate and the +# recorder tested opposite conditions: +# _needs_adoption: user_exists "$owner" || return 0 # no user -> NEEDS +# _mark_all_adopted: user_exists "$owner" || continue # no user -> SKIP +# So a single manifest without a user could never be recorded and kept +# _has_unadopted_packages at yes forever. One rule now, used by both. +# +# The write also swallowed every error (2>/dev/null on the redirect AND +# `|| rm -f` on the move), so a missing or read-only state directory looked +# exactly like a successful record. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # the state directory is sorted now: adopted.list lives under progress/ + _ad="$T/adopt"; mkdir -p "$_ad/manifests" "$_ad/progress" + for f in haveuser nouser; do echo "/usr/bin/$f" > "$_ad/manifests/$f.files"; done + _out="$( STATE="$_ad"; MANIFESTS="$_ad/manifests"; SCRIPTS="$_ad/scripts" + STATE_PROGRESS="$_ad/progress" + warn(){ :; } + user_exists(){ [ "$1" = haveuser ]; } # only one package has a user + pkg_owner_name(){ echo "$1"; } + is_done(){ return 1; } # none built by lfs-helper here + PKGUSR_PREFIX="p_" + # _is_not_a_package strips the prefix first, so its helper is needed too + for fn in _ADOPTED_LIST _manifest_size _adopted_size _mark_adopted \ + _mark_all_adopted _needs_adoption _has_unadopted_packages \ + unprefix_pkg_user _is_not_a_package; do + eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" + done + _has_unadopted_packages || echo "gate-was-closed-before-the-pass" + _mark_all_adopted + # the package WITHOUT a user must be recorded too -- that is the fix + grep -q '^nouser ' "$_ad/progress/adopted.list" 2>/dev/null \ + || echo "a package without a user is never recorded" + # and the gate must now be shut + _has_unadopted_packages && echo "the gate stays open after the pass" + # but a manifest that GROWS has new files to hand over -- reopen + echo /usr/bin/extra >> "$_ad/manifests/haveuser.files" + _has_unadopted_packages || echo "a grown manifest is not re-adopted" )" + if [ -z "$_out" ]; then + ok "the adoption gate closes once the pass has run" + else + printf '%s\n' "$_out" | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + fi + # a failed write must be reported, not swallowed: an unrecorded adoption + # means the pass runs forever and nothing on disk says why + _w="$( STATE=/dev/null/nope + warn(){ echo "reported"; } + for fn in _ADOPTED_LIST _manifest_size _mark_adopted; do + eval "$(sed -n "/^$fn() {/,/^}/p" "$helper_src")" + done + _mark_adopted p 12 2>/dev/null && echo "claimed-success" )" + case "$_w" in + *reported*) ok "a failed adoption record is reported, not swallowed" ;; + *) bad "a failed adoption record still looks like success" ;; + esac +fi + +# ---- chapter-8 configuration must be generated, and must not rot ----------- # +# nsswitch.conf, the timezone and the dynamic-loader config were never +# generated: the step table only harvested chapter 9. Without /etc/nsswitch.conf +# glibc falls back to compiled-in defaults; without a timezone the system runs +# UTC forever. +# +# Two of those sections carry ids the BOOK TOOLCHAIN generated -- +# idm139921653990176 8.5.2.1. Adding nsswitch.conf +# -- which change every time the book is rebuilt. Naming one in the table +# works exactly once and then resolves to nothing, silently. Silently is the +# real danger: /etc/shells went missing from finished systems because +# cfg_shells resolved to zero blocks and was skipped without a word. So a step +# may name an id OR a title fragment, and this test fails if any chapter-8 +# entry stops resolving. +python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYCFG8' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +try: + from bs4 import BeautifulSoup +except ImportError: + print(" SKIP beautifulsoup4 not installed"); sys.exit(0) + +if not hasattr(lfs, "CHROOT_CONFIG_STEPS_8"): + print(" FAIL chapter-8 configuration is not generated at all"); sys.exit(1) + +soup = BeautifulSoup(open(sys.argv[2], encoding='utf-8', errors='replace').read(), + 'html.parser') +secs = {sid: (t, sec) for sid, t, sec in lfs.iter_sections(soup)} + +for name, sid in lfs.CHROOT_CONFIG_STEPS_8: + e = lfs._resolve_section(secs, sid) + if not e: + problems.append("%s: '%s' resolves to no section" % (name, sid)) + elif not e[1]["commands"]: + problems.append("%s: resolved but has no command blocks" % name) + +# the resolver itself: id first, then title, and never a silent guess +if lfs._resolve_section(secs, "ch-config-hostname") is None: + problems.append("a stable section id no longer resolves") +if lfs._resolve_section(secs, "Adding nsswitch.conf") is None: + problems.append("a section cannot be found by its title") +if lfs._resolve_section(secs, "zzz-no-such-section") is not None: + problems.append("a nonexistent section resolves to something") +# an ambiguous fragment must NOT be resolved by picking the first match +if lfs._resolve_section(secs, "Configuring") is not None: + problems.append("an ambiguous title fragment is silently resolved") + +# /tools deletion and stripping are opt-in: neither should ever be a default. +# The DEFAULT is the question, so the machine's real config must not answer +# it: on the built system the recorded interview said remove_tools=yes and +# this reported a bug that was actually the user's own choice. +import os, tempfile +os.environ["LFS_STORE"] = tempfile.mkdtemp() +if lfs.remove_tools(): + problems.append("/tools is deleted by default") +if lfs.strip_binaries(): + problems.append("binaries are stripped by default") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS chapter-8 configuration resolves by id or title, and cannot rot") +PYCFG8 +_count_rc $? + +# cfg_rm-tools deletes a root-owned tree, so it must be a root step -- never a +# package user's job, and never adoptable +grep -q 'root_steps.append("cfg_rm-tools")' "$LFS_TOOL" \ + && ok "removing /tools is a root step" \ + || bad "removing /tools is not a root step" + +# ---- every build decision is asked UP FRONT --------------------------------- # +# The build used to discover decisions halfway through, or at the very end: +# last_build_step.sh blocked on `passwd root` after a six-hour build, and its +# own comment admits the non-interactive path leaves the system +# "unbootable-but-known". Collecting the answers in `session` is what lets +# `lfs run` and `build-all` finish without a person sitting there. +# +# They must also REACH the chroot: the config file is on the host and the +# chroot environment is reset, so anything not written into the tree's env +# file is simply not there when it is needed. +python3 - "$LFS_TOOL" <<'PYASK' +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +keys = [k for k, _, _ in lfs._SESSION_KEYS] +for k in ("timezone", "remove_tools", "strip", "main_user"): + if k not in keys: + problems.append("'%s' is not asked during session" % k) + +# The DEFAULT of remove_tools/strip is checked in the chapter-8 block above, +# against an EMPTY store -- one check, one place. A second copy here read the +# real machine's interview answers and reported remove_tools=yes, the user's +# own choice, as a bug. One decision held in two places, in the suite itself. + +# each answer has to be written into the tree, or the chroot never sees it +body = open(sys.argv[1]).read() +i = body.index("def write_chroot_env") +j = body.find("\ndef ", body.index("_make_world_readable(path)", i)) +w = body[i:j if j > i else len(body)] +for var in ("LFS_TIMEZONE", "LFS_MAIN_USER", "LFS_STRIP", "LFS_REMOVE_TOOLS"): + if var not in w: + problems.append("%s never reaches the chroot" % var) + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every build decision is collected before the build starts") +PYASK +_count_rc $? + +# and the accounts step must USE the configured name rather than stopping to +# ask for it -- the whole point of asking early +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + case "$(_slice_fn "$helper_src" cmd_init_accounts)" in + *main_user*) ok "the final step uses the login account chosen up front" ;; + *) bad "the final step stops to ask for a name already configured" ;; + esac +fi + +# ---- a section that ships a FILE instead of commands ----------------------- # +# Book 9.6.8 (The rc.site File) has no <kbd class="command"> at all. It prints +# /etc/sysconfig/rc.site as a <pre class="auto"> block with every line +# commented out, for the reader to copy and uncomment. The parser harvests +# command blocks only, so cfg_rcsite resolved to zero blocks and the file was +# never created -- the same silent-nothing failure that once lost /etc/shells. +# +# An all-commented file is still worth installing: it documents every knob the +# boot scripts have, in the place the boot scripts look for it. +python3 - "$LFS_TOOL" "$LFS_BOOK" <<'PYRCSITE' +import sys, subprocess, tempfile, os, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +if not hasattr(lfs, "_auto_file_block"): + print(" FAIL a section that ships a file instead of commands is ignored") + sys.exit(1) + +raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() +blk = lfs._auto_file_block(raw, "ch-config-site", "/etc/sysconfig/rc.site") +if not blk: + problems.append("rc.site is not extracted from the book") +else: + d = tempfile.mkdtemp() + sh = blk.replace("/etc/sysconfig", d + "/etc/sysconfig") + open(d + "/t.sh", "w").write("set -e\n" + sh + "\n") + r = subprocess.run(["bash", d + "/t.sh"], capture_output=True, text=True) + out = d + "/etc/sysconfig/rc.site" + if r.returncode != 0: + problems.append("the generated rc.site block is not valid bash: %s" + % r.stderr.strip()[:80]) + elif not os.path.isfile(out): + problems.append("the block runs but writes no file") + else: + body = open(out).read() + if len(body.splitlines()) < 50: + problems.append("rc.site came out truncated (%d lines)" + % len(body.splitlines())) + # a quoted heredoc: the body is full of # and shell metacharacters and + # must land byte-for-byte as the book prints it, not expanded + if "$" in body and "EOF" not in blk.split("\n")[1]: + problems.append("the heredoc is not quoted -- the body would expand") + +# a section with neither commands nor a file body must still be reported, not +# silently skipped +if lfs._auto_file_block(raw, "ch-config-sysklogd", "/etc/x") is not None: + problems.append("a prose section is treated as if it shipped a file") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a section that ships a file body is installed, not skipped") +PYRCSITE +_count_rc $? + +# ---- the package-user prefix must be one decision, not three --------------- # +# lfs-helper has ALWAYS read a prefix and defaulted to `p`: +# PKGUSR_PREFIX="${LFS_PKGUSR_PREFIX-$(_read_pkgusr_prefix || echo p)}" +# but nothing ever wrote the setting -- `lfs` did not put it in the chroot env +# and `packagemanager` did not implement it at all. So the default applied by +# accident, and the built system would look for `gcc` while the build had +# created `p_gcc`. Same species as cfg_bootscripts and adopted.list: one +# decision held in several places that could disagree. +# +# Prefixing MUST be idempotent. It is applied to names typed by a person +# (gcc), to directory names read back from disk (p_gcc), and to values that +# already went through it -- lfs-helper's own comment names p_p_gcc as the +# thing to avoid. +python3 - "$LFS_TOOL" <<'PYPFX' +import sys, os, tempfile, importlib.machinery as m +problems = [] +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") +if not os.path.exists(pmp): + print(" SKIP packagemanager not found next to the tool"); sys.exit(0) +pm = m.SourceFileLoader('pm', pmp).load_module() + +if not hasattr(lfs, "pkgusr_prefix"): + problems.append("lfs cannot say what the package-user prefix is") +if not hasattr(pm, "pkgusr_name"): + problems.append("packagemanager does not implement the prefix") + +if not problems: + # the two tools must agree, or the build and the built system diverge + if lfs.pkgusr_prefix() != pm.pkgusr_prefix(): + problems.append("lfs and packagemanager disagree: %r vs %r" + % (lfs.pkgusr_prefix(), pm.pkgusr_prefix())) + if pm.pkgusr_prefix() != "p_": + problems.append("the default prefix is not p_ (got %r)" + % pm.pkgusr_prefix()) + if pm.pkgusr_name("gcc") != "p_gcc": + problems.append("a package name is not prefixed") + # idempotent -- the whole reason every call site can apply it blindly + if pm.pkgusr_name(pm.pkgusr_name("gcc")) != "p_gcc": + problems.append("prefixing twice gives p_p_gcc") + if pm.unprefix_pkgusr("p_gcc") != "gcc": + problems.append("the prefix cannot be stripped for display") + # the OTHER kinds of account must not be touched: an application user and + # a collector group are not packages + if pm.pkgusr_name("u_firefox") != "u_firefox": + problems.append("an application user was given the package prefix") + if pm.pkgusr_name(pm.collector_prefix() + "python") \ + != pm.collector_prefix() + "python": + problems.append("a collector group was given the package prefix") + # the home is the ACCOUNT's, whichever kind of name it is asked with + if pm.pkgusr_home("gcc") != pm.pkgusr_home("p_gcc"): + problems.append("a package name and its user name give different homes") + +# and it has to REACH the chroot, or lfs-helper falls back to a guess +body = open(sys.argv[1]).read() +if "LFS_PKGUSR_PREFIX" not in body: + problems.append("the prefix never reaches the chroot") +if "pkgusr_prefix" not in [k for k, _, _ in lfs._SESSION_KEYS]: + problems.append("the prefix is never asked for") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS lfs, lfs-helper and packagemanager share one package-user prefix") +PYPFX +_count_rc $? + +# lfs-helper's side: it must take the value from the environment lfs wrote, +# and an explicitly EMPTY value must mean "no prefix" -- not fall through to +# the default and rename every user out from under an existing tree +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # Behaviour, not text (1.13.26 added a config-wins rule): a config that + # NAMES a prefix beats a stale empty export, and a config that says + # explicitly none still yields none. + _pfxq="$T/pfxq"; _pfxr="$T/pfxr"; mkdir -p "$_pfxq" "$_pfxr" + printf '{ "pkgusr_prefix": "p" }\n' > "$_pfxq/config.json" + printf '{ "pkgusr_prefix": "" }\n' > "$_pfxr/config.json" + _runpfx() { + { [ "$2" = empty ] && echo 'export LFS_PKGUSR_PREFIX=""' || echo 'unset LFS_PKGUSR_PREFIX' + sed -n '/^_read_pkgusr_prefix() {/,/^}/p' "$helper_src" \ + | sed "s|/usr/share/lfs/config.json|$1/config.json|" + sed -n '/^_cfg_prefix=/,/^unset _cfg_prefix/p' "$helper_src" + sed -n '/^PKGUSR_PREFIX="${PKGUSR_PREFIX%_}"/,+1p' "$helper_src" + echo 'echo "[$PKGUSR_PREFIX]"'; } | bash 2>/dev/null | tail -1 + } + if [ "$(_runpfx "$_pfxq" empty)" = "[p_]" ] \ + && [ "$(_runpfx "$_pfxq" unset)" = "[p_]" ] \ + && [ "$(_runpfx "$_pfxr" empty)" = "[]" ]; then + ok "an empty prefix means none, and a stale empty export does not override the config" + else + bad "an empty prefix falls through to the default" + fi +fi + +# ---- accounts live in subdirectories by kind ------------------------------- # +# Flat under /usr/src, a hundred package users and a handful of config steps +# were one undifferentiated list, and nothing about a directory said what kind +# of thing it was. Now: +# /usr/src/pkgusr/p_gcc a package +# /usr/src/cfg/p_cfg_bootscripts a config step that installs files +# All three tools have to agree, or the build creates a home the built system +# never looks in. +python3 - "$LFS_TOOL" <<'PYSPLITHOME' +import sys, os, tempfile, importlib.machinery as m +problems = [] +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +pmp = os.path.join(os.path.dirname(sys.argv[1]) or ".", "packagemanager") +if not os.path.exists(pmp): + print(" SKIP packagemanager not found next to the tool"); sys.exit(0) +d = tempfile.mkdtemp(); os.environ["PKGUSR_BASE"] = d +pm = m.SourceFileLoader('pm', pmp).load_module() + +for k in ("target_pkgusr_home", "target_cfguser_home", + "pkgusr_home", "cfguser_home"): + if k not in lfs.LAYOUT_DEFAULTS: + problems.append("layout has no '%s'" % k) +if lfs.LAYOUT_DEFAULTS.get("pkgusr_home") == lfs.LAYOUT_DEFAULTS.get("cfguser_home"): + problems.append("packages and config steps share one directory") +if not lfs.LAYOUT_DEFAULTS.get("pkgusr_home", "").endswith("/pkgusr"): + problems.append("packages do not default to /usr/src/pkgusr") +if not lfs.LAYOUT_DEFAULTS.get("cfguser_home", "").endswith("/cfg"): + problems.append("config steps do not default to /usr/src/cfg") + +# a package and a config step must land in DIFFERENT roots, whichever form of +# the name they are asked with +if pm.pkgusr_home("gcc") == pm.pkgusr_home("cfg_bootscripts"): + problems.append("a package and a config step share a root") +if not pm.pkgusr_home("gcc").startswith(os.path.join(d, "pkgusr")): + problems.append("a package is not under the package root") +if not pm.pkgusr_home("cfg_bootscripts").startswith(os.path.join(d, "cfg")): + problems.append("a config step is not under the config root") +# prefixed form must resolve identically -- callers pass both +if pm.pkgusr_home("p_cfg_bootscripts") != pm.pkgusr_home("cfg_bootscripts"): + problems.append("the prefixed name lands somewhere else") + +# listing must walk BOTH roots, or half the accounts vanish +os.makedirs(os.path.join(d, "pkgusr", pm.pkgusr_name("gcc"))) +os.makedirs(os.path.join(d, "cfg", pm.pkgusr_name("cfg_bootscripts"))) +got = pm.list_all_users() +for want in ("gcc", "cfg_bootscripts"): + if want not in got: + problems.append("'%s' is missing from the account list %r" % (want, got)) + +# and both roots must reach the chroot +body = open(sys.argv[1]).read() +for var in ("LFS_PKGUSR_ROOT", "LFS_CFGUSR_ROOT"): + if var not in body: + problems.append("%s never reaches the chroot" % var) + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS packages and config steps live in separate roots") +PYSPLITHOME +_count_rc $? + +# lfs-helper's side: one chokepoint that routes by kind, and the passes that +# walk every account must walk both roots -- not just the package one +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SRCROOT=/tmp/_hr; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + # pkg_owner_name too: pkgusr_home_for calls it, and without it every + # home came back as the bare root with no account on the end + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + unprefix_pkg_user pkg_owner_name pkgusr_home_for pkgusr_roots; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # a package gets the package prefix and the package root + [ "$(pkgusr_home_for gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "package-in-wrong-root" + [ "$(pkgusr_home_for p_gcc)" = "$SRCROOT/pkgusr/p_gcc" ] || echo "prefixed-package-in-wrong-root" + # a config step gets the CONFIG prefix and the config root -- not the + # package prefix stacked on top of its own + [ "$(pkgusr_home_for cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "cfg-in-wrong-root" + # and a legacy p_cfg_ name is repaired rather than carried forward + [ "$(pkgusr_home_for p_cfg_bootscripts)" = "$SRCROOT/cfg/cfg_bootscripts" ] || echo "stacked-cfg-not-repaired" + [ "$(pkgusr_roots | wc -l)" = 2 ] || echo "not-both-roots-walked" )" + if [ -z "$_o" ]; then + ok "lfs-helper routes each account to the root for its kind" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + # no home may be built by hand any more -- that is how the two roots drift + grep -q '"\$SRCROOT/\$name"' "$helper_src" \ + && bad "a home is still built from SRCROOT directly" \ + || ok "every home goes through the one chokepoint" +fi + +# ---- the home a build cd's into is the home the account has ---------------- # +# The build creates the account with useradd -d <home>, then cd's into that +# home to run the staged script. Those two must be the same path, and they +# are reached from DIFFERENT names: cmd_add_user has the owner +# ("p_man-pages"), build_pkg has the step name ("man-pages"). +# +# pkgusr_home_for routed by kind but never applied the prefix, so: +# # created package user p_man-pages ... +# line 2673: cd: /usr/src/pkgusr/man-pages: No such file or directory +# Chapter 7 did not catch it -- root steps mkdir -p their own directory first, +# so it surfaced at the first real package user, nine steps in. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SRCROOT=/usr/src; PKGUSR_ROOT="$SRCROOT/pkgusr"; CFGUSR_ROOT="$SRCROOT/cfg" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + pkg_owner_name unprefix_pkg_user pkgusr_home_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # THE bug: a step name and its owner must give one home + [ "$(pkgusr_home_for man-pages)" = "$(pkgusr_home_for p_man-pages)" ] \ + || echo "step name and owner give different homes" + # and it must be the account's, i.e. prefixed + [ "$(pkgusr_home_for man-pages)" = "/usr/src/pkgusr/p_man-pages" ] \ + || echo "the home is not the account's: $(pkgusr_home_for man-pages)" + # book titles are capitalised, tarballs are not + [ "$(pkgusr_home_for Man-Pages)" = "$(pkgusr_home_for man-pages)" ] \ + || echo "a capitalised book title gives a different home" + # earlier stages fold onto the real package -- util-linux-tmp is + # util-linux's, which is where its files end up + [ "$(pkgusr_home_for util-linux-tmp)" = "$(pkgusr_home_for util-linux)" ] \ + || echo "an earlier stage gets its own home" + [ "$(pkgusr_home_for gcc-pass1)" = "$(pkgusr_home_for gcc)" ] \ + || echo "a pass1 build gets its own home" + # config steps still route to their own root, and wear ONE prefix -- + # their own. This used to read p_cfg_bootscripts: the package prefix + # stacked on the step name's own, so the account claimed to be a package + # while living in the config root. + [ "$(pkgusr_home_for cfg_bootscripts)" = "/usr/src/cfg/cfg_bootscripts" ] \ + || echo "a config step is not under the config root" )" + if [ -z "$_o" ]; then + ok "a step name and its owner resolve to the same account home" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the rule must be reachable from shell, or every other script re-derives it + # -- and a third copy is how the roots drift apart + grep -q 'pkgusr-home)' "$helper_src" \ + && ok "the home rule is available to scripts as 'lfs-helper pkgusr-home'" \ + || bad "shell scripts have to re-derive where an account lives" +fi + +# nothing outside the chokepoints may build an account path by hand +for _f in lfs-completion.bash; do + _p="$(dirname "$LFS_TOOL")/$_f" + [ -f "$_p" ] || continue + if grep -qE '"?/usr/src/\$[a-z_]+' "$_p"; then + bad "$_f builds an account path by hand" + else + ok "$_f asks for the account home instead of building it" + fi +done + +# ---- chapter-8 config steps must not stop the build ------------------------ # +# Two ways they did, both from lifting a sub-section out of its parent package. +# +# cfg_timezone (8.5.2.2) ends with an INTERACTIVE menu whose only output is +# the name to put in the placeholder: +# tzselect +# ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime +# `session` already asks for the timezone, so both are answered. The same +# section also unpacks tzdata with a path relative to GLIBC's build directory +# (`tar -xf ../../tzdata2025b.tar.gz`), which resolves nowhere once the step +# stands on its own. +python3 - "$LFS_TOOL" <<'PYCFG8RUN' +import sys, re, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] +body = ("tar -xf ../../tzdata2025b.tar.gz\n" + "zic -d $ZONEINFO -p America/New_York\n" + "tzselect\n" + "ln -sfv /usr/share/zoneinfo/<xxx> /etc/localtime\n") +out, left = lfs._fill_placeholders(body, "cfg_timezone") +if left: + problems.append("the timezone step still asks for %s" % left) +if "<xxx>" in out: + problems.append("the timezone placeholder is not filled in") +if re.search(r'^\s*tzselect\s*$', out, flags=re.M): + problems.append("the interactive tzselect menu still runs") +if "../../tzdata" in out: + problems.append("tzdata is still read relative to glibc's build directory") +if "SOURCES_DIR" not in out: + problems.append("tzdata is not taken from the sources directory") +if "/usr/share/zoneinfo//etc/localtime" in out.replace(" ", ""): + problems.append("the localtime symlink lost its zone name") +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the timezone step is answered from config, not from a prompt") +PYCFG8RUN +_count_rc $? + +# cfg_ld rewrites /etc/ld.so.conf -- which glibc created and still owns -- and +# makes one directory. Nothing NEW appears, so the manifest count is 0 and the +# step was failed as a silent no-op: +# mkdir: created directory '/etc/ld.so.conf.d' +# !! cfg_ld reported success but installed NO files. +# That is the normal shape of a configuration step. Packages must still fail. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # to the end of the branch, not a fixed window: the message grew a + # breakdown of where the count came from, and a +6 window stopped + # reaching the `return 1` it is asserting + _blk="$(sed -n '/reported success but installed NO files/,/^ fi$/p' "$helper_src")" + # the whole function, not a fixed window above the message: the branch + # gained a third case (a re-run over its own work) and a -B 14 window + # stopped reaching the config-step test it is asserting + _pre="$(_slice_fn "$helper_src" cmd_build)" + case "$_pre" in + *'_is_not_a_package "$name"'*) + ok "a config step may install no new files; a package may not" ;; + *) bad "installing no new files still fails a configuration step" ;; + esac + case "$_blk" in + *'return 1'*) ok "a package that installs nothing still fails" ;; + *) bad "the no-files check no longer fails a package" ;; + esac +fi + +# ---- a symlinked install directory is checked at its target ---------------- # +# Book 7.x makes /bin, /sbin, /lib and /lib64 symlinks into /usr. `test -d` +# FOLLOWS a symlink but `stat` does not, so verify compared the LINK's own +# mode and owner -- always lrwxrwxrwx root:root -- against what a directory +# should look like, and reported all of them on every run: +# /bin root:root lrwxrwxrwx (want root:install, group-writable) +# Chowning a symlink changes nothing that matters, and the target (/usr/bin) +# is in the same list and checked on its own. Both the report and the --fix +# path need the guard: without it, --fix "repairs" a symlink forever. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # There used to be two copies of the install-dir rule and so two guards to + # count. cmd_verify calls _vfy_install_dirs now, so there is one -- and + # the check and the fix share it, which is the property this was after. + _vid="$(_slice_fn "$helper_src" _vfy_install_dirs)" + if printf '%s\n' "$_vid" | grep -q '\[ -L "\$live" \] && continue'; then + ok "a symlinked install dir is skipped by both the check and the fix" + else + bad "a symlinked install dir is still reported" + fi + # and the guard must come BEFORE the stat, or it reports then skips + if sed -n '/install directories not in the expected state/,-12p' \ + "$helper_src" >/dev/null 2>&1; then :; fi + _blk="$(grep -A 4 '\[ -d "\$live" \] || continue' "$helper_src" | head -20)" + case "$_blk" in + *'-L "$live"'*) ok "the symlink guard runs before stat" ;; + *) bad "stat runs before the symlink guard" ;; + esac +fi + +# ---- verify must write every finding to a file ----------------------------- # +# The console prints the first eight of each kind and then +# ... and 44265 more +# which is a number and nothing you can act on: it cannot be grepped, diffed +# against the previous run, or fed to which-package. Every finding goes to a +# log, tagged by kind, while the console stays short. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^VERIFY_LOG=' "$helper_src" || _p="$_p;verify has no log file" + grep -q '_vfy_log_open' "$helper_src" || _p="$_p;the log is never opened" + # every truncated list must also be written in full + for _k in wrong-owner unclaimed installdir; do + grep -q "$_k" "$helper_src" || _p="$_p;nothing logs '$_k' findings" + done + # and the summary must say where it is, or nobody finds it + grep -q 'full report: \$VERIFY_LOG' "$helper_src" \ + || _p="$_p;the summary never names the log" + # a read-only state dir must not abort verify -- the report still matters + grep -q 'VERIFY_LOG=""; return 0' "$helper_src" \ + || _p="$_p;verify dies instead of running when the log cannot be written" + + _o="$( STATE=/tmp/_vst; SNAP_ROOT=/; VERIFY_LOG=/tmp/_vst/verify.log + rm -rf /tmp/_vst + eval "$(sed -n '/^_vfy_log_open() {/,/^}/p' "$helper_src")" + eval "$(sed -n '/^_vfy_log() {/,/^}/p' "$helper_src")" + _vfy_log_open + _vfy_log wrong-owner "/usr/bin/python3 is root want p_python" + _vfy_log unclaimed "/usr/bin/lfs-helper" + [ -s /tmp/_vst/verify.log ] || echo "the log is empty after findings" + grep -q '^wrong-owner' /tmp/_vst/verify.log || echo "wrong-owner not tagged" + grep -q '^unclaimed' /tmp/_vst/verify.log || echo "unclaimed not tagged" + # a fresh run must not append to the last one's findings + _vfy_log_open + grep -q '^wrong-owner' /tmp/_vst/verify.log && echo "the log is not reset per run" + # an unwritable location must be survivable + VERIFY_LOG=/dev/null/nope/verify.log + _vfy_log_open >/dev/null 2>&1 || echo "an unwritable log aborts verify" )" + [ -n "$_o" ] && _p="$_p;$(printf '%s' "$_o" | tr '\n' ';')" + + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify writes every finding to a log, tagged by kind" + fi +fi + +# ---- the built system gets the whole toolchain, and it has an owner -------- # +# Only lfs-helper was copied into the tree, because only lfs-helper runs during +# the build. But the built system needs the rest -- packagemanager cannot be +# installed BY packagemanager -- and a system that boots without them has no +# way to install anything at all. They are a few files and they refresh on +# every `lfs run`. +# +# They are also software installed into /usr/bin, so they get a package user +# like anything else. Without one they are reported forever as +# 1516 root-owned file(s) that NO manifest claims: +# /usr/bin/lfs-helper +# which is true, unhelpful, and buries the cases that matter. +python3 - "$LFS_TOOL" <<'PYTOOLS' +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +problems = [] + +for t in ("lfs-helper", "packagemanager", "blfs"): + if t not in getattr(lfs, "TOOLCHAIN_SCRIPTS", ()): + problems.append("%s is not copied into the tree" % t) + +# the owner is a package user like any other, so it carries the prefix +if lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER != "p_pkgusr": + problems.append("the tools' owner is %r, not p_pkgusr" + % (lfs.pkgusr_prefix() + lfs.PKGUSR_TOOLS_USER)) + +# the uid must be read from the TREE's passwd, never this machine's -- a host +# uid means nothing inside the chroot, the same mistake book 3.1 warns about +d = tempfile.mkdtemp() +os.makedirs(os.path.join(d, "etc")) +if lfs._uid_in_tree(d, "p_pkgusr") is not None: + problems.append("a uid is invented when the tree has no passwd file") +with open(os.path.join(d, "etc", "passwd"), "w") as f: + f.write("root:x:0:0::/root:/bin/bash\np_pkgusr:x:10042:10042::/usr/src:/bin/bash\n") +if lfs._uid_in_tree(d, "p_pkgusr") != 10042: + problems.append("the tree's own passwd file is not used") +if lfs._uid_in_tree(d, "nosuchuser") is not None: + problems.append("an unknown user resolves to something") + +if problems: + for p in problems: print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the whole toolchain is installed and owned by p_pkgusr") +PYTOOLS +_count_rc $? + +# lfs-helper claims them once /etc/passwd is real -- lfs cannot, because on a +# fresh tree the account does not exist until book 7.6 +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '_adopt_pkgusr_tools' "$helper_src" || _p="$_p;the tools are never claimed" + grep -q 'PKGUSR_TOOLS=' "$helper_src" || _p="$_p;there is no list of tools to claim" + # it must run in build-all, beside the other adoption pass + case "$(_slice_fn "$helper_src" establish_ownership)" in + *_adopt_pkgusr_tools*) ;; + *) _p="$_p;the tools are not claimed when ownership is established" ;; + esac + # and it must never take another package's file + sed -n '/^_adopt_pkgusr_tools() {/,/^}/p' "$helper_src" \ + | grep -q 'user_exists "\$cur" && continue' \ + || _p="$_p;claiming the tools could steal another package's file" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the tools are claimed once the user database exists" + fi +fi + +# ---- an ADOPTED package gets a real home, like a built one ----------------- # +# Packages built in a run owned their files and had full homes; packages +# ADOPTED from chapter 5-6 manifests in the same run did not: +# drwxr-xr-x 1 p_bison p_bison 148 p_bison <- built +# drwxr-xr-x 1 root root 86 p_gcc <- adopted +# 86 vs 148 bytes: the adopted homes were missing the profile links and had +# never been chowned. cmd_add_user was called with BOTH streams suppressed, +# so whatever went wrong said nothing. Adoption now initialises the home +# explicitly (init_package_user_home is idempotent) and stderr is let through. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(sed -n '/^_vfy_manifest_ownership() {/,/^}/p' "$helper_src")" + _p="" + case "$_blk" in + *'init_package_user_home "$owner"'*) ;; + *) _p="$_p;an adopted package's home is never initialised" ;; + esac + case "$_blk" in + *'cmd_add_user "$owner" >/dev/null 2>&1'*) + _p="$_p;adoption still hides why a user could not be created" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an adopted package gets an initialised, owned home" + fi +fi + +# ---- one account, one prefix ----------------------------------------------- # +# A config step used to be `p_cfg_bootscripts`: pkg_owner_name applied the +# PACKAGE prefix to every name regardless of kind, so the account wore its own +# `cfg_` and `p_` on top of it. Neither prefix then identified the account -- +# `p_` said "package" while it lived in /usr/src/cfg -- and every rule that +# reads a prefix had to special-case the stacked form. +# +# The fix is that kind is decided ONCE (pkgusr_kind) and the prefix and the +# root are both answered from it, so a name cannot be given one kind's prefix +# and placed in another kind's root. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for \ + unprefix_pkg_user pkg_owner_name pkgusr_home_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + [ "$(pkg_owner_name cfg_bootscripts)" = "cfg_bootscripts" ] \ + || echo "a config step still gets the package prefix: $(pkg_owner_name cfg_bootscripts)" + [ "$(pkg_owner_name gcc)" = "p_gcc" ] \ + || echo "a package lost its prefix: $(pkg_owner_name gcc)" + # kind decides both answers, so they cannot disagree + [ "$(pkgusr_kind cfg_nsswitch)" = "cfg" ] || echo "a cfg step is not kind cfg" + [ "$(pkgusr_kind gcc)" = "pkg" ] || echo "a package is not kind pkg" + # a tree or a config written before the split holds stacked names; they + # are repaired on the way through rather than carried forward + [ "$(pkg_owner_name p_cfg_bootscripts)" = "cfg_bootscripts" ] \ + || echo "a stacked legacy name survives: $(pkg_owner_name p_cfg_bootscripts)" + # and the whole thing is still idempotent -- that property is what lets + # every call site apply it without knowing which kind of name it holds + for _n in gcc p_gcc cfg_bootscripts p_cfg_bootscripts; do + _a="$(pkg_owner_name "$_n")"; _b="$(pkg_owner_name "$_a")" + [ "$_a" = "$_b" ] || echo "not idempotent: $_n -> $_a -> $_b" + done )" + if [ -z "$_o" ]; then + ok "each kind of account wears exactly one prefix, its own" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# packagemanager must reach the same answer, or the built system looks for +# accounts the build never created +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pm_src" ]; then + python3 - "$pm_src" <<'PYONEPFX' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +ns = {"os": __import__("os"), "BASE_DIR": "/usr/src", + "PKGUSR_SUBDIR": "pkgusr", "CFGUSR_SUBDIR": "cfg", + "_DEFAULT_CONFIG": {"collector_prefix": "sysgroup", "user_prefix": "u", + "pkgusr_prefix": "p", "cfguser_prefix": "cfg", + "main_user": ""}} +ns["load_config"] = lambda: dict(ns["_DEFAULT_CONFIG"]) +for fn in ("collector_prefix", "user_prefix", "pkgusr_prefix", "cfguser_prefix", + "pkgusr_kind", "pkgusr_name", "unprefix_pkgusr", "pkgusr_root", + "pkgusr_home"): + m = re.search(r"^def %s\(.*?(?=\n\ndef |\n\n# |\nPKGUSR_SUBDIR)" % fn, src, re.S | re.M) + if not m: + problems.append("packagemanager has no %s" % fn) + continue + exec(m.group(0), ns) +if not problems: + want = { + # name -> (account name, home) + "gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), + "p_gcc": ("p_gcc", "/usr/src/pkgusr/p_gcc"), + "cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), + # the stacked legacy form is repaired, not carried forward + "p_cfg_bootscripts": ("cfg_bootscripts", "/usr/src/cfg/cfg_bootscripts"), + "u_firefox": ("u_firefox", "/usr/src/u_firefox"), + } + for given, (name, home) in want.items(): + got = ns["pkgusr_name"](given) + if got != name: + problems.append("packagemanager names %s as %s, want %s" % (given, got, name)) + got = ns["pkgusr_home"](given) + if got != home: + problems.append("packagemanager puts %s at %s, want %s" % (given, got, home)) + again = ns["pkgusr_name"](ns["pkgusr_name"](given)) + if again != ns["pkgusr_name"](given): + problems.append("packagemanager is not idempotent on %s" % given) +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS packagemanager and lfs-helper name each kind identically") +PYONEPFX + _count_rc $? +fi + +# ---- a step that is not a package leaves no home behind --------------------- # +# cmd_build did `mkdir -p "$(pkgusr_home_for "$name")"` for EVERY root step, +# including the ones _is_not_a_package will never give an account. A sanity run +# then found /usr/src/pkgusr/p_init-dirs and p_init-files: directories with no +# matching account, which every ownership pass has to report as strays. +# +# init-*, last-step and refind own no files, so they are staged and run out of +# the state directory instead -- beside the scripts they came from. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ + pkg_owner_name pkgusr_home_for _is_not_a_package step_stage_dir; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + # an empty answer must not read as "not under an account root" + command -v step_stage_dir >/dev/null 2>&1 \ + || echo "there is no step_stage_dir -- every root step still gets a home" + for _s in init-dirs init-files last-step refind; do + _d="$(step_stage_dir "$_s" 2>/dev/null)" + [ -n "$_d" ] || { echo "$_s resolves to no directory at all"; continue; } + case "$_d" in + "$PKGUSR_ROOT"/*|"$CFGUSR_ROOT"/*) + echo "$_s still gets a home under an account root" ;; + esac + done + # a real package is unaffected: it still runs from its user's home + [ "$(step_stage_dir gcc)" = "$PKGUSR_ROOT/p_gcc" ] \ + || echo "a package no longer runs from its own home: $(step_stage_dir gcc)" )" + if [ -z "$_o" ]; then + ok "a step with no account leaves no directory under the account roots" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the staged path, the mkdir, the cd and the failure banner must be ONE + # value. They were four separate calls to pkgusr_home_for, free to disagree. + _cb="$(_slice_fn "$helper_src" cmd_build)" + case "$_cb" in + *'cd "$(pkgusr_home_for'*) + bad "the build still re-derives the directory it cd's into" ;; + *) ok "the staged script, the cd and the banner share one path" ;; + esac +fi + +# ---- the account roots are a decision, not a side effect -------------------- # +# /usr/src/pkgusr and /usr/src/cfg came out root:root 0755 because `mkdir -p` +# on the first home created them with root's umask. Nothing said so, and +# nothing would have noticed it changing. +# +# They carry the same root:install 1775 as /usr/src above them -- one rule for +# the subtree instead of three directories with two rules. The STICKY BIT is +# what makes group-writable safe and is not optional: without it any member of +# `install` could delete or rename another package's home. +# +# They are still NOT install dirs. Being in never_adopt_list is the other +# half: the mode is shared, but no package may ever take one as its own. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _ep="$(_slice_fn "$helper_src" ensure_pkgusr_roots)" + case "$_ep" in + *'chmod 1775'*) ;; + *) _p="$_p;the account roots have no stated mode" ;; + esac + case "$_ep" in + *set_install_dir_owner*) ;; + *) _p="$_p;the account roots do not get the install group" ;; + esac + # sticky, or group-write means any package can delete another's home + case "$_ep" in + *17[0-9][0-9]*) ;; + *) _p="$_p;the account roots are group-writable WITHOUT the sticky bit" ;; + esac + case "$(_slice_fn "$helper_src" init_package_user_home)" in + *ensure_pkgusr_roots*) ;; + *) _p="$_p;the account roots are still created as a side effect" ;; + esac + # never_adopt_list must reach them through pkgusr_roots -- a second literal + # list is one more place to forget when a root is added + case "$(_slice_fn "$helper_src" never_adopt_list)" in + *pkgusr_roots*) ;; + *) _p="$_p;never_adopt_list does not cover the account roots" ;; + esac + # and they must NOT be install dirs + if _slice_fn "$helper_src" install_dirs_list | grep -qx '/usr/src/pkgusr'; then + _p="$_p;the package root is an install dir -- any package user could delete a home" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the account roots are created deliberately and never adopted" + fi +fi + +# ---- /sources is normalised on the way into the chroot ---------------------- # +# Book 3.1 wants $LFS/sources root:root 1777. The host `lfs` account comes from +# book 4.3's own useradd, which pins no uid, so it lands wherever the host had a +# gap -- 10753 on a Debian host. Package users start at 10000, so that uid does +# not merely read as a number inside the chroot, it COLLIDES with a real package +# user and tarballs report as owned by something unrelated. +# +# _normalize_sources was only called from get-sources and restart, so a tree set +# up before it existed kept the host uid forever. Entry is the one path every +# build goes through, whatever it did before. +python3 - "$LFS_TOOL" <<'PYSRCNORM' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +problems = [] +if not m: + problems.append("cmd_bs_chroot is gone") +else: + body = m.group(0) + enter = body.split('if action == "enter"') + if len(enter) < 2: + problems.append("chroot enter is gone") + elif "_normalize_sources" not in enter[1]: + problems.append("nothing normalises /sources on chroot entry") + prep = body.split('if action == "prepare"') + if len(prep) > 1 and "_normalize_sources" not in prep[1].split('if action ==')[0]: + problems.append("nothing normalises /sources after the 7.2 handover") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS /sources is given back to root on the way into the chroot") +PYSRCNORM +_count_rc $? + +# ---- a wrapper must never stand in for a privileged tool -------------------- # +# make_wrappers ships a `chgrp` and a `chown` that print a note and exit 0, so a +# PACKAGE USER cannot change ownership. They are ordinary files on $PATH. If +# that directory is ever ahead of /usr/bin for root, root's own privileged calls +# resolve to them too -- and they report success while doing nothing. +# +# A tree ran 34 packages that way: init-pkgusr said +# # 46 install directories now belong to group 'install' +# with an empty failure list, while every one was still root:root. No package +# user could write to /usr/bin, `verify --fix` could not repair it (its chown +# went through the same wrapper), and nothing said why. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # the wrappers really do exit 0 without acting -- that is the hazard + _slice_fn "$helper_src" make_wrappers | grep -q 'package users cannot change ownership' \ + || _p="$_p;the chown wrapper no longer explains itself" + # so ownership and mode must go through the binary, by absolute path + for _fn in real_chown real_chgrp real_chmod; do + _slice_fn "$helper_src" "$_fn" | grep -q . \ + || _p="$_p;there is no $_fn -- a wrapper can shadow root's own calls" + done + _slice_fn "$helper_src" _real_tool | grep -q '/usr/bin/' \ + || _p="$_p;_real_tool does not resolve an absolute path" + # and no privileged pass may call the bare command any more + for _fn in cmd_init_pkgusr init_package_user_home _vfy_install_dirs ensure_pkgusr_roots; do + # only where the tool is INVOKED -- start of a command, or after + # &&/||/;/!. Matching it anywhere caught the warning text that explains + # the hazard, which would have made this test impossible to satisfy. + if _slice_fn "$helper_src" "$_fn" \ + | grep -vE '^[[:space:]]*#' \ + | grep -qE '(^[[:space:]]*|&&[[:space:]]*|\|\|[[:space:]]*|;[[:space:]]*|![[:space:]]*)(chown|chgrp|chmod) '; then + _p="$_p;$_fn still calls a bare chown/chgrp/chmod" + fi + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "root's ownership calls cannot resolve to a package-user wrapper" + fi + + # exit 0 is not proof. init-pkgusr must read the group back. + _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" + _p="" + # owner AND group: the rule is root:install, and reading back only the group + # is how /usr/share sat at lfs:install without anything noticing + case "$_blk" in + *'stat -c %U:%G'*) ;; + *) _p="$_p;init-pkgusr trusts the exit code instead of reading the result back" ;; + esac + case "$_blk" in + *'NO install directories'*) ;; + *) _p="$_p;init-pkgusr reports zero install directories as success" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "init-pkgusr confirms the install group took, and zero is an error" + fi +fi + +# ---- an empty override must not disable the whole scheme -------------------- # +# install_dirs_list is overridable through $INSTALLDIRS and returned early when +# the file existed. An override yielding nothing meant NO directory was given +# to the install group, NO directory was protected from adoption, and `verify` +# reported a clean tree because it walked the same empty list. Nine callers, +# one silent no-op. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( STATE=/tmp/_idl.$$; INSTALLDIRS="$STATE/installdirs.lst" + mkdir -p "$STATE" + eval "$(_slice_fn "$helper_src" install_dirs_list)" + warn() { :; } + # no override at all -> the built-in list + [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ + || echo "the built-in list is empty" + # an override with only comments and blanks -> fall back, never nothing + printf '# just a comment\n\n \n' > "$INSTALLDIRS" + [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ + || echo "an empty override silently disables every install directory" + # a real override is still honoured + printf '/usr/bin\n/usr/lib\n' > "$INSTALLDIRS" + [ "$(install_dirs_list 2>/dev/null | grep -c .)" = 2 ] \ + || echo "a real override is no longer honoured" + rm -rf "$STATE" )" + if [ -z "$_o" ]; then + ok "an override that lists nothing falls back to the built-in list" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- an option is not a package name ---------------------------------------- # +# /usr/src/pkgusr/p_--version turned up in a sanity run: a home for an account +# that could never exist, left by something that reached add-user with +# "--version" in the name position. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _slice_fn "$helper_src" _refuse_option_as_name | grep -q 'looks like an option' \ + && case "$(_slice_fn "$helper_src" cmd_add_user)" in + *_refuse_option_as_name*) ok "a name starting with a dash is refused, not made into a user" ;; + *) bad "add-user does not refuse an option-shaped name" ;; + esac \ + || bad "there is no guard against an option being used as a package name" +fi + +# ---- the sanity report checks the group it prints --------------------------- # +# Section 1 printed owner:group from the start and asserted only the mode. A +# tree ran 34 packages with every install directory at root:root 775 -- +# group-writable, but group `root`, while package users are in `install`. The +# column that would have shown it was decoration. +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q "want 'install'" "$san" \ + || _p="$_p;the sanity report does not check the install group" + grep -q "a package has taken a shared directory" "$san" \ + || _p="$_p;the sanity report does not check who owns a shared directory" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report asserts the install-dir owner and group" + fi +fi + +# ---- one wrapper directory, not two ----------------------------------------- # +# make_wrappers wrote to $STATE/wrappers while /etc/pkgusr/bash_profile -- and +# packagemanager's copy of it -- put /usr/lib/pkgusr first on a package user's +# PATH. Nothing created that, so a package user's login shell led with a +# directory that did not exist, and the wrappers only ever applied because +# cmd_build overrides PATH on the command line. Two names for one decision. +# +# /usr/lib/pkgusr is the survivor: $STATE is build state, and packagemanager on +# the finished system hands its users the same profile. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^WRAPPERS="\${LFS_WRAPPERS:-\${SNAP_ROOT%/}/usr/lib/pkgusr}"' "$helper_src" \ + || _p="$_p;the wrapper directory is not /usr/lib/pkgusr" + grep -q 'WRAPPERS="\$STATE/wrappers"' "$helper_src" \ + && _p="$_p;the old \$STATE/wrappers definition is still there" + # the profile must not name the directory a second time + case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in + *'PATH=/usr/lib/pkgusr:'*) + _p="$_p;the profile still hardcodes the wrapper path instead of using \$WRAPPERS" ;; + esac + case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in + *'s|@WRAPPERS@|$WRAPPERS|'*) ;; + *) _p="$_p;the profile's PATH is not filled in from \$WRAPPERS" ;; + esac + # and it must be protected: under /usr/lib, which IS an install dir, so + # without this any package user could rewrite the wrappers that constrain it + case "$(_slice_fn "$helper_src" never_adopt_list)" in + *'"$WRAPPERS"'*) ;; + *) _p="$_p;the wrapper directory can be adopted or given to the install group" ;; + esac + # root-owned, by NAME or by ID. Before book 7.6 there is no /etc/passwd in + # the tree, so `chown root:root` fails with "invalid user" while `chown 0:0` + # always resolves -- the property is the ownership, not the spelling. + case "$(_slice_fn "$helper_src" make_wrappers)" in + *'chown root:root "$WRAPPERS"'*|*'chown 0:0 "$WRAPPERS"'*) ;; + *) _p="$_p;the wrappers are not explicitly root-owned" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "there is one wrapper directory, and it outlives the build" + fi +fi + +# packagemanager builds the same profile for the finished system, so it must +# reach the same path -- a second hardcoded copy is how these drifted +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$pm_src" ]; then + _p="" + grep -q '^WRAPPERS = os.environ.get("LFS_WRAPPERS", "/usr/lib/pkgusr")' "$pm_src" \ + || _p="$_p;packagemanager has no WRAPPERS setting" + grep -q 'PATH=/usr/lib/pkgusr:' "$pm_src" \ + && _p="$_p;packagemanager still hardcodes the wrapper path in the profile" + grep -q '_PKGUSR_BASH_PROFILE.replace("@WRAPPERS@", WRAPPERS)' "$pm_src" \ + || _p="$_p;packagemanager does not fill the wrapper path into the profile" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both tools write a profile pointing at the same wrapper directory" + fi +fi + +# ---- an install directory is root:install, owner AND group ------------------- # +# init-pkgusr set only the GROUP, while _vfy_install_dirs demands root:install +# and `verify --fix` applies both. Two rules for one directory. +# +# They disagreed on a real build: chapters 5-6 build as the `lfs` user, and book +# 7.2's handover only reaches one level below $LFS, so /usr/share stayed +# lfs-owned and came out +# lfs:install drwxrwxr-x /usr/share +# The group looked right and nothing said the owner was not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" + _p="" + # owner AND group -- through the shared rule, which applies both by number. + # This used to look for `real_chown root:install`, which is the by-name form + # that cannot work before book 7.6 writes /etc/passwd. + case "$_blk" in + *'set_install_dir_owner "$d"'*) ;; + *) _p="$_p;init-pkgusr sets the group but not the owner of an install directory" ;; + esac + case "$_blk" in + *'stat -c %U:%G'*) ;; + *) _p="$_p;init-pkgusr reads back only the group, so a wrong owner goes unnoticed" ;; + esac + # the two rules must want the same thing + case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in + *root:install*) ;; + *) _p="$_p;verify no longer wants root:install" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "init-pkgusr and verify agree that an install dir is root:install" + fi +fi + +# ---- every tool we install gets an owner ------------------------------------- # +# install_helper copies lfs/lfs-helper/packagemanager/blfs and calls +# _chown_tools. _install_pkgusr_helpers copied the hint's scripts beside them +# and never did -- so /usr/bin/list_package, uninstall_package, +# forall_direntries_from and the rest came out root:root while the four tools +# next to them belonged to p_pkgusr. Two copy routines, one claiming ownership. +# +# The WRAPPER directory is the deliberate exception: it stays root:root, because +# a package user that could rewrite chown or install could rewrite the rule that +# constrains it. +python3 - "$LFS_TOOL" <<'PYHELPEROWN' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _install_pkgusr_helpers\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_install_pkgusr_helpers is gone") +else: + body = m.group(0) + if "_chown_tools" not in body: + problems.append("the hint's helpers are copied but never given an owner") + if "wrapper_sub" not in body: + problems.append("the wrapper directory is not excluded from the chown") +if not re.search(r'^WRAPPERS = os\.environ\.get\("LFS_WRAPPERS"', src, re.M): + problems.append("lfs does not read the shared WRAPPERS setting") +# and the helper table must not spell the wrapper path out a fourth time +tbl = re.search(r"PKGUSR_HELPERS = \[(.*?)\n\]", src, re.S) +if tbl and '"usr/lib/pkgusr"' in tbl.group(1): + problems.append("PKGUSR_HELPERS still hardcodes the wrapper path") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every tool installed into the tree is given an owner") +PYHELPEROWN +_count_rc $? + +# ---- scratch never enters a manifest ---------------------------------------- # +# $BUILD_ROOT is the busiest directory on the system during a build: unpacked +# source trees, object files, the tarballs themselves. Every one is newer than +# the build stamp, so the touched-file scan swallowed the whole scratch tree +# into every package's manifest and then chowned it -- and the next package +# chowned it right back. A real `verify` came back with 3363 wrong-owner, of +# which 3348 were /build: +# /build/bash-5.3.tar.gz is p_shadow want p_acl +# The two snapshot scans already excluded it; the build's own scan did not. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cb="$(_slice_fn "$helper_src" cmd_build)" + _p="" + case "$_cb" in + *'-not -path "$_bld/*"'*) ;; + *) _p="$_p;the build scratch is still scanned into every manifest" ;; + esac + case "$_cb" in + *'BUILD_ROOT:-$r/build'*) ;; + *) _p="$_p;the scratch exclusion does not follow \$BUILD_ROOT" ;; + esac + # and verify must ignore scratch left in manifests written before this + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'BUILD_ROOT'*) ;; + *) _p="$_p;verify still reports scratch recorded by an older build" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build scratch is never recorded as a package's files" + fi +fi + +# ---- some files belong to no package ---------------------------------------- # +# Every package that installs an info page rewrites /usr/share/info/dir, and +# ldconfig rewrites /etc/ld.so.cache after every install. Recording those in a +# manifest makes verify report a conflict no repair can settle -- the next +# package takes them straight back: +# /usr/share/info/dir is p_gcc want p_binutils +# /etc/ld.so.cache is p_libcap want p_glibc +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SNAP_ROOT=/ + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + for f in /usr/share/info/dir /etc/ld.so.cache /var/cache/ldconfig/aux-cache; do + is_never_claimed "$f" || echo "$f can still be claimed by a package" + done + # the wrapper directory too: it is root:root on purpose + is_never_claimed /usr/lib/pkgusr/chown \ + || echo "a wrapper can be claimed by a package" + # but a real package file must NOT be swept up by this + is_never_claimed /usr/bin/bash \ + && echo "an ordinary package file is treated as unclaimable" )" + if [ -z "$_o" ]; then + ok "shared indexes and generated caches are owned by no package" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- the 7.2 handover is decided by the tree, not one inode ------------------ # +# `_owner_is_root` looks at $LFS/usr. The pass it guarded is +# `chown -h -R root:root` over EIGHT trees. A tree whose /usr was already +# root-owned got no handover at all, and everything chapters 5-6 created below +# it kept the host uid: +# drwxr-xr-x 1 lfs lfs /usr/share/gcc-15.2.0/python/libstdcxx +# lfs:install drwxrwxr-x /usr/share +# It must also refuse to run once package users own files -- chown -R root:root +# over /usr is right before chapter 8 and catastrophic after it. +python3 - "$LFS_TOOL" <<'PYHANDOVER' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +for fn in ("_handover_needed", "_handover_is_safe"): + if not re.search(r"^def %s\(" % fn, src, re.M): + problems.append("there is no %s" % fn) +m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if m and "_find_uid" not in m.group(0): + problems.append("_handover_needed still decides from a single directory") +m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) +# It must ask the TREE's passwd which accounts are package users, not infer it +# from a uid range: book 4.3 pins no uid for `lfs`, so on one host the build +# account came out 10753 -- inside the package range -- and a range filter +# excluded the very user it was looking for. +if m and "_pkg_users_in_tree" not in m.group(0): + problems.append("_handover_is_safe infers package users from a uid range") +if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): + problems.append("there is no _pkg_users_in_tree") +m2 = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if m2 and "PKG_UID_MIN" in m2.group(0): + problems.append("_handover_needed still filters the build user out by uid range") +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + if "_handover_needed" not in body: + problems.append("chroot prepare still guards 7.2 on _owner_is_root alone") + if "_handover_is_safe" not in body: + problems.append("chroot prepare would run chown -R over a chapter-8 tree") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the 7.2 handover is decided by the tree and refuses a built one") +PYHANDOVER +_count_rc $? + +# and a tree that already has the damage must be repairable +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q '\-user lfs' \ + || _p="$_p;verify cannot find files the build user still owns" + _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q 'real_chown -h root:root' \ + || _p="$_p;verify --fix cannot hand the build user's leftovers back to root" + case "$(_slice_fn "$helper_src" cmd_verify)" in + *_vfy_build_user_leftovers*) ;; + *) _p="$_p;verify does not run the build-user pass" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a tree the handover skipped can be repaired from inside the chroot" + fi +fi + +# ---- every function called is a function that exists ------------------------ # +# `verify --fix` called cmd_fix_orphans, which does not exist -- `fix-orphans` +# is dispatched as an ALIAS for `verify --fix`, so the call was both undefined +# and circular. It died mid-repair: +# line 4793: cmd_fix_orphans: command not found +# after chowning 3473 paths, leaving the pass half-done and the status wrong. +# +# A missing function in the REPAIR path is the worst place for one: it only +# fires when something is already broken, which is when it is least welcome. +for _t in lfs-helper; do + _src="$(dirname "$LFS_TOOL")/$_t" + [ -f "$_src" ] || continue + _missing="" + # comments stripped first: the fix for this bug NAMES the function it + # removed, and matching that text would make the test permanently red + for _fn in $(grep -vE '^[[:space:]]*#' "$_src" \ + | grep -oE '(^|[^_[:alnum:]])cmd_[a-z0-9_]+' \ + | grep -oE 'cmd_[a-z0-9_]+' | sort -u); do + grep -qE "^${_fn}\(\) \{" "$_src" || _missing="$_missing $_fn" + done + if [ -n "$_missing" ]; then + for _m in $_missing; do bad "$_t calls $_m, which is not defined"; done + else + ok "$_t defines every cmd_* function it calls" + fi +done + +# ---- a package builds in its own home, not a shared scratch ----------------- # +# /build was a permanent ownership fight. Every build's touched-file scan swept +# the whole tree in, so each manifest claimed every other package's sources and +# chowned them; the next package chowned them back. Repairing it made it worse, +# because the question had no answer: +# /build/bash-5.3.tar.gz is p_shadow want p_acl +# /build/bash-5.3.tar.gz is p_zstd want p_acl (after the repair) +# +# Under the package user's home it is settled by construction: the user creates +# the tree so the user owns it, and $SRCROOT is already excluded from every tree +# scan, so it can never reach a manifest at all. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( SNAP_ROOT=/; SRCROOT=/usr/src + PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg + PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" + STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" + PKGUSR_BUILD_SUBDIR=src + for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ + pkg_owner_name pkgusr_home_for _is_not_a_package build_root_for; do + eval "$(_slice_fn "$helper_src" "$fn")" + done + if ! command -v build_root_for >/dev/null 2>&1; then + echo "there is no build_root_for -- the scratch is still shared" + exit 0 # the rest would be noise, not findings + fi + # a package builds under its own account's home + [ "$(build_root_for gcc)" = "/usr/src/pkgusr/p_gcc/$PKGUSR_BUILD_SUBDIR" ] \ + || echo "gcc does not build in its own home: $(build_root_for gcc)" + # an earlier stage folds onto the same package, so it reuses that tree + [ "$(build_root_for util-linux-tmp)" = "$(build_root_for util-linux)" ] \ + || echo "an earlier stage builds somewhere else than its package" + # and it must be under $SRCROOT, which every tree scan already excludes -- + # that is what keeps it out of manifests without another find exclusion + case "$(build_root_for gcc)" in + "$SRCROOT"/*) ;; + *) echo "the build tree is not under SRCROOT, so scans will sweep it in" ;; + esac + # a step with no account keeps the shared scratch: it runs as root, which + # owns everything there anyway + case "$(build_root_for init-dirs)" in + /usr/src/*) echo "a step with no account was given a home to build in" ;; + esac )" + if [ -z "$_o" ]; then + ok "each package unpacks and builds inside its own account's home" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # and the build must actually be told about it + case "$(_slice_fn "$helper_src" cmd_build)" in + *"BUILD_ROOT='\$_bldroot'"*) + ok "the per-package build root is passed into the build" ;; + *) bad "the build still inherits whatever BUILD_ROOT it was started with" ;; + esac +fi + +# ---- the root symlinks are chowned, not followed ---------------------------- # +# Book 4.2 creates /bin -> usr/bin, /lib and /sbin as the build user. They came +# out lfs:lfs, and a chown WITHOUT -h follows the link and retargets /usr/bin +# instead -- silently doing the wrong thing to the right-looking path. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _blk="$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" + _p="" + case "$_blk" in + *'real_chown -h'*) ;; + *) _p="$_p;the build-user repair follows symlinks instead of chowning them" ;; + esac + case "$(printf '%s\n' "$_blk" | grep -vE '^[[:space:]]*#')" in + *'-type f'*) _p="$_p;the build-user scan skips symlinks and directories" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a symlink owned by the build user is chowned, not followed" + fi +fi + +# ---- the tree can be set up before book 7.6 exists -------------------------- # +# A chapter 5-6 snapshot restored before 7.6 has NO /etc/passwd -- the chroot +# prompt says "I have no name!". `chown root:install` needs `root` to resolve, +# so every call failed and init-pkgusr reported: +# # 0 install directories are now root:install (group-writable) +# # could not set the group on: /usr /usr/bin /usr/lib /usr/share ... +# while ensure_install_dirs_writable used `chgrp` and worked fine. Two rules +# for one directory, and only one of them survived an empty passwd file. +# +# Ownership is stored as a NUMBER, so use the number. +_p="" +_pre="$T/pre76" +mkdir -p "$_pre/etc" "$_pre/usr/share/man/man3" +: > "$_pre/etc/passwd" # exists, but empty: no root entry +printf 'install:x:9999:\n' > "$_pre/etc/group" +# the shape that broke: a name that cannot resolve +if chown root:install "$_pre/usr/share/man/man3" 2>/dev/null; then + : # this host HAS a root and an install group, so the fixture proves nothing +else + # numeric must work where the name did not + if chown -h 0:9999 "$_pre/usr/share/man/man3" 2>/dev/null; then + [ "$(stat -c '%u:%g' "$_pre/usr/share/man/man3")" = "0:9999" ] \ + || _p="$_p;a numeric chown did not take" + else + _p="$_p;even a numeric chown fails here -- fixture is not usable" + fi +fi +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # one helper, used by all three passes that set an install directory + _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ + || _p="$_p;the install-dir owner is not set by number" + _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ + || _p="$_p;the install-dir check reads names, which are UNKNOWN before 7.6" + for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do + _slice_fn "$helper_src" "$_fn" | grep -q 'set_install_dir_owner' \ + || _p="$_p;$_fn does not use the shared install-dir rule" + done + # and no pass may still reach for the names + for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do + _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ + | grep -qE 'chown root:install|chgrp install ' \ + && _p="$_p;$_fn still sets an install dir by name" + done +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "install directories are set by number, so a tree with no passwd works" +fi + +# ---- nothing is an orphan before there is a passwd file --------------------- # +# `find -nouser` asks whether a uid resolves. With no passwd file NOTHING +# resolves, so every file in the tree is an "orphan". On a restored chapter 5-6 +# tree that reported +# noowner / (no passwd entry) -> lfs +# noowner /dev (no passwd entry) -> lfs +# ... and 11797 more with no owner +# and `--fix` would have handed the ENTIRE tree -- root's own directories +# included -- to the build user in one pass. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ + || _p="$_p;$_fn runs before the user database exists" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the ownership passes wait for book 7.6 to write /etc/passwd" + fi +fi + +# ---- nothing privileged may need a name to resolve -------------------------- # +# init-pkgusr runs BEFORE book 7.6 creates /etc/passwd. The prompt in there +# literally reads "I have no name!" because not even root resolves. So +# `chown root:install` fails on every directory: +# # 0 install directories are now root:install (group-writable) +# # could not set the group on: /bin /sbin /lib /lib64 /etc /usr ... +# and every package user is then locked out of /usr/bin for the whole build. +# +# Ownership is stored as a NUMBER. uid 0 is root whether anything says so or +# not, and the install gid is ours -- init-pkgusr writes /etc/group by hand, so +# the group can exist long before any passwd entry does. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ + || _p="$_p;an install directory is set by name, which fails before 7.6" + _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ + || _p="$_p;the readback uses names, so a correct directory reads as wrong" + # No privileged pass may reach for the NAMES. Match the INVOCATION, not the + # string: these functions legitimately print "now root:install" and carry + # trailing comments saying the same, and matching those made the test + # impossible to satisfy while changing nothing about the behaviour. + for _fn in cmd_init_pkgusr _vfy_install_dirs ensure_install_dirs_writable; do + _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ + | grep -qE '(real_)?(chown|chgrp)( +-[a-zA-Z]+)* +(root:install|install) ' \ + && _p="$_p;$_fn still sets the group by name, which fails before 7.6" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "install directories are set by number, so 7.6 is not a prerequisite" + fi + + # and the orphan pass must not fire before there is a database to be + # orphaned FROM. With no passwd file `find -nouser` matches everything: a + # real run reported 11802 orphans including / and /proc, and --fix would have + # handed the entire tree to the build user in one pass. + _p="" + case "$(_slice_fn "$helper_src" _vfy_orphans)" in + *_user_db_ready*) ;; + *) _p="$_p;the orphan pass would hand the whole tree to the build user before 7.6" ;; + esac + case "$(_slice_fn "$helper_src" _has_orphaned_files)" in + *_user_db_ready*) ;; + *) _p="$_p;a tree with no passwd file is reported as entirely orphaned" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "an orphan needs a user database to be missing from" + fi +fi + +# ---- the build user's uid is not a range ------------------------------------ # +# Book 4.3's useradd pins no uid, so the host's `lfs` account lands wherever the +# host had a gap -- 10753 on a Debian host, INSIDE the package-user range. Any +# rule that identifies the build user by "uid outside 10000+" therefore excludes +# the very account it is looking for. That is the same mistake _normalize_sources +# exists to correct, and _handover_needed walked straight into it: it reported +# # 7.2 already done (tree owned by root) +# while /bin -> usr/bin was still lfs:lfs. +python3 - "$LFS_TOOL" <<'PYUIDRANGE' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_handover_needed is gone") +else: + body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) + if "PKG_UID_MIN" in body: + problems.append("_handover_needed still identifies the build user by uid range") +m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_handover_is_safe is gone") +else: + body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) + if "_pkg_users_in_tree" not in body: + problems.append("_handover_is_safe does not read the tree's own passwd") +if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): + problems.append("there is no _pkg_users_in_tree") +else: + m = re.search(r"^def _pkg_users_in_tree\(.*?(?=\n\ndef |\Z)", src, re.S | re.M) + if m and "pkgusr_prefix" not in m.group(0): + problems.append("_pkg_users_in_tree does not identify accounts by prefix") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS the build user is identified by the tree, never by a uid range") +PYUIDRANGE +_count_rc $? + +# ---- one step where ownership becomes real ---------------------------------- # +# Before book 7.6 there is no /etc/passwd, so no name resolves -- not even +# root's; the prompt in the chroot literally reads "I have no name!". Every +# ownership question asked before that point gets a confident wrong answer: +# +# chown root:install fails on every directory -> "0 install directories" +# find -nouser matches every file in the tree -> 11802 "orphans" +# +# Three separate bugs, one cause: work that needs a user database was being done +# before there was one. So there is one gate now. Nothing touches ownership +# before it; everything maintains ownership after it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in ownership_established ownership_possible establish_ownership \ + _ownership_checkpoint cmd_establish_ownership; do + _slice_fn "$helper_src" "$_fn" | grep -q . \ + || _p="$_p;there is no $_fn" + done + # the gate must test the user database and the tools it needs + _op="$(_slice_fn "$helper_src" ownership_possible)" + case "$_op" in + *_user_db_ready*) ;; + *) _p="$_p;the gate does not wait for the user database" ;; + esac + case "$_op" in + *chown*) ;; + *) _p="$_p;the gate does not check that chown is available" ;; + esac + # it must be durable, so a resumed or restored build does not repeat it + grep -q '^OWNERSHIP_MARK=' "$helper_src" \ + || _p="$_p;crossing the epoch is not recorded, so it repeats" + case "$(_slice_fn "$helper_src" establish_ownership)" in + *'> "$OWNERSHIP_MARK"'*) ;; + *) _p="$_p;establish_ownership never writes its marker" ;; + esac + # and everything ownership-related must be inside it, not scattered + _eo="$(_slice_fn "$helper_src" establish_ownership)" + for _need in cmd_init_pkgusr _vfy_build_user_leftovers _vfy_manifest_ownership \ + _adopt_pkgusr_tools _vfy_orphans; do + case "$_eo" in + *"$_need"*) ;; + *) _p="$_p;$_need does not happen at the ownership epoch" ;; + esac + done + # reachable by hand as well as automatically + grep -q 'establish-ownership)' "$helper_src" \ + || _p="$_p;there is no way to run the ownership step by hand" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "ownership is established once, at one step, and recorded" + fi + + # the passes that need a user database must refuse to run without one + _p="" + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ + || _p="$_p;$_fn would run before there is a user database" + done + case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in + *ownership_*) ;; + *) _p="$_p;verify would report 27 wrong install dirs on a chapter 5-6 tree" ;; + esac + # build-all must NOT set the package-user system up before the epoch -- + # that is what printed "0 install directories are now root:install" + case "$(_slice_fn "$helper_src" cmd_build_all)" in + *ownership_established*) ;; + *) _p="$_p;build-all still sets up ownership before 7.6 has run" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "nothing asks who owns what before there is a user database" + fi +fi + +# ---- the build tree must not collide with the skeleton ---------------------- # +# init_package_user_home symlinks .bash_profile, .bashrc and `build` into every +# package user's home -- `build` being the hint's build helper script. The +# per-package build tree was first called `build` too, so `mkdir -p ~/build` hit +# the symlink and the build died two lines later complaining about the wrong +# thing: +# mkdir: cannot create directory '.../p_gettext/build': File exists +# cd: /usr/src/pkgusr/p_gettext/build: Not a directory +# The recovery code then read that as a permissions problem and retried twice +# before advising collector groups. None of it was true. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" pkgusr_skel_links | grep -q . \ + || _p="$_p;the skeleton links are not named in one place" + # the home skeleton must be built FROM that list, not from a second copy + case "$(_slice_fn "$helper_src" init_package_user_home)" in + *pkgusr_skel_links*) ;; + *) _p="$_p;init_package_user_home keeps its own copy of the link list" ;; + esac + # and the build subdir must not be one of them + _sub="$(grep -m1 '^PKGUSR_BUILD_SUBDIR=' "$helper_src" \ + | sed 's/.*:-\([a-z_]*\)}.*/\1/')" + if [ -z "$_sub" ]; then + _p="$_p;there is no PKGUSR_BUILD_SUBDIR" + else + for _l in $(_slice_fn "$helper_src" pkgusr_skel_links \ + | grep -oE "'[^']+'|[a-z._:]+" | tr ' ' '\n'); do + case "$_l" in + "$_sub":*|"$_sub") _p="$_p;the build tree is called '$_sub', which the home skeleton already links" ;; + esac + done + fi + # and a name that is taken must be reported HERE, not as a cd failure later + case "$(_slice_fn "$helper_src" cmd_build)" in + *'is not a directory'*) ;; + *) _p="$_p;a build tree that is not a directory fails somewhere else" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build tree cannot collide with the package user's home skeleton" + fi +fi + +# ---- no function is defined twice ------------------------------------------ # +# `cmd_verify` was defined TWICE. bash keeps the last, so the tidy definition +# that called the _vfy_* helpers had never run -- and the surviving one carried +# its own inline copies of the manifest and install-dir rules, without any of +# the guards the helpers had grown. +# +# A finished 104/104 build therefore reported three things the real pass knows +# to leave alone, and --fix would have acted on all three: +# /usr/bin/hostname is p_inetutils want p_coreutils (inetutils owns it) +# /usr/share/info/dir is p_e2fsprogs want p_glibc (nobody owns it) +# /etc/group- is root want p_ncurses (useradd wrote it) +# +# A duplicate definition is invisible: both parse, one silently wins. +for _t in lfs-helper; do + _src="$(dirname "$LFS_TOOL")/$_t" + [ -f "$_src" ] || continue + _dups="$(grep -oE '^[a-z_][a-z0-9_]*\(\) \{' "$_src" | sort | uniq -d)" + if [ -n "$_dups" ]; then + printf '%s\n' "$_dups" | while IFS= read -r _d; do + bad "$_t defines ${_d%\(\) \{} more than once -- one of them never runs" + done + else + ok "$_t defines each function exactly once" + fi +done + +# and verify must USE the helpers rather than carry its own copy of the rules +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cv="$(_slice_fn "$helper_src" cmd_verify)" + _p="" + for _need in _vfy_install_dirs _vfy_manifest_ownership _vfy_orphans \ + _vfy_user_homes _vfy_build_user_leftovers _vfy_user_order; do + case "$_cv" in + *"$_need"*) ;; + *) _p="$_p;verify does not run $_need" ;; + esac + done + # no second copy of the ownership rule inline + case "$(printf '%s\n' "$_cv" | grep -vE '^[[:space:]]*#')" in + *'want $owner'*) _p="$_p;verify still carries its own copy of the ownership rule" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify runs the one implementation of each check" + fi +fi + +# ---- a collector group is named for the package ----------------------------- # +# It took the ACCOUNT name, so owner `p_perl` produced `nimgnu_p_perl` -- two +# prefixes, and the wrong two: `nimgnu_` says collector group, `p_` says package +# user, which it is not. The prompt offered it as a choice: +# 1) nimgnu_p_perl (everything owned by 'p_perl' -- fewer groups) +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( COLLECTOR_PREFIX=nimgnu; PKGUSR_PREFIX=p_ + eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" + eval "$(_slice_fn "$helper_src" collector_group_for)" + [ "$(collector_group_for p_perl)" = "nimgnu_perl" ] \ + || echo "a collector group still carries the package-user prefix: $(collector_group_for p_perl)" + # a bare package name is unchanged + [ "$(collector_group_for perl)" = "nimgnu_perl" ] \ + || echo "a bare package name does not reach the same group" + # a config step keeps its own prefix -- cfg_ is part of the step's name + [ "$(collector_group_for cfg_bootscripts)" = "nimgnu_cfg_bootscripts" ] \ + || echo "a config step's group name lost its kind: $(collector_group_for cfg_bootscripts)" )" + if [ -z "$_o" ]; then + ok "a collector group carries the collector prefix and no other" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- root's own files are root's ------------------------------------------- # +# Every cmd_add_user during a build writes /etc/passwd, /etc/group and Shadow's +# backups of them. Whichever package happened to be building at the time swept +# them into its manifest: +# /etc/.pwd.lock is root want p_ncurses +# /etc/group- is root want p_ncurses +# /etc/passwd- is root want p_psmisc +# ncurses did not install /etc/group-; useradd did, while ncurses was building. +# A package owning the file that says which packages exist is the one ownership +# mistake with no way back. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/ + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + for f in /etc/passwd /etc/passwd- /etc/group /etc/group- \ + /etc/shadow /etc/gshadow /etc/.pwd.lock; do + is_never_claimed "$f" || echo "$f can be claimed by a package" + done + # ordinary configuration is still a package's to own + is_never_claimed /etc/fstab \ + && echo "an ordinary config file was swept into the never-claim list" )" + if [ -z "$_o" ]; then + ok "the user database belongs to root and to no package" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the tools we install are ours, in bin AND sbin + _p="" + grep -q 'add_package_user install_package' "$helper_src" \ + || _p="$_p;the hint's helper scripts are not claimed by p_pkgusr" + case "$(_slice_fn "$helper_src" _adopt_pkgusr_tools)" in + *'/usr/bin /usr/sbin'*) ;; + *) _p="$_p;only /usr/bin is searched, so the sbin helpers stay unclaimed" ;; + esac + # and the wrappers must not be reported as needing an owner + case "$(_slice_fn "$helper_src" _vfy_unclaimed)" in + *is_never_claimed*) ;; + *) _p="$_p;verify asks who should own the wrappers, which are root's by design" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "our own tools get an owner; the wrappers stay root's" + fi +fi + +# ---- book 4.3 must not run on a built tree ---------------------------------- # +# `_chown_tree_to_lfs` is `chown -R lfs` over /usr, /etc, /var and the rest. +# Before chapter 5 that is book 4.3 and exactly right. On a BUILT tree it hands +# the finished system to the host's build account in one pass, and every +# package's ownership is gone -- recoverable only from the manifests. +# +# It happened: `lfs build-system run` on a 104/104 tree replayed step 4.2 and +# drwxrwxr-t 1 lfs install /mnt/lfs/usr +# came back with the host uid 10753 written through the tree. Inside the chroot +# that resolves to nothing -- the chroot's own `lfs` is 9998 -- so `ls -al /` +# showed a raw number where every owner should have been. +python3 - "$LFS_TOOL" <<'PY43' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def _chown_tree_to_lfs\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("_chown_tree_to_lfs is gone") +else: + body = m.group(0) + if "_handover_is_safe" not in body: + problems.append("book 4.3 would chown a built tree to the build user") + # the guard has to come BEFORE the chown, or it reports and then does it + # the INVOCATION, not the comment above the guard that names the command + g = body.find("if not _handover_is_safe") + c = body.find("_run_bash('chown -R lfs") + if c < 0: + problems.append("the 4.3 chown is no longer recognisable") + elif g < 0 or g > c: + problems.append("the 4.3 guard runs after the chown it is guarding") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS book 4.3 refuses to hand a built tree back to the build user") +PY43 +_count_rc $? + +# ---- the ownership step is visible ------------------------------------------ # +# It ran as an invisible checkpoint inside build-all: right place, right work, +# but it never appeared in `lfs-helper list`. The one moment the whole scheme +# turns on was the one moment you could not point at, resume from, or re-run. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # lfs puts it in the step order, straight after book 7.6 + grep -q 'order.insert(order.index("init-files") + 1, "init-ownership")' "$LFS_TOOL" \ + || _p="$_p;init-ownership is not in the step order" + # lfs-helper performs it itself -- there is no script for it in the book + case "$(_slice_fn "$helper_src" cmd_build)" in + *'"$name" = "init-ownership"'*) ;; + *) _p="$_p;lfs-helper would look for a book script for init-ownership" ;; + esac + # it owns no files, so it gets no account and no home + case "$(_slice_fn "$helper_src" _is_not_a_package)" in + *init-ownership*) ;; + *) _p="$_p;init-ownership would be given a package user" ;; + esac + # and running it must record itself, so list and next agree with reality + case "$(_slice_fn "$helper_src" cmd_build)" in + *'mark_done "$name"'*) ;; + *) _p="$_p;running init-ownership does not mark it built" ;; + esac + case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in + *mark_done*) ;; + *) _p="$_p;a tree that crossed the epoch still shows the step as pending" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the ownership step appears in the list, and can be run on its own" + fi +fi + +# ---- a chapter-8 tree can still be taken back from the build user ----------- # +# Refusing the recursive 7.2 chown on a built tree is right -- it would strip +# every package user in one pass. But refusing and stopping there left the tree +# with NO way forward: +# +# 4.3 had taken /usr, /etc, /var and the root symlinks, +# 7.2 said "not touching it, repair from the manifests instead", +# chroot enter said "the tree is not owned by root yet: run chroot prepare" +# +# -- pointing at the command that had just declined. The precise repair is to +# reclaim what the BUILD USER holds, by uid: that leaves every package user's +# files alone, so it is safe at any stage. +python3 - "$LFS_TOOL" <<'PYRECLAIM' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +for fn in ("_build_user_uid", "_handover_build_user_only"): + if not re.search(r"^def %s\(" % fn, src, re.M): + problems.append("there is no %s" % fn) +m = re.search(r"^def _handover_build_user_only\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + # selects by uid, so package users are untouched + if "-uid" not in body: + problems.append("the reclaim does not select on the build user's uid") + # user only: 4.3 chowned the user, so the inverse must not take the group + # as well -- that would strip `install` off every shared directory + # the INVOCATION -- the docstring above it explains why the group is left + # alone, and matching that text made this impossible to satisfy + code = "\n".join(l for l in body.splitlines() + if not l.lstrip().startswith("#")).split('"""') + code = "".join(code[::2]) + if "chown -h root:root" in code or "chown -R" in code: + problems.append("the reclaim takes the group too, undoing the install group") + if "chown -h root" not in body: + problems.append("the reclaim follows symlinks instead of chowning them") +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + body = m.group(0) + if "_handover_build_user_only" not in body: + problems.append("a chapter-8 tree is still left with no way forward") + # and entering must repair rather than point back at prepare + enter = body.split('if action == "enter"') + if len(enter) > 1 and "_handover_build_user_only" not in enter[1]: + problems.append("chroot enter still sends you to a command that declined") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS a built tree can be taken back from the build user without loss") +PYRECLAIM +_count_rc $? + +# end to end: the build user's paths go back to root, the package user's do not +if [ "$(id -u)" = 0 ] && command -v python3 >/dev/null 2>&1; then + _fx="$T/reclaim/mnt/lfs" + mkdir -p "$_fx/usr/bin" "$_fx/etc" "$_fx/var" "$_fx/tools" + ( cd "$_fx" && ln -sf usr/bin bin ) + echo pkg > "$_fx/usr/bin/owned-by-a-package" + echo cfg > "$_fx/etc/fstab" + # 4242 stands in for a package user, 4243 for the host's build account + chown 4242:4242 "$_fx/usr/bin/owned-by-a-package" + for _d in usr usr/bin etc var tools etc/fstab; do chown 4243 "$_fx/$_d"; done + chown -h 4243 "$_fx/bin" + _out="$(python3 - "$LFS_TOOL" "$_fx" <<'PYFX' +import sys, importlib.util +src = open(sys.argv[1]).read().split('if __name__ ==')[0] +spec = importlib.util.spec_from_loader("t", None) +m = importlib.util.module_from_spec(spec) +exec(compile(src, "lfs", "exec"), m.__dict__) +m._build_user_uid = lambda: 4243 +print(m._handover_build_user_only(sys.argv[2], run=True)) +PYFX +)" + _bad="" + [ "$(stat -c %u "$_fx/usr")" = 0 ] || _bad="$_bad;/usr was not taken back" + [ "$(stat -c %u "$_fx/bin")" = 0 ] || _bad="$_bad;the root symlink was not taken back" + [ "$(stat -c %u "$_fx/usr/bin/owned-by-a-package")" = 4242 ] \ + || _bad="$_bad;a package user's file was taken from it" + [ -L "$_fx/bin" ] || _bad="$_bad;the symlink was followed and replaced" + if [ -n "$_bad" ]; then + printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "reclaiming takes the build user's paths and leaves the packages' alone" + fi + rm -rf "$T/reclaim" +fi + +# ---- state lives under /usr/src, and it is sorted --------------------------- # +# It was /.lfs-pkgusr: a hidden directory at the root of the filesystem, sitting +# next to /boot and /etc as if it were part of the system being built. It is +# not -- it is this toolchain's working notes ABOUT that system -- and it holds +# twenty-two loose files at the top level. +# +# Under /usr/src it sits beside the accounts it describes, and the tree scans +# exclude ONE path instead of two, because everything under $SRCROOT already is. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + grep -q '^STATE="\${LFS_PKGUSR_DIR:-\${SRCROOT%/}/lfs-pkgusr}"' "$helper_src" \ + || _p="$_p;lfs-helper state is not under \$SRCROOT" + grep -q '^PKGUSR_DIR = "usr/src/lfs-pkgusr"' "$LFS_TOOL" \ + || _p="$_p;lfs state is not under /usr/src" + # both tools must name the SAME directory -- they read each other's manifests + case "$(grep -m1 '^STATE=' "$helper_src")" in + *lfs-pkgusr*) ;; + *) _p="$_p;the two tools no longer agree on where state lives" ;; + esac + # SRCROOT has to be defined before what derives from it: `set -u` turns a + # forward reference into "SRCROOT: unbound variable" and the tool will not + # even print its version + _isrc="$(grep -n '^SRCROOT=' "$helper_src" | head -1 | cut -d: -f1)" + _ista="$(grep -n '^STATE=' "$helper_src" | head -1 | cut -d: -f1)" + if [ -n "$_isrc" ] && [ -n "$_ista" ] && [ "$_isrc" -gt "$_ista" ]; then + _p="$_p;SRCROOT is defined after the paths derived from it" + fi + # nothing loose at the top: every state file belongs to a named group + for _v in PROGRESS SNAPSHOT SNAPDIRS INSTALLDIRS; do + case "$(grep -m1 "^$_v=" "$helper_src")" in + *'$STATE_PROGRESS/'*|*'$STATE_CONF/'*|*'$STATE_GROUPS/'*) ;; + *) _p="$_p;$_v is still a loose file at the top of the state dir" ;; + esac + done + # and the directories are created together, so no path can find one missing + case "$(_slice_fn "$helper_src" mkstate)" in + *STATE_PROGRESS*STATE_GROUPS*STATE_CONF*) ;; + *) _p="$_p;mkstate does not create every state directory" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "state lives under /usr/src, sorted into named directories" + fi +fi + +# ---- colour means one thing --------------------------------------------------# +# Four colours, one meaning each, the same in both tools -- they print into the +# same terminal one after the other, and a log where the same colour means two +# different things is worse than no colour at all. Prose gets none: colour on +# every other line stops carrying information. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in say ok warn note detail hint die fail; do + grep -qE "^$_fn\(\) +\{" "$helper_src" \ + || _p="$_p;lfs-helper has no $_fn" + done + # say() is prose and must not paint anything + case "$(grep -m1 '^say()' "$helper_src")" in + *C_*) _p="$_p;plain prose is coloured" ;; + esac + # the vocabulary is the interface: no ad-hoc colour outside those helpers + # the helper DEFINITIONS are where the colour lives; everything else must go + # through them + _adhoc="$(grep -nE '(say|echo) "\$\{C_(DIM|WARN|ERR|OK)\}' "$helper_src" \ + | grep -vE '^[0-9]+:(ok|warn|note|detail|hint|die|fail)\(\)' | wc -l)" + [ "${_adhoc:-0}" = 0 ] \ + || _p="$_p;$_adhoc line(s) still paint colour by hand instead of using a helper" + # lfs must offer the same set, or the two halves of a build read differently + for _fn in warn note ok fail hint; do + grep -qE "^def $_fn\(" "$LFS_TOOL" || _p="$_p;lfs has no $_fn" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both tools share one colour vocabulary, and prose has none" + fi +fi + +# ---- the collector export must land where the reader looks ------------------ # +# `lfs` copies the export INTO the tree; `lfs-helper` reads it back out. Those +# two paths are the whole mechanism -- nothing else connects them -- and when +# the state directory was sorted, the writer kept using the top level while the +# reader moved into groups/. A rebuild with a perfectly good export then asked +# +# 'p_gcc' needs to install into: /usr/lib/bfd-plugins +# that directory belongs to package 'p_binutils' +# Which group should share it? +# +# for a directory the file already answered. Silent, because a question that +# gets asked looks the same as a question that was never answered. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + # what the writer writes + _w="$(python3 - "$LFS_TOOL" <<'PYW' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def _install_collector_groups\(.*?(?=\n\ndef )", src, re.S | re.M) +if m: + d = re.search(r'dst_dir = pkgusr_state\(lfs, "([a-z]+)"\)', m.group(0)) + print(d.group(1) if d else "TOPLEVEL") +else: + print("MISSING") +PYW +)" + # what the reader reads + _r="$(grep -o '\$STATE_[A-Z]*/collector-groups\.import' "$helper_src" \ + | head -1 | sed 's|\$STATE_\([A-Z]*\)/.*|\1|' | tr 'A-Z' 'a-z')" + _p="" + [ "$_w" = groups ] || _p="$_p;lfs writes the collector export to '$_w', not groups/" + [ "$_r" = groups ] || _p="$_p;lfs-helper reads the collector export from '$_r', not groups/" + [ "$_w" = "$_r" ] || _p="$_p;the two tools disagree about where the export lives" + # an edit on the host has to reach the build, so entry re-copies it + case "$(python3 - "$LFS_TOOL" <<'PYE' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) +body = m.group(0) if m else "" +enter = body.split('if action == "enter"') +print("yes" if len(enter) > 1 and "_install_collector_groups" in enter[1] else "no") +PYE +)" in + yes) ;; + *) _p="$_p;editing the export on the host never reaches the build" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the collector export is written where lfs-helper reads it" + fi + + # end to end: a decision in the file must not be asked about again + _cg="$T/collgroups"; mkdir -p "$_cg/groups" + printf '# prefix|nimgnu\ndir|/usr/lib/bfd-plugins|nimgnu_bfd-plugins\n' \ + > "$_cg/groups/collector-groups.import" + _o="$( STATE="$_cg"; STATE_GROUPS="$_cg/groups"; SNAP_ROOT=/ + COLLECTOR_MAP="$_cg/groups/collector-map"; COLLECTOR_PREFIX=nimgnu + PKGUSR_PREFIX="p_" + eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" + eval "$(_slice_fn "$helper_src" collector_group_for)" + # the lookup runs before any of these matter, but they are called on the + # way to it + group_exists() { return 1; } + user_in_group() { return 1; } + eval "$(_slice_fn "$helper_src" sanitise_group_name)" + eval "$(_slice_fn "$helper_src" choose_collector_group)" + # a missing helper must FAIL, not skip: an empty result reading as a + # pass is the trap this suite has fallen into twice + command -v choose_collector_group >/dev/null 2>&1 \ + || { echo "there is no choose_collector_group"; exit 0; } + # no tty, so it can only answer from what it already knows + # <dir> <owner> <user> -- the user is the package asking for access + _g="$(choose_collector_group /usr/lib/bfd-plugins p_binutils p_gcc \ + </dev/null 2>/dev/null)" + [ "$_g" = nimgnu_bfd-plugins ] \ + || echo "a directory the export already answered is asked about again: got '$_g'" )" + if [ -z "$_o" ]; then + ok "a directory named in the export is never asked about again" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- verify must not report its own state directory ------------------------- # +# The state directory moved under /usr/src in 1.9.0, and /usr/src IS an install +# directory -- so `verify` began walking its own manifests, scripts and logs and +# reporting each one as a root-owned file no package claims. A clean 104/104 +# build came back with +# 448 root-owned file(s) that NO manifest claims +# and all 448 were this directory, including the two temp files the scan had +# just written for itself: +# unclaimed /usr/src/lfs-pkgusr/tmp/found.13431 +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr + WRAPPERS=/usr/lib/pkgusr + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + command -v is_never_claimed >/dev/null 2>&1 \ + || { echo "there is no is_never_claimed"; exit 0; } + for f in "$STATE/manifests/gcc.files" "$STATE/scripts/gcc.sh" \ + "$STATE/logs/gcc-all.log" "$STATE/progress/steps-built" \ + "$STATE/tmp/found.13431"; do + is_never_claimed "$f" || echo "verify still reports $f" + done + # an account home under the SAME parent is a package's, and must not be + # swept up by this + is_never_claimed /usr/src/pkgusr/p_gcc/VERSION \ + && echo "a package user's own file was excluded too" )" + if [ -z "$_o" ]; then + ok "verify does not report the build's own notes as unclaimed files" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi +fi + +# ---- the two tools share one tree snapshot ---------------------------------- # +# `lfs` takes the last snapshot outside the chroot at the end of chapter 6; +# `lfs-helper` diffs against it for the first package inside. It is a handover, +# and it works only while both name the same file. 1.9.0 moved the reader to +# progress/tree-snapshot and left the writer on .snapshot at the top level, so +# the handover snapshot was silently lost and the first chroot package diffed +# against nothing. +python3 - "$LFS_TOOL" "$(dirname "$LFS_TOOL")/lfs-helper" <<'PYSNAP' +import re, sys +lfs = open(sys.argv[1]).read() +problems = [] +if '".snapshot"' in lfs or '".snapshot-dirs"' in lfs: + problems.append("lfs still writes the snapshot to .snapshot at the top level") +if 'pkgusr_state(lfs, "progress")' not in lfs: + problems.append("lfs does not write the snapshot into progress/") +for want in ('"tree-snapshot"', '"tree-snapshot-dirs"'): + if want not in lfs: + problems.append("lfs does not name %s" % want) +try: + helper = open(sys.argv[2]).read() +except OSError: + helper = "" +if helper: + m = re.search(r'^SNAPSHOT="([^"]+)"', helper, re.M) + if not m or "tree-snapshot" not in m.group(1): + problems.append("lfs-helper reads a different snapshot file") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS both tools read and write the same tree snapshot") +PYSNAP +_count_rc $? + +# ---- add-user names the account through the chokepoint ---------------------- # +# It used its argument raw for the ACCOUNT and pkg_owner_name only for the HOME. +# Fine while every caller passes an already-prefixed owner -- cmd_build does. +# last_build_step.sh did not: `lfs-helper add-user wget` produced an account +# called `wget` living in /usr/src/pkgusr/p_wget, and the sanity report found +# both halves without being able to connect them: +# !! accounts without a known prefix: wget urllib3 requests ... +# ? /usr/src/pkgusr/p_wget/ has no matching account (stray directory) +# !! group wget (gid 10082) is outside every convention +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _au="$(_slice_fn "$helper_src" cmd_add_user)" + _p="" + case "$(printf '%s\n' "$_au" | grep -vE '^[[:space:]]*#')" in + *'name="$(pkg_owner_name "$name")"'*) ;; + *) _p="$_p;add-user creates the account from the raw name" ;; + esac + # the normalisation must come BEFORE the account is created, or it renames + # nothing + _i1="$(printf '%s\n' "$_au" | grep -n 'name="$(pkg_owner_name' | head -1 | cut -d: -f1)" + _i2="$(printf '%s\n' "$_au" | grep -n 'useradd ' | head -1 | cut -d: -f1)" + if [ -n "$_i1" ] && [ -n "$_i2" ] && [ "$_i1" -gt "$_i2" ]; then + _p="$_p;add-user normalises the name after creating the account" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "add-user gives the account and its home the same name" + fi + + # last_build_step.sh was the caller that exposed it -- it passed bare names. + # It is gone (1.10.0) and this checked a file that no longer exists, so it + # never ran again. The requirement outlives it: `packagemanager setup` is + # the next caller to create accounts from outside the tool, and it must go + # through add-user rather than useradd. Assert that when setup lands. +fi + +# ---- the sanity report reads the sorted state, and knows when a build is done # +# Two findings on a perfect 105/105 tree, both from the report and neither from +# the build: +# +# adopted.list: MISSING (the adoption gate will never close) +# -- it read $STATE/adopted.list; the file is progress/adopted.list +# +# !! /usr/bin is already STICKY -- sealed before the build finished +# -- the build HAD finished, and seal-install-dirs is supposed to do that +# +# The sticky bit means opposite things on either side of that line: during the +# build it is a bug, afterwards its ABSENCE is. +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q 'PROG="\$STATE/progress"' "$san" \ + || _p="$_p;the sanity report still reads the flat state layout" + grep -q 'BUILD_FINISHED' "$san" \ + || _p="$_p;the sanity report does not know whether the build finished" + grep -q 'the build finished but the seal never fired' "$san" \ + || _p="$_p;a finished build with no seal is not reported" + # an empty manifest is a config step that installed nothing, not an orphan + grep -q 'grep -q . "\$m" 2>/dev/null || continue' "$san" \ + || _p="$_p;empty manifests are still reported as missing an account" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report reads the sorted state and judges the seal in context" + fi +fi + +# ---- every tool can say which build it is ----------------------------------- # +# Section 0 of the sanity report exists because a stale copy in the chroot has +# cost more debugging time here than anything else. packagemanager_install had +# no --version, so that row read +# packagemanager_install Please provide an install script with $2 +# (1.12.0) packagemanager_install is retired -- the check inverted: its +# PRESENCE anywhere in the shipped tree is now the bug. +if [ -f "$(dirname "$LFS_TOOL")/packagemanager_install" ]; then + bad "the retired packagemanager_install is back in the tree" +else + ok "the fifth tool stays retired" +fi + +# ---- no finding that repairing makes worse ---------------------------------- # +# Two passes disagreed about a mail spool, and each undid the other: +# +# lfs-helper verify --fix builduser /var/mail/lfs lfs -> root +# lfs-helper verify 1 root-owned file(s) that NO manifest claims: +# /var/mail/lfs +# +# The build-user pass gave it to root because the build user held it; the +# unclaimed pass then reported that root holds a file no manifest claims. +# Repairing it produced the next run's finding. +# +# Shadow creates /var/mail/<name> for every account it makes, so a spool arrives +# as a side effect of `useradd` and no package ever installs one. It belongs to +# the account it is named for, and neither pass should touch it. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr; WRAPPERS=/usr/lib/pkgusr + eval "$(_slice_fn "$helper_src" never_claim_list)" + eval "$(_slice_fn "$helper_src" _never_claim_load)" + eval "$(_slice_fn "$helper_src" is_never_claimed)" + command -v is_never_claimed >/dev/null 2>&1 \ + || { echo "there is no is_never_claimed"; exit 0; } + for f in /var/mail/lfs /var/mail/p_gcc; do + is_never_claimed "$f" || echo "$f is still fought over by two passes" + done + # not the whole of /var, though -- a package's files there are its own + is_never_claimed /var/lib/something \ + && echo "the exclusion is too broad: it swallowed /var/lib" )" + if [ -z "$_o" ]; then + ok "a mail spool belongs to its account, and no pass claims it" + else + printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done + fi + + # the general property: nothing the build-user pass hands to root may then be + # reported by the unclaimed pass, or --fix and verify never agree + case "$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" in + *is_never_claimed*) ok "the build-user pass skips files no package may own" ;; + *) bad "the build-user pass can create a finding for the unclaimed pass" ;; + esac +fi + +# ---- every door to `chown -R lfs` has the same guard ------------------------ # +# There were FOUR places that hand the build tree to the `lfs` user, and only +# one of them had the check that refuses on a tree past chapter 7: +# +# _chown_tree_to_lfs book 4.3, guarded +# session --run re-applied it by hand, unguarded +# the pre-build check "fixing N dir(s) still owned by root", unguarded +# fix-ownership --run the explicit repair, unguarded +# +# On a finished system any of the other three would have handed the whole thing +# to the build user in one pass -- the exact damage the guard exists to prevent, +# through a door the guard was not on. It already happened once via a replayed +# step 4.2. +python3 - "$LFS_TOOL" <<'PYCHOWN' +import re, sys +raw = open(sys.argv[1]).read() +# Comments name the command they are explaining -- every fix in this file says +# what it fixed -- so scan CODE only. Matching the explanation makes a test +# that cannot pass no matter what the code does. +src = "\n".join("" if l.lstrip().startswith("#") else l + for l in raw.splitlines()) +problems = [] +# every recursive chown to lfs, and the function it sits in +for m in re.finditer(r"chown -R lfs", src): + start = src.rfind("\ndef ", 0, m.start()) + if start < 0: + problems.append("a chown -R lfs sits outside any function") + continue + end = src.find("\ndef ", m.end()) + body = src[start:end if end > 0 else len(src)] + name = re.match(r"\ndef ([a-zA-Z_0-9]+)", body).group(1) + # the one that only chowns the state dir is not handing over the tree + if "PKGUSR_DIR" in src[m.start()-120:m.end()+120]: + continue + if "_handover_is_safe" not in body: + problems.append("%s hands the tree to the build user unguarded" % name) +if problems: + for p in sorted(set(problems)): + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS every path that gives the tree to the build user is guarded") +PYCHOWN +_count_rc $? + +# ---- session says what to do next ------------------------------------------- # +# It ended on `export LFS=/mnt/lfs # <-- run this in your shell` and stopped. +# Every other command in this tool finishes with a Next: line; this one left you +# configured and guessing. +python3 - "$LFS_TOOL" <<'PYNEXT' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"^def cmd_bs_session\(.*?(?=\n\ndef )", src, re.S | re.M) +problems = [] +if not m: + problems.append("cmd_bs_session is gone") +else: + body = m.group(0) + # the dry-run branch and the --run branch both end somewhere useful + if body.count('"Next:"') < 2: + problems.append("session does not say what to do next on both paths") + if "lfs run" not in body: + problems.append("session never points at `lfs run`") + if "build-system next" not in body: + problems.append("session never offers `build-system next`") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS session ends by saying what to run next") +PYNEXT +_count_rc $? + +# ---- comm gets sorted input, in one collation ------------------------------- # +# The snapshot diff is `comm -13 <(before) <(after)`. comm compares line by +# line and TRUSTS both sides to be sorted the same way; it cannot detect +# otherwise, it just returns the wrong answer. Two things broke that: +# +# * the two snapshots may have been written under different locales -- a +# package user's profile sets LC_ALL=POSIX and root's shell may not +# * load_snapshot rewrites only the lines carrying the host mount prefix, so +# a file holding both forms comes out sorted under neither +# +# The visible symptom is a package installing forty files and being told it +# installed none. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + for _fn in snapshot snapshot_dirs load_snapshot; do + _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ + || _p="$_p;$_fn does not pin the collation comm depends on" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "both sides of the snapshot diff are sorted the same way" + fi + + # the mixed-prefix case, end to end + _sd="$T/snapsort"; mkdir -p "$_sd" + printf '%s\n' /mnt/lfs/usr/bin/z /usr/bin/a /mnt/lfs/usr/bin/b > "$_sd/snap" + _got="$( SNAP_ROOT=/; LFS_HOST_MOUNT=/mnt/lfs + eval "$(_slice_fn "$helper_src" load_snapshot)" + load_snapshot "$_sd/snap" | tr '\n' ' ' )" + case "$_got" in + "/usr/bin/a /usr/bin/b /usr/bin/z ") ok "a snapshot holding both path forms comes back sorted" ;; + *) bad "load_snapshot returns unsorted input to comm: [$_got]" ;; + esac +fi + +# ---- "installed NO files" has to say which of the three it is --------------- # +# There are exactly three ways to reach zero and they need different fixes: +# nothing was written, it was written somewhere excluded from tracking, or it +# was written over files another package owns and dropped by the filter. +# Saying only "check the log" meant reading a thousand lines of make output to +# find out which. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _cb="$(_slice_fn "$helper_src" cmd_build)" + _p="" + case "$_cb" in + *'new files (snapshot diff)'*) ;; + *) _p="$_p;the zero-files failure does not report the snapshot diff count" ;; + esac + case "$_cb" in + *'files touched since stamp'*) ;; + *) _p="$_p;the zero-files failure does not report the touched count" ;; + esac + case "$_cb" in + *'already belongs to another package'*) ;; + *) _p="$_p;the zero-files failure does not distinguish a dropped install" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package that installed nothing says which kind of nothing" + fi +fi + +# ---- a re-run over its own work is not a silent failure --------------------- # +# The zero-files check counts what THIS RUN wrote, which is the wrong question +# when a package is built a second time. Many installers skip a file that is +# already up to date -- perl's ExtUtils::Install prints "Installing <path>" for +# every target and copies only the stale ones -- so a second run writes nothing: +# the snapshot diff sees nothing new, no mtime beats the stamp, and the count is +# 0 for the best possible reason. +# +# xml-parser hit exactly that: forty files on disk, every one owned by +# p_xml-parser, and the step failed as a no-op. Counting cannot tell that from +# a real failure. The tree can. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _package_already_owns_files | grep -q . \ + || _p="$_p;there is no _package_already_owns_files" + case "$(_slice_fn "$helper_src" cmd_build)" in + *_package_already_owns_files*) ;; + *) _p="$_p;a re-run over its own work still fails as a silent no-op" ;; + esac + # it must ask about the OWNER, and only inside install directories -- looking + # at the whole filesystem would be slow and would count build scratch + _pa="$(_slice_fn "$helper_src" _package_already_owns_files)" + case "$_pa" in + *'-user "$owner"'*) ;; + *) _p="$_p;the check does not look for files this package owns" ;; + esac + case "$_pa" in + *install_dirs_list*) ;; + *) _p="$_p;the check scans outside the install directories" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package that already owns its files is not failed as a no-op" + fi + + # both directions, against a real tree + if [ "$(id -u)" = 0 ]; then + _ow="$T/owns"; mkdir -p "$_ow/usr/lib" + echo x > "$_ow/usr/lib/thing" + _r="$( SNAP_ROOT="$_ow" + user_exists() { [ "$1" = p_test ]; } + install_dirs_list() { printf '%s\n' /usr/lib; } + eval "$(_slice_fn "$helper_src" _package_already_owns_files)" + # owned by root -> this package installed nothing, which IS a failure + _package_already_owns_files p_test && echo "false-pass" + # owned by the package -> it installed, whenever that was + chown 4242 "$_ow/usr/lib/thing" 2>/dev/null + user_exists() { [ "$1" = p_test ]; } + stat -c %U "$_ow/usr/lib/thing" >/dev/null 2>&1 || echo "fixture broken" )" + case "$_r" in + "") ok "a package owning nothing is still reported as a failure" ;; + *) bad "the check passes a package that installed nothing: $_r" ;; + esac + rm -rf "$_ow" + fi +fi + +# ---- a script outside the tool can ask for an account NAME ------------------ # +# `pkgusr-home` was exposed and the account name was not, so a shell script +# could ask where a package lives but not what its user is called. +# last_build_step.sh did the only thing left, which was assume: +# +# lfs-helper add-user wget +# chown -R "wget:wget" ... -> chown: invalid user: 'wget:wget' +# +# Right by accident while add-user used the raw name; broken the moment add-user +# started applying the prefix, as everything else already did. Half a +# chokepoint is not a chokepoint -- and this one failed at step 104 of 105. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" cmd_owner_name | grep -q 'pkg_owner_name' \ + || _p="$_p;there is no way to ask for a package's account name" + grep -q 'owner-name)' "$helper_src" \ + || _p="$_p;owner-name is not reachable as a command" + grep -q 'lfs-helper owner-name <package>' "$helper_src" \ + || _p="$_p;owner-name is not in the help" + # it must agree with pkgusr-home about the same package + _n="$(bash "$helper_src" owner-name wget 2>/dev/null)" + _h="$(bash "$helper_src" pkgusr-home wget 2>/dev/null)" + case "$_h" in + */"$_n") ;; + *) _p="$_p;owner-name and pkgusr-home disagree: '$_n' vs '$_h'" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a package's account name and its home come from the same place" + fi +fi + +# The caller that exposed this was last_build_step.sh, which is gone (1.10.0). +# The rule is not: any script outside the tool that needs an account name must +# ASK -- `lfs-helper owner-name` -- instead of gluing a prefix on itself. +# `packagemanager setup` inherits it; assert it there when setup lands. + +# ---- the ESP hint prints a path you can actually type ----------------------- # +# Two sources feed it and they disagree about the prefix: `lsblk -rno NAME` +# gives a bare name, `blkid -o device` gives a full path. Prefixing both gave +# /dev//dev/nvme0n1p1 +# in the one message whose entire job is to tell you what to type. +python3 - "$LFS_TOOL" <<'PYESP' +import re, sys +src = open(sys.argv[1]).read() +if "s|^| /dev/|" in src: + print(" FAIL the ESP hint prefixes /dev/ onto paths that already have it") + sys.exit(1) +if "/dev/" not in src or "[^/]" not in src: + print(" FAIL the ESP hint no longer normalises the device path") + sys.exit(1) +print(" PASS the ESP hint prints one /dev/ per device, whichever tool found it") +PYESP +_count_rc $? + +# ---- a bootloader is opt-in ------------------------------------------------- # +# The default was `refind`, so every build generated a rEFInd step -- and most +# people running this already have a working ESP and a bootloader that finds it. +# The machine booted in order to build LFS at all. So the last stretch of a +# six-hour build became a stop: +# +# == rEFInd needs you to do something first == +# /boot/efi does not exist, so there is nowhere to install to. +# +# A question nobody asked to be asked. A bootloader is a decision about the +# whole machine, not a package. +python3 - "$LFS_TOOL" <<'PYBL' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +m = re.search(r"^def bootloader\(.*?(?=\n\ndef )", src, re.S | re.M) +if not m: + problems.append("bootloader() is gone") +else: + body = "\n".join("" if l.lstrip().startswith("#") else l + for l in m.group(0).splitlines()) + body = "".join(body.split('"""')[::2]) + if '"refind"' in body: + problems.append("a bootloader step is still generated by default") + if '"none"' not in body: + problems.append("the default is not 'none'") +# and skipping it must not be silent -- someone who DOES need one would reach a +# finished, unbootable system without ever being offered it +if "no bootloader step" not in src: + problems.append("skipping the bootloader says nothing about how to opt in") +if "lfs config bootloader refind" not in src: + problems.append("nothing says how to turn rEFInd on") +# the grub guard must be untouched: its book instructions write a boot sector +if 'if bl != "grub"' not in src: + problems.append("the grub script is no longer withheld unless chosen") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS no bootloader is set up unless you ask for one") +PYBL +_count_rc $? + + +# ---- the sort and the comparison must agree --------------------------------- # +# 1.9.6 pinned LC_ALL=C on the sorts feeding comm and stopped there. That is +# half a fix: comm validates ordering in ITS OWN locale, so C-sorted input +# compared under en_GB.UTF-8 is "not sorted" as far as comm is concerned -- +# +# comm: file 1 is not in sorted order +# comm: file 2 is not in sorted order +# comm: input is not in sorted order +# +# and the diff it returns is then simply wrong. Pinning either side alone +# guarantees they disagree. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _bare="$(grep -nE '(^|[^=[:alnum:]_])comm ' "$helper_src" \ + | grep -vE 'LC_ALL=C comm' | grep -vE '^[0-9]+:[[:space:]]*#')" + if [ -n "$_bare" ]; then + printf '%s\n' "$_bare" | while IFS= read -r m; do + bad "comm without LC_ALL=C: ${m%%:*}" + done + else + ok "every comm compares in the locale its input was sorted in" + fi + # and every sort whose output reaches a comm is pinned too + _p="" + for _fn in snapshot snapshot_dirs load_snapshot _vfy_unclaimed; do + _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ + || _p="$_p;$_fn sorts in the ambient locale" + done + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every sort feeding a comm is pinned to the same locale" + fi +fi + + +# ---- the scripts in the tree must match the tools outside it ---------------- # +# The step order and the generated scripts live in the TREE; the tools live +# outside it. Nothing connected the two, so a tree generated by an older +# version went on running steps that version had: +# +# ===== [104/105] last-step ===== +# chown: invalid user: 'wget:wget' +# +# under an lfs-helper (1.10.0) that had removed `last-step` entirely and no +# longer knew what it was. The chroot TOOLS already carry a build id for +# exactly this reason -- a stale copy has cost more debugging time here than +# anything else -- and the generated scripts did not. +python3 - "$LFS_TOOL" <<'PYSTAMP' +import re, sys +src = open(sys.argv[1]).read() +problems = [] +if "generated-by" not in src: + problems.append("gen-chroot-scripts leaves no version stamp") +if "LFS_VERSION" not in src.split("generated-by")[1][:400]: + problems.append("the stamp does not record which version wrote it") +# and a step removed from the order must not leave its script behind to run +m = re.search(r"^def cmd_bs_gen_chroot_scripts\(.*?(?=\n\ndef )", src, re.S | re.M) +body = m.group(0) if m else "" +if "stale" not in body or "os.remove" not in body: + problems.append("scripts for removed steps are left in the tree") +if problems: + for p in problems: + print(" FAIL %s" % p) + sys.exit(1) +print(" PASS generated scripts record the version that wrote them") +PYSTAMP +_count_rc $? + +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _warn_if_scripts_are_stale | grep -q 'LFS_HELPER_VERSION' \ + || _p="$_p;lfs-helper does not compare its version against the scripts'" + case "$(_slice_fn "$helper_src" cmd_build_all)" in + *_warn_if_scripts_are_stale*) ;; + *) _p="$_p;build-all runs an order it never checked" ;; + esac + # a missing script must blame the likely cause, not just offer to regenerate + case "$(_slice_fn "$helper_src" cmd_build)" in + *'no longer exists in this version'*) ;; + *) _p="$_p;a step removed from the tools reports only a missing file" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a step order older than the tools is reported, not obeyed" + fi + + # end to end: an unstamped tree, and a mismatched one + _st="$T/stamp"; mkdir -p "$_st/progress" + _r="$( STATE_PROGRESS="$_st/progress"; LFS_HELPER_VERSION="9.9.9" + eval "$(_slice_fn "$helper_src" _warn_if_scripts_are_stale)" + warn() { echo "WARNED: $*"; } + _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "unstamped-passed" + printf 'version=9.9.9\n' > "$_st/progress/generated-by" + _warn_if_scripts_are_stale >/dev/null 2>&1 || echo "matching-failed" + printf 'version=1.0.0\n' > "$_st/progress/generated-by" + _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "mismatch-passed" )" + case "$_r" in + "") ok "an unstamped or mismatched step order is caught, a matching one is not" ;; + *) printf '%s\n' "$_r" | while IFS= read -r m; do + [ -n "$m" ] && bad "stale-script check is wrong: $m" + done ;; + esac + rm -rf "$_st" +fi + +# ---- unpacked sources are scratch, and no scan should walk them ------------- # +# Build trees moved from /build into each package user's home in 1.7.6. The +# snapshot scans followed; the ownership scans did not. So `lfs-helper verify` +# began walking every unpacked source tree in the system -- it ran for minutes +# with no output -- and a finished tree filled the sanity report with tcl's own +# documentation: +# +# !! files with no owner (first 40): +# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm +# +# A tarball can carry any uid it likes. Unpacked sources are not installed, +# nothing owns them, and asking who does has no answer. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" scan_prune_paths | grep -q 'PKGUSR_BUILD_SUBDIR' \ + || _p="$_p;the prune list does not cover the per-package build trees" + # ONE list -- there were five and they had already diverged + for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do + _slice_fn "$helper_src" "$_fn" | grep -q 'SCAN_PRUNE' \ + || _p="$_p;$_fn does not use the shared prune list" + done + # an ARRAY, not a string through eval: the patterns contain *, and eval lets + # the shell expand them before find ever sees them + case "$(_slice_fn "$helper_src" scan_prune_set)" in + *'SCAN_PRUNE+=('*) ;; + *) _p="$_p;the prune list is not built as an array" ;; + esac + if _slice_fn "$helper_src" _vfy_orphans | grep -q 'eval find'; then + _p="$_p;the orphan scan splices its prune list through eval" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "every ownership scan skips the same scratch, from one list" + fi + + # end to end: a build tree is pruned, a real file is not + _pt="$T/prune" + mkdir -p "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html" "$_pt/usr/bin" + touch "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" + if [ "$(id -u)" = 0 ]; then + chown 4321 "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" + _r="$( SNAP_ROOT="$_pt"; STATE="$_pt/usr/src/lfs-pkgusr" + PKGUSR_BUILD_SUBDIR=src + PKGUSR_ROOT="$_pt/usr/src/pkgusr"; CFGUSR_ROOT="$_pt/usr/src/cfg" + pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } + eval "$(_slice_fn "$helper_src" scan_prune_paths)" + eval "$(_slice_fn "$helper_src" scan_prune_set)" + scan_prune_set + find "$_pt" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null \ + | sed "s|$_pt||" | sort | tr '\n' ' ' )" + case "$_r" in + "/usr/bin/real ") ok "an unpacked source tree is skipped, an installed file is not" ;; + *) bad "the prune list is wrong: got [$_r]" ;; + esac + fi + rm -rf "$_pt" +fi + +# the sanity report has to skip the same thing, or it reports what verify does not +san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" +if [ -f "$san" ]; then + _p="" + grep -q 'PKGROOT/\*/src' "$san" \ + || _p="$_p;the sanity report still walks the package build trees" + # a config step writing root-owned files is not an orphaned manifest + grep -q 'writes ROOT-owned files' "$san" \ + || _p="$_p;root-owned config files are still reported as missing an account" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the sanity report skips scratch and does not invent findings" + fi +fi + +# ---- verify must not work in silence, or one fork at a time ----------------- # +# `lfs-helper verify` ran for over five minutes printing nothing, on a tree it +# had already verified clean. Two separate faults: +# +# * a fork per path, twice over -- `stat -c %U` and a process substitution +# inside is_never_claimed -- across 66265 manifest paths +# * no output at all until every pass had finished +# +# A command that works for minutes in silence is indistinguishable from one that +# has hung, and was reported as one. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + # the never-claim list is read once, not per call + _slice_fn "$helper_src" is_never_claimed | grep -q '< <(never_claim_list)' \ + && _p="$_p;is_never_claimed forks for every path it is asked about" + _slice_fn "$helper_src" _never_claim_load | grep -q . \ + || _p="$_p;the never-claim list is not cached" + # owners come from one batched stat, not one per path + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'xargs -0 -r stat'*) ;; + *) _p="$_p;the ownership pass still runs one stat per path" ;; + esac + # a REAL tab: stat does not expand \t, so '%U\tname' comes back as one field + # and the map silently stays empty -- correct but slow, the hardest wrong to see + # comments stripped: the fix NAMES the broken format while explaining it + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership \ + | grep -vE '^[[:space:]]*#')" in + *'%U\t%n'*) _p="$_p;the stat format uses a literal backslash-t" ;; + esac + # and it must say what it is doing, before it does it + _nd="$(_slice_fn "$helper_src" cmd_verify | grep -c 'detail "# ')" + [ "${_nd:-0}" -ge 5 ] \ + || _p="$_p;verify announces only $_nd of its passes" + case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in + *'[%d/%d]'*) ;; + *) _p="$_p;the long pass shows no progress" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "verify says what it is doing, and does it in one pass not 66000" + fi + + # the batched lookup must actually map something -- an empty map is the bug + _pf="$T/statmap"; mkdir -p "$_pf" + touch "$_pf/a" "$_pf/b" "$_pf/c" + printf '%s\n' "$_pf/a" "$_pf/b" "$_pf/c" > "$_pf/man" + _n="$( _TAB=$'\t' + declare -A own=() + while IFS="$_TAB" read -r o nm; do + [ -n "$nm" ] && own["$nm"]="$o" + done < <(grep -v '^[[:space:]]*$' "$_pf/man" \ + | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null) + echo "${#own[@]}" )" + [ "$_n" = 3 ] \ + && ok "the batched owner lookup maps every path it is given" \ + || bad "the batched owner lookup mapped $_n of 3 paths" + rm -rf "$_pf" +fi + +# ---- the build must not end quietly with no way to log in ------------------- # +# init-accounts asks for a root password, but it is skippable: a scripted run +# has no terminal, and an interactive one can be answered with a blank line. +# So the LAST thing build-all says is whether anyone can actually log in -- +# because it is the last moment it can still be fixed. After the reboot the +# chroot is gone, and the way back is booting the host and mounting the tree. +helper_src="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$helper_src" ]; then + _p="" + _slice_fn "$helper_src" _warn_if_no_login | grep -q . \ + || _p="$_p;nothing checks whether the system can be logged into" + # it must be the LAST thing build-all does + _ba="$(_slice_fn "$helper_src" cmd_build_all)" + case "$_ba" in + *_warn_if_no_login*) ;; + *) _p="$_p;build-all can finish without saying whether you can log in" ;; + esac + _last="$(printf '%s\n' "$_ba" | grep -vE '^[[:space:]]*(#|$)' | tail -3)" + case "$_last" in + *_warn_if_no_login*) ;; + *) _p="$_p;the login check is not the last thing build-all does" ;; + esac + # it must check the RESULT, not that the step ran + case "$(_slice_fn "$helper_src" _warn_if_no_login)" in + *shadow*) ;; + *) _p="$_p;it trusts init-accounts instead of checking for a password" ;; + esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the build's last word is whether you can log in afterwards" + fi + + # all three outcomes, against a real shadow file + _lg="$T/login"; mkdir -p "$_lg" + _run_login() { # <shadow contents> -> "ok" or "warn" + ( ETC="$_lg"; say(){ :; }; ok(){ :; }; warn(){ echo warn; } + printf 'root:x:0:0::/root:/bin/bash\nbob:x:1000:1000::/home/bob:/bin/bash\n' > "$_lg/passwd" + printf '%s' "$1" > "$_lg/shadow" + eval "$(_slice_fn "$helper_src" _warn_if_no_login)" + _warn_if_no_login | head -1 ) 2>/dev/null + } + _bad="" + [ -z "$(_run_login 'root:$6$a:1:::::: +bob:!:1::::::')" ] || _bad="$_bad;a root password is not accepted" + [ -z "$(_run_login 'root:*:1:::::: +bob:$6$b:1::::::')" ] || _bad="$_bad;a user password is not accepted" + [ -n "$(_run_login 'root:*:1:::::: +bob:!:1::::::')" ] || _bad="$_bad;a system nobody can log into is not reported" + if [ -n "$_bad" ]; then + printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a locked-out system is reported; a usable one is not" + fi + rm -rf "$_lg" +fi + +# ---- root's shell history is not a package's file --------------------------- # +# A real verify reported: [62/221] sqlite /root/.bash_history root -> p_sqlite +# -- the timestamp scan swept root's dotfiles into whichever manifest was open, +# and --fix would have handed root's shell history to p_sqlite. And the tools' +# own stores (/usr/share/lfs, /usr/share/blfs) plus the completion file were +# reported as claimed by nobody, which is true of the stores (runtime caches) +# and false of the completion file (make install put it there). +_p="" +_ncout="$( + { echo 'ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr' + eval_src(){ _slice_fn "$helper_src" "$1"; } + _slice_fn "$helper_src" never_claim_list + echo '_NEVER_CLAIM=()' + _slice_fn "$helper_src" _never_claim_load + _slice_fn "$helper_src" is_never_claimed + cat <<'NCEOF' +for f in /root/.bash_history /usr/share/blfs/books/x.html \ + /usr/share/lfs/tool-stamps.json /usr/bin/gcc; do + is_never_claimed "$f" && echo "shielded $f" || echo "claimable $f" +done +NCEOF + } | bash 2>/dev/null +)" +_ncout2="$( + { echo 'ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr' + _slice_fn "$helper_src" never_claim_list + echo '_NEVER_CLAIM=()' + _slice_fn "$helper_src" _never_claim_load + _slice_fn "$helper_src" is_never_claimed + cat <<'NC2' +for f in /etc/ssl/certs/ISRG_Root_X1.pem /etc/pki/anchors/x.p11-kit \ + /etc/pkgusr/stacks/sway.stack /usr/share/lfs-pkgusr/x.sh; do + is_never_claimed "$f" && echo "shielded $f" || echo "claimable $f" +done +NC2 + } | bash 2>/dev/null +)" +echo "$_ncout" | grep -q "shielded /root/.bash_history" \ + || _p="$_p;/root is claimable -- root's dotfiles can be swept into a manifest" +echo "$_ncout" | grep -q "shielded /usr/share/blfs/books/x.html" \ + || _p="$_p;the blfs store is claimable" +echo "$_ncout" | grep -q "shielded /usr/share/lfs/tool-stamps.json" \ + || _p="$_p;the lfs store is claimable" +for _shield in /etc/ssl/certs/ISRG_Root_X1.pem /etc/pki/anchors/x.p11-kit \ + /etc/pkgusr/stacks/sway.stack /usr/share/lfs-pkgusr/x.sh; do + echo "$_ncout2" | grep -q "shielded $_shield" \ + || _p="$_p;$_shield can be swept into a package's manifest" +done +echo "$_ncout" | grep -q "claimable /usr/bin/gcc" \ + || _p="$_p;never-claim shields too much" +[ "$(grep -c -- '-not -path "\$r/root/\*"' "$helper_src")" = 2 ] \ + || _p="$_p;a snapshot scan still sweeps /root (and would repollute a manifest)" +_slice_fn "$helper_src" _adopt_pkgusr_tools | grep -q 'bash-completion/completions' \ + || _p="$_p;the completion file is claimed by nobody" +_slice_fn "$helper_src" cmd_verify | grep -q '_adopt_pkgusr_tools' \ + || _p="$_p;verify --fix never refreshes the tools' own manifest" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "root's dotfiles and the tool stores are nobody's install; the completion file is ours" +fi + +# ---- fix-home: the home that never met its files ---------------------------- # +# add_package_user creates /usr/src/<account>; the tools use +# /usr/src/pkgusr/<account>. The old repair repointed the passwd entry and +# left the directory behind -- a real bootstrap ended with ten husks beside +# the roots (/usr/src/p_wget, /usr/src/p_requests, ...). fix-home is the one +# implementation of the whole repair; the other two tools must call it. +echo +echo "== fix-home repairs a misplaced home, and only as much as is safe ==" +_fh="$T/fixhome"; mkdir -p "$_fh/etc" "$_fh/src/pkgusr" "$_fh/state" +mkdir -p "$_fh/src/pkgusr/p_wget/log" "$_fh/src/p_wget/log" "$_fh/src/p_requests" +echo 'shared profile' > "$_fh/src/pkgusr/p_wget/.bash_profile" +echo 'shared profile' > "$_fh/src/p_wget/.bash_profile" # identical +echo 'old build log' > "$_fh/src/p_wget/log/first-try.log" # unique +echo 'DIFFERENT' > "$_fh/src/p_wget/build" # differs +echo 'the real build' > "$_fh/src/pkgusr/p_wget/build" +echo reqfile > "$_fh/src/p_requests/pkg.lst" +printf 'root:x:0:0::/root:/bin/bash\np_wget:x:10004:10004::%s/src/pkgusr/p_wget:/bin/bash\np_requests:x:10005:10005::%s/src/p_requests:/bin/bash\n' \ + "$_fh" "$_fh" > "$_fh/etc/passwd" + +_fh_run() { env LFS_SRC_ROOT="$_fh/src" LFS_ETC="$_fh/etc" \ + LFS_PKGUSR_PREFIX=p LFS_PKGUSR_DIR="$_fh/state" \ + bash "$helper_src" "$@" ; } + +_fh_run fix-home wget >/dev/null 2>&1 +_rc=$? +if [ "$_rc" = 1 ] && [ -f "$_fh/src/p_wget/log/first-try.log" ]; then + ok "fix-home without --run reports (exit 1) and changes nothing" +else + bad "fix-home dry run: exit $_rc, or it moved files" +fi + +_fh_wget_out="$(_fh_run fix-home wget --run 2>&1)" +_fh_run fix-home requests --run >/dev/null 2>&1 +_p="" +echo "$_fh_wget_out" | grep -q "differs: .*p_wget/build" \ + || _p="$_p;a kept file is not NAMED (the user is sent diffing directories by hand)" +[ -f "$_fh/src/pkgusr/p_wget/log/first-try.log" ] || _p="$_p;the unique file was not moved into the real home" +[ -e "$_fh/src/p_wget/.bash_profile" ] && _p="$_p;the identical stray copy was not deleted" +[ "$(cat "$_fh/src/p_wget/build" 2>/dev/null)" = "DIFFERENT" ] || _p="$_p;a file that DIFFERS was not kept" +[ "$(cat "$_fh/src/pkgusr/p_wget/build" 2>/dev/null)" = "the real build" ] || _p="$_p;the real home's file was touched" +[ -d "$_fh/src/p_requests" ] && _p="$_p;a cleanly movable husk was left behind" +[ -f "$_fh/src/pkgusr/p_requests/pkg.lst" ] || _p="$_p;the movable husk's files did not arrive" +grep -q '^p_requests:.*:'"$_fh"'/src/pkgusr/p_requests:' "$_fh/etc/passwd" \ + || _p="$_p;the passwd entry was not repointed" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "fix-home: moves what is missing, deletes what is identical, keeps what differs, repoints passwd" +fi + +# the summary must count what the passes repaired -- a real run merged four +# husks and claimed four tools, then said "repaired 0 path(s)" +_slice_fn "$helper_src" cmd_verify | grep -q 'fixed + _vfy_n_fixed' \ + && ok "verify's summary counts the repairs its passes made" \ + || bad "verify still reports 'repaired 0' after repairing things" + +# ---- one loop: the engine hands its script to lfs-helper build (step 5) ----- # +# cmd_build takes a script that is not in $SCRIPTS, normalises the name so the +# records land under the step name, and does everything the engine's private +# loop did. The engine keeps only front-matter and calls it. +_d5="$T/step5"; mkdir -p "$_d5/root/usr/bin" "$_d5/src/pkgusr" "$_d5/state" +_p="" +if command -v useradd >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then + userdel p_dmy5 >/dev/null 2>&1; groupdel p_dmy5 >/dev/null 2>&1 + getent group install >/dev/null || groupadd -g 9999 install + _d5_run() { env LFS_PKGUSR_DIR="$_d5/state" LFS_SRC_ROOT="$_d5/src" \ + LFS_SNAP_ROOT="$_d5/root" LFS_COLLECTOR_PREFIX=nimgnu \ + LFS_PKGUSR_PREFIX=p bash "$helper_src" "$@" ; } + _d5_run add-user dmy5 >/dev/null 2>&1 + _d5h="$(_d5_run pkgusr-home dmy5)" + chgrp install "$_d5/root/usr/bin" && chmod g+w "$_d5/root/usr/bin" + printf '#!/bin/bash\ninstall_pkg() { :; }\ncase "${1:-all}" in all) echo hi > %s/root/usr/bin/dmy5-tool ;; esac\n' \ + "$_d5" > "$_d5h/install_p_dmy5" + chown p_dmy5: "$_d5h/install_p_dmy5" 2>/dev/null + _d5_run build p_dmy5 --script "$_d5h/install_p_dmy5" --phase all --force \ + >/dev/null 2>&1 + [ -f "$_d5/root/usr/bin/dmy5-tool" ] \ + || _p="$_p;a handed-over script did not build" + [ "$(stat -c %U "$_d5/root/usr/bin/dmy5-tool" 2>/dev/null)" = p_dmy5 ] \ + || _p="$_p;the handed-over build's files do not belong to the package user" + [ -f "$_d5/state/manifests/dmy5.files" ] \ + || _p="$_p;the records landed under the ACCOUNT name, not the step name (or nowhere)" + userdel p_dmy5 >/dev/null 2>&1; groupdel p_dmy5 >/dev/null 2>&1 +fi +grep -q -- '--script' "$helper_src" \ + || _p="$_p;cmd_build cannot take a script by path" +# NEVER DROP A CLASS OF DEPENDENCIES. --ignore-recommended, used to keep +# texlive out of a sway build, also removed texlive's OWN deps (all +# recommended) -- so it was scheduled first and built against a freetype +# that did not exist. Unwanted packages are NAMED, one at a time. +# deps=required is a CHOICE OF GRAPH, per entry, and the recommendations +# worth keeping are named as entries of their own. (The first attempt at +# this failed because the ORDER was wrong -- texlive's deps are all +# recommended, so dropping them left it first in the plan. That was fixed +# by the topological sort; the choice itself is sound.) +grep -q 'deps.*== "required"' "$pm_src" \ + || _p="$_p;a stack entry cannot ask for the required graph only" +# ...and --status must report THAT graph: it asked without deps= and showed +# 272 packages for a plan that builds 99. A status that does not match the +# run is worse than no status. +grep -q 'no_recommended=(e\["opts"\]' "$pm_src" \ + || _p="$_p;--status counts a different dependency graph than the build uses" +# THE STACKS MUST TRAVEL WITH THE TOOLS. The chroot refresh copied the four +# binaries only, so a chroot kept its first sway.stack forever -- the user +# updated the tools four times and kept reading a 39-entry file. +# (only meaningful where the stacks ship beside the tool, as they do in a +# real install; a bare checkout of the tool alone has nothing to copy) +[ -d "$(dirname "$LFS_TOOL")/stacks" ] && +python3 - "$LFS_TOOL" <<'PYSTKREF' || _p="$_p;the chroot keeps a stale stack file after a tool refresh" +import sys, os, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader("lfs", sys.argv[1]).load_module() +d = tempfile.mkdtemp() +os.makedirs(os.path.join(d, "usr/bin")) +os.makedirs(os.path.join(d, "etc/pkgusr/stacks")) +open(os.path.join(d, "etc/pkgusr/stacks/sway.stack"), "w").write("old\n") +open(os.path.join(d, "etc/pkgusr/stacks/machine.conf"), "w").write("[mesa]\nmine=yes\n") +lfs._refresh_chroot_tools(d, quiet=True) +got = open(os.path.join(d, "etc/pkgusr/stacks/sway.stack")).read() +assert got.strip() != "old", "the stack was not refreshed" +# the machine's own options are never overwritten +assert "mine=yes" in open(os.path.join(d, "etc/pkgusr/stacks/machine.conf")).read() +PYSTKREF +# ...and every entry point uses the SAME sync: `run` and `chroot enter` did, +# `sync-tools` did not, so which command you used decided what got updated. +_slice_py="$(sed -n '/^def cmd_bs_sync_tools/,/^def /p' "$LFS_TOOL")" +printf '%s' "$_slice_py" | grep -q '_refresh_chroot_tools' \ + || _p="$_p;sync-tools does not refresh the tools and stacks the other paths do" +# a NEWER stack in the chroot is never replaced by an older one from the host +grep -q '_stack_version' "$LFS_TOOL" \ + || _p="$_p;a newer stack file in the chroot can be overwritten by an older one" +# AND IT MUST FIND THEM WHEN INSTALLED. The running tool is /usr/bin/lfs, +# so looking only beside itself meant /usr/bin/stacks -- nothing, in silence, +# while the host had 31 entries and the chroot kept 23. +grep -q '"/etc/pkgusr/stacks"' "$LFS_TOOL" \ + || _p="$_p;an installed lfs cannot find the stack files to refresh" +grep -q 'no stack files found to refresh' "$LFS_TOOL" \ + || _p="$_p;finding no stacks to refresh is silent" +_stk="$(dirname "$LFS_TOOL")/stacks/sway.stack" +if [ -f "$_stk" ]; then + # The recommendations are FOLLOWED again: a blanket --ignore-recommended + # dropped things the builds hard-require (curl for cmake, icu for + # libxml2). Unwanted branches are named instead. + grep -q 'deps=required' "$_stk" \ + && _p="$_p;the stack drops recommendations wholesale again" + grep -qE '^avoid +qt6' "$_stk" \ + || _p="$_p;the heavy branches are not named as avoided" + # AVOID MUST REACH THE INSTALLER. It printed "never built by this stack" + # and passed nothing on, so the packages were built anyway. + grep -q 'ignore=",".join' "$pm_src" \ + || _p="$_p;the stack's avoid list never reaches the installer" + # ...and it must reach the STATUS too: it reported 273 packages for a run + # that skips qt6, okular, LibreOffice and CUPS by name. + grep -q 'skipped_by_avoid' "$pm_src" \ + || _p="$_p;--status counts packages the run will skip" + # (a reachability prune was tried in 1.12.82 and reverted: `blfs order` + # reports one parent per package, so a shared dependency recorded under + # an ignored parent was dropped -- harfbuzz then failed on graphite2. + # Unwanted leaves are named in the stack instead.) + grep -qE '^avoid +plasma-activities' "$_stk" \ + || _p="$_p;the KDE leaves an avoided okular pulls in are not named" + for _want in harfbuzz pango gdk-pixbuf glib-networking adwaita-icon-theme; do + grep -qE "^book +$_want" "$_stk" \ + || _p="$_p;$_want is a recommendation we rely on but no longer name" + done +fi +# (1.12.0) the engine is retired: packagemanager's front door is +# lfs-helper pm-install, and pm-install hands the script to the one loop. +_pm_src5="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$_pm_src5" ]; then + grep -q '"pm-install"' "$_pm_src5" \ + || _p="$_p;packagemanager does not call lfs-helper pm-install" +fi +_slice_fn "$helper_src" cmd_pm_install | grep -qF 'install_$user_name" ' \ + || _p="$_p;pm-install does not hand its prepared script to the one loop" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "one loop: the engine hands its script over, records land under the step name" +fi +rm -rf "$_d5" + +# ---- a failing phase stops the run, and our contract is known ---------------- # +# ghostscript: configure failed, and `make so`, `gs ...` and the whole install +# ran anyway. `unpack_pkg && build_pkg || exit 1` reads like it stops on +# failure and does the opposite -- POSIX disables set -e inside a function +# whose status is being tested. And the block that would have PREVENTED the +# failure ("If you have installed the recommended dependencies, remove the +# copies of freetype, lcms2, libjpeg...") was commented out as conditional, +# though these tools install recommended dependencies by definition. +_p="" +_bt5="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt5" ]; then + # what is EMITTED, not what is written about it: the explanation of this + # very bug quotes the old line, and a grep over the whole file cannot tell + # a comment from code + grep -qE "^\s*w\('.*unpack_pkg && build_pkg" "$_bt5" \ + && _p="$_p;the dispatcher calls phases in a test context, which switches off set -e inside them" + # (the dispatcher is the shared runner's: each phase is its own process + # under set -e, and the test switch is an if, not an and-list) + grep -q 'PKGUSR_PHASE="\$1" bash "\$_pp_self" </dev/null' "$PHASES_LIB" \ + || _p="$_p;the dispatcher does not run each phase as its own process" + sed -n '/^pkgusr_run() {/,/^}/p' "$PHASES_LIB" | grep -q '^ set -e$' \ + || _p="$_p;the dispatcher does not run under set -e" + grep -q '_pp_run_tests && _pp_test_step' "$PHASES_LIB" \ + || _p="$_p;the test condition is an and-list, which aborts the run under set -e when false" + grep -qE '^\s*w\(.*(unpack_pkg|_enter_build|_fetch|_mirrors_for)\(\) \{' "$_bt5" \ + && _p="$_p;blfs still emits its own scaffolding beside the runner" + python3 - "$_bt5" <<'PYGD' || _p="$_p;the conditional guard does not know what our contract installs" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +class P: + def __init__(self, txt): self.txt = txt; self.name = "p" + def get_text(self): return self.txt +class Pre: + def __init__(self, txt): self.p = P(txt) + def find_previous(self, names): return self.p +# recommended/required -> our contract satisfies it, so the block RUNS +assert b._conditional_guard(Pre("If you have installed the recommended dependencies on your system, remove the copies:")) is None +assert b._conditional_guard(Pre("If you have installed the required tools, run:")) is None +# optional -> skipped, whatever else the sentence says +assert b._conditional_guard(Pre("If you installed the optional sphinx module, create the man page:")) +assert b._conditional_guard(Pre("If you have installed the optional recommended extras, run:")) +# a demonstration that runs the package's own program is not a build step +out = b._comment_own_program_demos("make\ngs -q -dBATCH x.eps", {"installed_programs": "gs, gsc"}) +assert "# gs -q" in out and out.splitlines()[0] == "make", out +PYGD +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a failing phase stops the run; blocks conditioned on what we install are kept" +fi + +# ---- the tool must survive being replaced while it runs ---------------------- # +# bash reads a script incrementally, by byte offset. Replace the file mid-run +# and execution resumes at that offset in the NEW content, mid-word: +# /usr/bin/lfs-helper: line 8312: mmand: command not found +# /usr/bin/lfs-helper: line 8314: syntax error near unexpected token `)' +# The package had already installed; the tool died on its own source. +_p="" +_g1="$(grep -vE '^\s*(#|$)' "$helper_src" | head -1)" +[ "$_g1" = "{" ] \ + || _p="$_p;lfs-helper is not wrapped in a group command -- a mid-run replacement corrupts it" +[ "$(tail -1 "$helper_src" | cut -c1)" = "}" ] \ + || _p="$_p;the group command is not closed at the end of the file" +bash -n "$helper_src" 2>/dev/null || _p="$_p;lfs-helper does not parse" +# and it still works, wrapped +_gv="$(bash "$helper_src" --version 2>&1)" +case "$_gv" in lfs-helper*build*) ;; *) _p="$_p;the wrapped tool cannot report its version: $_gv" ;; esac +_gh="$(bash "$helper_src" owner-name gcc 2>&1)" +[ "$_gh" = "p_gcc" ] || _p="$_p;the wrapped tool's commands do not run: $_gh" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the tool reads itself once, so replacing it mid-run cannot corrupt the run" +fi + +# ---- "answer for me" means one thing ----------------------------------------- # +# `packagemanager stack sway --run --yes` still stopped to ask which +# collector group to use -- the exact thing --yes exists to prevent. The +# group prompt honoured LFS_ASSUME_YES, packagemanager --yes sets PM_YES, and +# pm-install read PM_YES: three spellings of one decision. +_p="" +grep -q '^assume_yes()' "$helper_src" \ + || _p="$_p;there is no single definition of assume-yes" +for _spell in PM_YES LFS_ASSUME_YES; do + _r="$({ _slice_fn "$helper_src" assume_yes + echo 'assume_yes && echo YES || echo ASK'; } | env "$_spell=1" bash 2>/dev/null)" + [ "$_r" = YES ] || _p="$_p;$_spell=1 does not mean assume-yes" +done +_r="$({ _slice_fn "$helper_src" assume_yes + echo 'assume_yes && echo YES || echo ASK'; } | env -u PM_YES -u LFS_ASSUME_YES bash 2>/dev/null)" +[ "$_r" = ASK ] || _p="$_p;assume-yes is on when neither variable is set" +# the group prompt itself must use it, and say what it chose +_gp="$(_slice_fn "$helper_src" choose_collector_group)" +[ -n "$_gp" ] || _gp="$(grep -A6 'Which group should share THIS DIRECTORY' "$helper_src")" +printf '%s' "$_gp" | grep -q 'assume_yes' \ + || _p="$_p;the collector-group prompt still has its own idea of --yes" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "--yes answers every prompt, whichever tool asked" +fi + +# ---- the book's order survives the split into phases ------------------------- # +# A page can go user -> root -> user. GLib does: build with introspection +# disabled (user), install gobject-introspection (root), reconfigure with +# introspection enabled (user). Sorting blocks into two streams kept each +# stream's order and lost the INTERLEAVING, so the reconfigure ran before +# g-ir-scanner existed: ERROR: Program 'g-ir-scanner' not found +_p="" +_bt7="$(dirname "$LFS_TOOL")/blfs" +_bk7="$(ls "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*.html 2>/dev/null | head -1)" +if [ -f "$_bt7" ] && [ -n "$_bk7" ]; then + _god="$T/gliborder"; mkdir -p "$_god" + python3 "$_bt7" --book-file "$_bk7" script glib2 -o "$_god" >/dev/null 2>&1 + _gof="$(ls "$_god"/install_GLib-* 2>/dev/null | head -1)" + if [ -n "$_gof" ]; then + _ins="$(sed -n '/^install_pkg/,/^}/p' "$_gof" | grep -vE '^\s*(#|$)')" + _i_gi="$(printf '%s\n' "$_ins" | grep -n 'gi-build install' | head -1 | cut -d: -f1)" + _i_cfg="$(printf '%s\n' "$_ins" | grep -n 'introspection=enabled' | head -1 | cut -d: -f1)" + if [ -z "$_i_gi" ] || [ -z "$_i_cfg" ]; then + _p="$_p;glib's introspection dance is not in the install phase at all" + elif [ "$_i_gi" -ge "$_i_cfg" ]; then + _p="$_p;glib reconfigures for introspection BEFORE installing gobject-introspection" + fi + printf '%s\n' "$(sed -n '/^build_pkg/,/^}/p' "$_gof")" | grep -q 'introspection=enabled' \ + && _p="$_p;the reconfigure is still stranded in the build phase" + fi + rm -rf "$_god" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a page that goes user -> root -> user keeps the book's order" +fi + +# ---- a required edge beats a recommended one --------------------------------- # +# shared-mime-info REQUIRES GLib; GLib merely RECOMMENDS shared-mime-info. +# The walk marks a node visited on entry, so exploring GLib's recommended +# list reached shared-mime-info while GLib was still in progress, skipped its +# required GLib edge as "already visited", and emitted it FIRST -- which +# fails at `Dependency "glib-2.0" not found`. A recommendation is a +# preference; a requirement is a fact. +_p="" +_bt6="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt6" ]; then + python3 - "$_bt6" <<'PYORD' || _p="$_p;a recommended edge can still put a package before something it REQUIRES" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +# the exact shape of the GLib / shared-mime-info cycle +order = ["smi", "glib", "dfu"] +edges = { + "smi": [{"anchor": "glib", "kind": "required"}], + "glib": [{"anchor": "smi", "kind": "recommended"}], + "dfu": [{"anchor": "glib", "kind": "required"}], +} +out = b._dependencies_come_first(list(order), edges) +assert out.index("glib") < out.index("smi"), out +assert out.index("glib") < out.index("dfu"), out +# a cycle made only of required edges must terminate, not spin +out2 = b._dependencies_come_first(["a", "b"], {"a": [{"anchor": "b", "kind": "required"}], + "b": [{"anchor": "a", "kind": "required"}]}) +assert set(out2) == {"a", "b"}, out2 +# A RECOMMENDATION IS A DEPENDENCY. texlive's deps are all recommended, so +# a required-only pass left it at 26 with FreeType at 46 -- and it built +# against a freetype that did not exist. +# ...AND REQUIRED HAS THE LAST WORD. With the recommended pass running +# second it could undo the strict one: xorg7-lib REQUIRES libxcb and still +# came out ahead of it (32 vs 37), with no edge back at all. +# A REAL TOPOLOGICAL SORT, not a repair pass. Three times a pass "fixed" +# an order and a later pass undid it (texlive, GLib, libxcb). Kahn's +# algorithm emits a node only when everything it depends on is out. +chain = b._dependencies_come_first( + ["c", "b", "a"], + {"c": [{"anchor": "b", "kind": "required"}], + "b": [{"anchor": "a", "kind": "recommended"}]}) +assert chain == ["a", "b", "c"], chain + +# BREAK ONLY INSIDE THE CYCLE, and release only the knot. "Fewest blocking +# dependencies" picked ffmpeg, whose recommended edge to libass was dropped +# (ffmpeg 179, libass 208) -- and the required-cycle fallback then emitted +# EVERY remaining node in its original order, throwing away the ordering of +# everything after the knot. +chain2 = b._dependencies_come_first( + ["ffmpeg", "libass", "harfbuzz", "freetype2"], + {"ffmpeg": [{"anchor": "libass", "kind": "recommended"}, + {"anchor": "freetype2", "kind": "recommended"}], + "libass": [{"anchor": "freetype2", "kind": "required"}, + {"anchor": "harfbuzz", "kind": "required"}], + "harfbuzz": [{"anchor": "freetype2", "kind": "recommended"}], + "freetype2": [{"anchor": "harfbuzz", "kind": "recommended"}]}) +assert chain2.index("libass") < chain2.index("ffmpeg"), chain2 +assert chain2.index("harfbuzz") < chain2.index("libass"), chain2 + +# BREAK THE LEAST. Dropping every recommended edge among the remaining +# nodes put ghostscript (112) before FreeType (130): gs RECOMMENDS FreeType +# and sits in a different cycle (gs -> cups -> cups-filters -> gs), and the +# blanket drop took its FreeType edge along with the cycle's. +mixed = b._dependencies_come_first( + ["gs", "cups", "cupsfilters", "ft"], + {"gs": [{"anchor": "ft", "kind": "recommended"}, + {"anchor": "cups", "kind": "recommended"}], + "cups": [{"anchor": "cupsfilters", "kind": "required"}], + "cupsfilters": [{"anchor": "gs", "kind": "required"}]}) +assert mixed.index("ft") < mixed.index("gs"), mixed +# a required-only cycle is left as it came, not shuffled forever +knot = b._dependencies_come_first( + ["x", "y"], {"x": [{"anchor": "y", "kind": "required"}], + "y": [{"anchor": "x", "kind": "required"}]}) +assert set(knot) == {"x", "y"} and len(knot) == 2, knot +strict = b._dependencies_come_first( + ["xorglib", "other", "xcb"], + {"xorglib": [{"anchor": "xcb", "kind": "required"}], + "other": [{"anchor": "xorglib", "kind": "recommended"}]}) +assert strict.index("xcb") < strict.index("xorglib"), strict +rec = b._dependencies_come_first( + ["tex", "cairo", "ft"], + {"tex": [{"anchor": "cairo", "kind": "recommended"}], + "cairo": [{"anchor": "ft", "kind": "required"}]}) +assert rec.index("ft") < rec.index("cairo") < rec.index("tex"), rec +PYORD + +# package-fixes.conf MUST PARSE. A duplicate section makes configparser +# raise, and the whole file is then ignored -- every fix in it silently +# stops applying. +python3 - <<'PYFIXCONF' || _p="$_p;package-fixes.conf does not parse (a duplicate section?)" +import configparser, os, sys +for cand in ("stacks/package-fixes.conf", + os.path.join(os.path.dirname(os.environ.get("LFS_TOOL", "")), + "stacks", "package-fixes.conf")): + if cand and os.path.isfile(cand): + configparser.ConfigParser().read(cand) + break +PYFIXCONF + +# THE LAYOUT NEEDS FLAGS TOO, with the same guard as the prefixes. +for _lf in base-dir pkgusr-subdir cfguser-subdir; do + grep -q -- "\"--$_lf\"" "$pm_src" \ + || _p="$_p;the layout cannot be set with a command (--$_lf)" +done +grep -q 'account homes already exist under the current layout' "$pm_src" \ + || _p="$_p;changing the layout does not check for existing homes" +# and the report must name the config file it really read +grep -q 'def _lfs_config_path' "$pm_src" \ + || _p="$_p;the report hardcodes the default config path" + +# A SETTING SAVED WHERE THE TOOL DOES NOT READ IS NOT SAVED. `config +# --pkgusr-prefix ""` wrote packagemanager.conf and said "Saved" -- while the +# value is read from config.json, which is authoritative. --show still said +# "p_, the lfs config". +grep -q 'def _save_shared_key' "$pm_src" \ + || _p="$_p;a shared key is written where it will not be read" +_ss="$T/sharedstore"; mkdir -p "$_ss" +printf '{"pkgusr_prefix": "p"}\n' > "$_ss/config.json" +LFS_STORE="$_ss" python3 - "$pm_src" <<'PYSHARED' >/dev/null 2>&1 +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._save_shared_key("pkgusr_prefix", "") +PYSHARED +python3 - "$_ss/config.json" <<'PYSHARED2' || _p="$_p;the authoritative config was not updated" +import sys, json +assert json.load(open(sys.argv[1]))["pkgusr_prefix"] == "", open(sys.argv[1]).read() +PYSHARED2 + +# THE PACKAGE-USER PREFIX MUST BE SETTABLE, AND GUARDED. It could only be +# changed by editing config.json by hand -- and it is the one prefix that +# would orphan every account on the system. +grep -q -- '"--pkgusr-prefix"' "$pm_src" \ + || _p="$_p;the package-user prefix cannot be set with a command" +grep -q 'def _accounts_with_prefix' "$pm_src" \ + || _p="$_p;changing the prefix does not check for existing accounts" +python3 - "$pm_src" <<'PYPFXGUARD' || _p="$_p;the existing-account check is wrong" +import sys, types, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +class E: + def __init__(s, n, d): s.pw_name, s.pw_dir = n, d +pm.pwd = types.SimpleNamespace(getpwall=lambda: [ + E("p_gcc", pm.BASE_DIR + "/pkgusr/p_gcc"), + E("root", "/root")]) +assert pm._accounts_with_prefix("p") == ["p_gcc"] +assert pm._accounts_with_prefix("zz") == [] +PYPFXGUARD + +# THE DIRECTORIES ARE CONFIGURABLE TOO. BASE_DIR has read `pkgusr_home` from +# packagemanager.conf for a long time; the two SUBDIRECTORIES were +# environment-only, so a tree with a different layout depended on the +# environment being right every time. +grep -q '_early_conf("pkgusr_subdir"' "$pm_src" \ + || _p="$_p;the package subdirectory cannot be set in the config file" +grep -q '_early_conf("cfguser_subdir"' "$pm_src" \ + || _p="$_p;the config-user subdirectory cannot be set in the config file" +grep -q '"pkgusr_subdir", "cfguser_subdir"' "$pm_src" \ + || _p="$_p;the layout keys do not travel with the tree" +_lay="$T/layout.conf" +printf 'pkgusr_subdir=packages\ncfguser_subdir=config\n' > "$_lay" +# ...and from the LFS config, which is what makes it travel with the tree +_lst="$T/layoutstore"; mkdir -p "$_lst" +printf '{"pkgusr_subdir": "packages"}\n' > "$_lst/config.json" +_lv3="$(LFS_STORE="$_lst" python3 - "$pm_src" <<'PYLAYOUT3' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +print(pm.PKGUSR_SUBDIR) +PYLAYOUT3 +)" +[ "$_lv3" = "packages" ] \ + || _p="$_p;the layout is not read from the lfs config (got: $_lv3)" +# and the report must name the real keys, not tell people to export variables +grep -q 'every value below can be set in either file' "$pm_src" \ + || _p="$_p;config --show still tells people to export variables" +_lv="$(PKGUSR_CONFIG="$_lay" python3 - "$pm_src" <<'PYLAYOUT' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +print(pm.PKGUSR_SUBDIR, pm.CFGUSR_SUBDIR) +PYLAYOUT +)" +[ "$_lv" = "packages config" ] \ + || _p="$_p;the layout keys are not read from the config (got: $_lv)" +_lv2="$(PKGUSR_CONFIG="$_lay" PKGUSR_SUBDIR=fromenv python3 - "$pm_src" <<'PYLAYOUT2' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +print(pm.PKGUSR_SUBDIR) +PYLAYOUT2 +)" +[ "$_lv2" = "fromenv" ] || _p="$_p;the environment no longer overrides the layout config" + +# THE BUILT SYSTEM'S CONFIG, WRITABLE BEFORE IT EXISTS. Until now the only +# way to give the new system a different prefix or layout was to wait for the +# tree and edit it there. +grep -q 'target-config.json' "$LFS_TOOL" \ + || _p="$_p;there is no host-side config for the built system" +grep -q 'def cmd_config_target' "$LFS_TOOL" \ + || _p="$_p;the target config cannot be shown or changed" +# ...and interactively, like `packagemanager config` and `env --ask` +grep -q 'INTERACTIVE, LIKE THE OTHERS' "$LFS_TOOL" \ + || _p="$_p;the target config has no interactive mode" +_tga="$T/tgtask"; mkdir -p "$_tga" +printf 'p\n\n\npackages\n\n\n' \ + | LFS_STORE="$_tga" python3 "$LFS_TOOL" config --target --ask >/dev/null 2>&1 +python3 - "$_tga/target-config.json" <<'PYASK' || _p="$_p;the interactive walk does not store answers" +import sys, json +d = json.load(open(sys.argv[1])) +assert d.get("pkgusr_prefix") == "p", d +assert d.get("pkgusr_subdir") == "packages", d +assert "cfguser_prefix" not in d, d # blank means "leave it alone" +PYASK +_tg="$T/tgtstore"; mkdir -p "$_tg" +LFS_STORE="$_tg" python3 "$LFS_TOOL" config --target pkgusr_prefix p </dev/null >/dev/null 2>&1 +python3 - "$_tg/target-config.json" <<'PYTGT' || _p="$_p;the target config is not written" +import sys, json +assert json.load(open(sys.argv[1]))["pkgusr_prefix"] == "p" +PYTGT +# and the tree's own config still wins over it +grep -q 'the tree overrides both' "$LFS_TOOL" \ + || _p="$_p;the target config would override a config the tree already has" + +# ...AND THE SESSION MUST NOT NAME ACCOUNTS FROM THE HOST'S PREFIX. The user +# set pkgusr_prefix empty on their host and the session that builds the NEW +# system showed "pkgusr_prefix = (unset)" -- the host's setting about to name +# another machine's accounts. And nothing said which config file was in use. +grep -q 'SAY WHICH FILE, AND WHOSE PREFIXES' "$LFS_TOOL" \ + || _p="$_p;the session config takes prefixes from the host without saying so" +grep -q 'Session config: (%s)" % config_path()' "$LFS_TOOL" \ + || _p="$_p;the session does not name the config file it is using" +# an EMPTY prefix is a decision (accounts named plainly), not a gap -- the +# display read "(unset)" for both, which is how a host's empty value nearly +# named a whole build system's accounts +grep -q 'accounts will be named without a prefix' "$LFS_TOOL" \ + || _p="$_p;an empty prefix is shown as if nothing had been chosen" + +# THE BUILD SYSTEM'S CONFIG IS ITS OWN. The chroot scripts export +# LFS_*_PREFIX, and the values were read from the HOST -- so a prefix changed +# on the user's own machine was baked into the build system's environment, +# where it overrode the tree's config. That is how an empty prefix reached a +# chroot whose config says "p". +grep -q 'def _tree_prefix' "$LFS_TOOL" \ + || _p="$_p;the chroot's exported prefixes come from the host" +python3 - "$LFS_TOOL" <<'PYTREEPFX' || _p="$_p;the tree's config does not decide its own prefixes" +import sys, os, json, tempfile, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +d = tempfile.mkdtemp() +os.makedirs(os.path.join(d, "usr/share/lfs")) +json.dump({"pkgusr_prefix": "p"}, + open(os.path.join(d, "usr/share/lfs/config.json"), "w")) +assert lfs._tree_prefix(d, "pkgusr_prefix", "") == "p" # tree wins +assert lfs._tree_prefix(d, "collector_prefix", "hostval") == "hostval" +assert lfs._tree_prefix("/nonexistent", "pkgusr_prefix", "hv") == "hv" +PYTREEPFX + +# THE PREFIXES AND DIRECTORIES MUST BE VISIBLE IN ONE PLACE. The user could +# not find where they are set -- fairly: prefixes come from config.json, then +# packagemanager.conf, then the environment, then a default, and the +# DIRECTORIES were environment-only and documented nowhere. +grep -q 'def _config_sources' "$pm_src" \ + || _p="$_p;nothing reports the prefixes and directories with their sources" +grep -q 'Prefixes and directories' "$pm_src" \ + || _p="$_p;config --show does not print the table" +python3 - "$pm_src" <<'PYCFGSRC' || _p="$_p;the config report is missing a value or a source" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +rows = pm._config_sources() +labels = [r[0] for r in rows] +for want in ("package-user prefix", "config-user prefix", "collector prefix", + "base directory", "package homes", "config-user homes"): + assert want in labels, (want, labels) +for label, value, source in rows: + assert value, label # every row has a value + assert source, label # and says where it came from +PYCFGSRC + +# ONE FILE, TWO READERS. package-fixes.conf carries build options (read by +# apply_machine_opts, emitted as -Dkey=value) AND graph keys like `needs` +# (read by blfs). Each must ignore the other's keys, or mesa gets +# ERROR: Unknown option: "needs". +python3 - "$pm_src" <<'PYTWOREADERS' || _p="$_p;a dependency key is passed to meson as a build option" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp() +conf = os.path.join(d, "fixes.conf") +open(conf, "w").write("[mesa]\nneeds = glslang\nllvm = enabled\n") +script = os.path.join(d, "install_Mesa-1") +open(script, "w").write("#!/bin/bash\nmeson setup ..\n") +os.environ["PM_PACKAGE_FIXES"] = conf +os.environ["PM_MACHINE_CONF"] = "/nonexistent" +pm.apply_machine_opts("mesa", script) +body = open(script).read() +assert "-Dneeds=" not in body, body +assert "-Dllvm=enabled" in body, body +# and a script that ALREADY has the bad flag (written before 1.13.24) is +# repaired, not left to fail forever: merging is idempotent, so nothing else +# would ever take it out +open(script, "w").write("#!/bin/bash\nmeson setup -Dneeds=glslang -Dllvm=enabled ..\n") +pm.apply_machine_opts("mesa", script) +body = open(script).read() +assert "-Dneeds=" not in body, body +assert "-Dllvm=enabled" in body, body +PYTWOREADERS + +# A DEPENDENCY THE BOOK DOES NOT RECORD. libass hard-requires harfbuzz in +# its configure and its page lists only FreeType, FriBidi, Fontconfig and +# NASM -- so no ordering could place it, and no entry order either, because +# both sit inside one plan. package-fixes.conf can add the edge. +_fx2="$(dirname "$LFS_TOOL")/stacks/package-fixes.conf" +if [ -f "$_fx2" ]; then + grep -q 'needs = harfbuzz' "$_fx2" \ + || _p="$_p;libass's undeclared harfbuzz dependency is not recorded" +fi +grep -q '_extra_deps_for' "$_bt" \ + || _p="$_p;the generator cannot be told about a dependency the book omits" +python3 - "$_bt" <<'PYXTRA' || _p="$_p;an added dependency is not shaped like a book one" +import sys, os, tempfile, importlib.machinery as m +d = tempfile.mkdtemp(); p = os.path.join(d, "fixes.conf") +open(p, "w").write("[libass]\nneeds = harfbuzz\n") +os.environ["BLFS_PACKAGE_FIXES"] = p +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +got = b._extra_deps_for("libass") +assert got and got[0]["anchor"] == "harfbuzz", got +assert got[0]["kind"] == "required", got +# every key a book-parsed dep has, or consumers KeyError on it +for k in ("anchor", "kind", "name_version", "external"): + assert k in got[0], (k, got[0]) +assert b._extra_deps_for("nothing-here") == [] +PYXTRA +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a required dependency is built before its dependant, whatever the recommendations say" +fi + +# ---- installing the tools must not re-mode the system ------------------------ # +# `install -d` applies its mode to a directory that already exists, so +# `make install` re-moded /usr/bin from 775 to 755 and silently broke the +# package-user system -- package users are in group install and need +# group-write there. The next build failed with permission errors that had +# nothing to do with the package. +_p="" +_mkf="$(dirname "$LFS_TOOL")/Makefile" +if [ -f "$_mkf" ]; then + grep -qE '^\s*\$\(INSTALL\) -d ' "$_mkf" \ + && _p="$_p;the Makefile creates directories with install -d, which re-modes existing ones" + grep -q 'MKDIR' "$_mkf" \ + || _p="$_p;the Makefile has no non-destructive way to create a directory" + # and for real: a group-writable /usr/bin must survive an install + _mkd="$T/makeinstall"; mkdir -p "$_mkd/usr/bin"; chmod 775 "$_mkd/usr/bin" + ( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_mkd" ) >/dev/null 2>&1 + [ "$(stat -c %a "$_mkd/usr/bin" 2>/dev/null)" = 775 ] \ + || _p="$_p;make install changed the mode of an existing /usr/bin ($(stat -c %a "$_mkd/usr/bin" 2>/dev/null))" + rm -rf "$_mkd" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "installing the tools leaves the system's directories exactly as they were" +fi + +# ---- the library cache, without touching anyone's symlinks ------------------ # +# /etc/ld.so.cache is rewritten by every package that installs a library, and +# ldconfig RENAMES a temp file over it -- which a sticky /etc allows only to +# the owner. So package users skip it and the tools refresh it afterwards. +# But ldconfig also creates SONAME symlinks, and run as root those land +# root-owned inside package trees (this is how libfreetype.so.6 came to be +# owned by p_libpaper). -X rebuilds the cache and touches no links. +_p="" +_slice_fn "$helper_src" make_wrappers | grep -q 'WRAPPERS/ldconfig' \ + || _p="$_p;a package user still runs ldconfig and fails on the sticky /etc" +grep -q 'ldconfig -X' "$helper_src" \ + || _p="$_p;the cache refresh may rewrite symlinks in package-owned directories" +grep -qE '^\s*soft .*-- ldconfig$' "$helper_src" \ + && _p="$_p;ldconfig is run without -X somewhere" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the library cache is refreshed as root, and no package's symlinks are touched" +fi + +# ---- rearranging a directory nothing filled --------------------------------- # +# With git's "untar the html docs" block commented, the page still +# reorganises what it would have extracted: +# mv /usr/share/doc/git-2.53.0/{git*.adoc,man-pages/text} +# mv: cannot stat '/usr/share/doc/git-2.53.0/git*.adoc' +_p="" +python3 - "$_bt" <<'PYSHUFFLE' || _p="$_p;a move from a directory nothing fills still runs" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_orphan_tree_shuffles( + "# tar -xf ../git-htmldocs.tar.xz -C /usr/share/doc/git-2.53.0 &&\n" + "mkdir -vp /usr/share/doc/git-2.53.0/man-pages/{html,text} &&\n" + "mv /usr/share/doc/git-2.53.0/{git*.adoc,man-pages/text}") +for line in out.splitlines(): + assert line.startswith("#"), line # the whole chain goes +# a directory something DOES fill keeps its mv +keep = b._comment_orphan_tree_shuffles( + "make install DESTDIR=/usr/share/doc/foo\nmv /usr/share/doc/foo/a /usr/share/doc/foo/b") +assert not keep.splitlines()[1].startswith("#"), keep +PYSHUFFLE +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a move out of a directory only a skipped step would fill is skipped too" +fi + +# ---- installing docs nothing built ------------------------------------------ # +# Git generates its man pages and html docs in an optional block the guard +# skips, then installs them unconditionally -- so `make install-man` tried to +# build them and died on `asciidoc: command not found`, after everything else +# had installed. +_p="" +python3 - "$_bt" <<'PYDOCINST' || _p="$_p;an install of docs nothing built still runs" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +# nothing builds them -> skipped, and the chain after goes too +out = b._comment_orphan_doc_installs( + "# make html\nmake install &&\nmake htmldir=/x install-html &&\n" + "mkdir -vp /x/html &&\nmv /x/a /x/html") +assert "# make htmldir=/x install-html &&" in out, out +assert "# mkdir -vp /x/html &&" in out, out +assert "# mv /x/a /x/html" in out, out +assert "\nmake install &&" in out, out # the real install survives +# something DOES build them -> kept +keep = b._comment_orphan_doc_installs("make man\nmake install\nmake install-man") +assert keep == "make man\nmake install\nmake install-man", keep +PYDOCINST +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an install-<x> whose generation step was skipped is skipped with its chain" +fi + +# ---- the directory was never the problem ------------------------------------ # +# /usr/include/gstreamer-1.0/gst/audio -- nothing to grant (already writable) +# The package CAN write there. It cannot finish installing over a FILE owned +# by another package: meson copies the contents, then chmod/chown the +# destination, which only its owner may do -> EPERM, reported as "Unhandled +# python OSError" with no path. Ownership follows the installer. +_p="" +grep -q '^meson_takeover_from_log()' "$helper_src" \ + || _p="$_p;a meson install over another package's file is never repaired" +_slice_fn "$helper_src" cmd_build | grep -q 'meson_takeover_from_log "$log"' \ + || _p="$_p;the repair loop does not try the file takeover" +_mt="$T/mt"; mkdir -p "$_mt/dest"; echo old > "$_mt/dest/hdr.h" +printf 'Installing /src/hdr.h to %s\nERROR: Unhandled python OSError\n' \ + "$_mt/dest" > "$_mt/log" +_mtrun() { # $1 = owner reported for the file + { echo 'say(){ echo "$*"; }'; echo 'soft(){ shift 2; "$@"; }' + echo 'real_chown(){ echo "CHOWN $*"; }'; echo 'pkgusr_kind(){ echo pkg; }' + printf 'stat(){ case "$*" in *hdr.h) echo %s ;; *) command stat "$@" ;; esac; }\n' "$1" + python3 - "$helper_src" <<'PYMT' +import sys +s = open(sys.argv[1]).read() +i = s.index('meson_takeover_from_log() {') +d, j = 0, i +while True: + if s[j] == '{': d += 1 + elif s[j] == '}': + d -= 1 + if d == 0: break + j += 1 +print(s[i:j+1]) +PYMT + printf 'meson_takeover_from_log %s p_installer\n' "$_mt/log" + } | bash 2>/dev/null +} +_mtrun p_other | grep -q 'CHOWN p_installer' \ + || _p="$_p;a file owned by another package is not handed to the installer" +_mtrun root | grep -q 'CHOWN' \ + && _p="$_p;root's files are taken over" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a file being installed over is handed to the installer, and root's are not" +fi + +# ---- a group on a leaf is no use if the chain is closed --------------------- # +# Writing into .../gst/audio needs SEARCH on every directory above it, and +# those belong to other packages. And the loop printed a header and then +# nothing when every candidate was skipped, so a failing package looked like +# a repair that had not run. +_p="" +_slice_fn "$helper_src" grant_dir_access | grep -q 'A GROUP ON A LEAF IS NO USE' \ + || _p="$_p;a grant does not check that the path above it is searchable" +_slice_fn "$helper_src" auto_grant_from_log | grep -q 'nothing to grant (already writable' \ + || _p="$_p;a skipped candidate is silent, so the repair looks like it never ran" +if [ "$(id -u)" = 0 ] && id -u nobody >/dev/null 2>&1; then + _ch="$T/chain"; mkdir -p "$_ch/a/b/c"; chmod 750 "$_ch/a"; chmod 755 "$_ch/a/b" + _up="$_ch/a/b/c" + while :; do + _up="$(dirname "$_up")" + case "$_up" in /|"$T"|"") break ;; esac + [ -d "$_up" ] || break + su -s /bin/sh -c "test -x '$_up'" nobody 2>/dev/null && break + chmod o+x "$_up" + done + su -s /bin/sh -c "test -x '$_ch/a/b/c'" nobody 2>/dev/null \ + || _p="$_p;the search path was not opened all the way down" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a grant opens the search path above it, and every candidate reports its outcome" +fi + +# ---- one grant per DIRECTORY, not per tree ---------------------------------- # +# The dedup collapsed every failing path to its tree top, so once one +# directory under /usr/include/gstreamer-1.0 was granted, the one that +# actually failed (.../gst/audio) was skipped -- two repair rounds, same +# failure. +_p="" +_slice_fn "$helper_src" auto_grant_from_log | grep -q 'DEDUP BY DIRECTORY, NOT BY TREE TOP' \ + || _p="$_p;the grant loop still dedups by tree top" +_gn="$T/need.txt" +printf '%s\n' /usr/include/gstreamer-1.0/gst/uridownloader \ + /usr/include/gstreamer-1.0/gst/audio \ + /usr/include/gstreamer-1.0/gst/audio > "$_gn" +_gout="$({ printf 'need="$(cat %s)"\n' "$_gn" + echo 'granted=0; user=p_x' + echo 'grant_dir_access(){ echo "GRANT $1"; }' + echo 'collector_tree_top(){ echo /usr/include/gstreamer-1.0; }' + _slice_fn "$helper_src" auto_grant_from_log \ + | sed -n '/# DEDUP BY DIRECTORY, NOT BY TREE TOP/,/done <<< "\$need"/p' \ + | sed 's/^ //;s/^local /_x=/' + } | bash 2>/dev/null | grep -c GRANT)" +[ "$_gout" = 2 ] \ + || _p="$_p;the grant loop skipped a distinct directory (granted $_gout of 2)" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every distinct directory that failed gets its own grant" +fi + +# ---- meson says "OSError" and names nothing ---------------------------------- # +# gst-plugins-bad ended with an unhandled OSError; the directory it could not +# write is on the line BEFORE it, which is the only place meson puts it. Two +# repair rounds ran with nothing to grant. +_p="" +_slice_fn "$helper_src" unwritable_dirs_from_log | grep -q 'EVERY DESTINATION IT PRINTED' \ + || _p="$_p;a meson install failure yields only the last directory" +# meson installs through PYTHON, not our wrappers, so nothing on our side +# sees the EACCES -- the only record is every destination it announced. +_mall="$T/mesonall.log" +cat > "$_mall" <<'MESONALL' +Installing /src/a.h to /usr/include/gstreamer-1.0/gst/uridownloader +Installing /src/b.h to /usr/include/gstreamer-1.0/gst/audio +ERROR: Unhandled python OSError. This is probably not a Meson bug +MESONALL +_mc="$(tail -c 2097152 "$_mall" | awk '/^Installing .* to \//{ print $NF }' | sort -u | wc -l)" +[ "$_mc" = 2 ] \ + || _p="$_p;not every announced destination is collected (got $_mc)" +# ...and a directory the package can already write must be skipped +_slice_fn "$helper_src" grant_dir_access | grep -q 'ALREADY WRITABLE IS NOTHING TO GRANT' \ + || _p="$_p;a writable directory would still get a new group" +_ml="$T/meson.log" +cat > "$_ml" <<'MESONEOF' +Installing /src/a.h to /usr/include/gstreamer-1.0/gst/audio +ERROR: Unhandled python OSError. This is probably not a Meson bug +MESONEOF +_mg="$(tail -c 524288 "$_ml" | awk ' + /^Installing .* to \// { last = $NF } + /Unhandled python OSError|\[Errno 13\]|Permission denied/ { if (last != "") print last }' | sort -u)" +[ "$_mg" = "/usr/include/gstreamer-1.0/gst/audio" ] \ + || _p="$_p;the meson install destination is not extracted (got: $_mg)" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a meson install error resolves to the directory it was installing into" +fi + +# ---- a relative complaint needs the cd that preceded it --------------------- # +# git's install pipes into tar: +# (cd '/usr/lib/perl5/5.42/site_perl' && umask 022 && tar xof -) +# /usr/bin/tar: ./Git: Cannot mkdir: Permission denied +# The path in the error is RELATIVE, so the permission repair found nothing to +# grant -- while the directory needing it is named one line above. +_p="" +_slice_fn "$helper_src" unwritable_dirs_from_log | grep -q 'A RELATIVE PATH NEEDS THE' \ + || _p="$_p;a relative permission error yields no directory to grant" +_rl="$T/relcd.log" +cat > "$_rl" <<'RELEOF' +(cd perl/build/lib && tar cf - .) | \ +(cd '/usr/lib/perl5/5.42/site_perl' && umask 022 && tar xof -) +/usr/bin/tar: ./Git: Cannot mkdir: Permission denied +RELEOF +_got="$({ printf 'log=%s\n' "$_rl" + _slice_fn "$helper_src" unwritable_dirs_from_log \ + | sed -n '/# (plain awk: match() with a capture array/,/sort -u)"/p' \ + | sed 's/^ //;s/^local /_x=/' + echo 'printf "%s" "$_cdpaths"'; } | bash 2>/dev/null)" +[ "$_got" = "/usr/lib/perl5/5.42/site_perl" ] \ + || _p="$_p;the cd before a relative error is not resolved (got: $_got)" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a relative permission error resolves to the directory the shell cd'd into" +fi + +# ---- tar, like cp, must not fail on someone else's directory ---------------- # +# Git installs its translations through tar, which then stamps the +# destination DIRECTORIES -- owned by other packages, sticky because we +# sealed them. 104 files had installed correctly and the build failed: +# tar: ./zh_TW: Cannot change mode to rwxr-xr-t: Operation not permitted +_p="" +grep -q 'WRAPPERS/tar' "$helper_src" \ + || _p="$_p;tar is not wrapped, so a package cannot extract into a shared directory" +_tw="$T/tarwrap"; mkdir -p "$_tw" +sed -n '/^real=\/usr\/bin\/tar/,/^exit \$rc/p' "$helper_src" > "$_tw/body" +if [ -s "$_tw/body" ]; then + printf '#!/bin/bash\n' > "$_tw/meta.sh" + printf 'echo "tar: ./zh_TW: Cannot utime: Operation not permitted" >&2\n' >> "$_tw/meta.sh" + printf 'echo "tar: Exiting with failure status due to previous errors" >&2\nexit 2\n' >> "$_tw/meta.sh" + printf '#!/bin/bash\necho "tar: x: Cannot open: No such file" >&2\nexit 2\n' > "$_tw/real.sh" + chmod 755 "$_tw/meta.sh" "$_tw/real.sh" + for _k in meta real; do + { printf '#!/bin/bash\n'; sed "s|^real=/usr/bin/tar|real=$_tw/$_k.sh|" "$_tw/body"; } > "$_tw/w_$_k" + chmod 755 "$_tw/w_$_k" + done + "$_tw/w_meta" -xf x >/dev/null 2>&1 \ + || _p="$_p;the tar wrapper fails on a harmless metadata complaint" + "$_tw/w_real" -xf x >/dev/null 2>&1 \ + && _p="$_p;the tar wrapper swallows a real error" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "tar tolerates stamping a directory it does not own, and nothing else" +fi + +# ---- verify checks the package you named ------------------------------------ # +# `lfs-helper verify xorg7-lib --fix` read only $1 for the flag, ignored the +# name, and started walking all 533 packages. +_p="" +_slice_fn "$helper_src" cmd_verify | grep -q 'ONE PACKAGE, IF ONE IS NAMED' \ + || _p="$_p;verify ignores a package name and scans everything" +_slice_fn "$helper_src" cmd_verify | grep -q '_vfy_manifest_ownership "$fix" "$only"' \ + || _p="$_p;the named package is not passed to the ownership pass" +_slice_fn "$helper_src" _vfy_manifest_ownership | grep -q 'or just the one the caller named' \ + || _p="$_p;the ownership pass cannot be narrowed to one package" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "verify checks only the package you name" +fi + +# ---- the shell startup helpers BLFS snippets call ---------------------------- # +# BLFS pages drop /etc/profile.d snippets that call pathprepend; the book +# defines those in postlfs/profile and LFS does not install them, so rust's +# rustc.sh said "pathprepend: command not found" on every package-user login. +_p="" +grep -q '^install_profile_helpers()' "$helper_src" \ + || _p="$_p;nothing installs the path helpers BLFS snippets call" +_slice_fn "$helper_src" cmd_init_pkgusr | grep -q 'install_profile_helpers' \ + || _p="$_p;setting up the package-user system does not install them" +# ...and the command people ACTUALLY run on a fresh system is the bootstrap, +# `packagemanager setup` -- wiring it only into cmd_init_pkgusr meant nothing +# happened. +grep -q 'install-profile)' "$helper_src" \ + || _p="$_p;there is no way to install the profile helpers on their own" +# and its one question takes --yes on the command line: the difference is +# whether /opt/rustc/bin is on PATH, and asking people to export a variable +# to answer it is a poor trade +grep -q -- '-y|--yes) LFS_ASSUME_YES=1' "$helper_src" \ + || _p="$_p;install-profile cannot be answered without setting an environment variable" +grep -q '"lfs-helper", "install-profile"' "$pm_src" \ + || _p="$_p;the bootstrap does not install the profile helpers" +if [ "$(id -u)" = 0 ]; then + _pfd="$T/profilehelp"; mkdir -p "$_pfd" + cat > "$T/pfrun.sh" <<PFOUTER +SNAP_ROOT=$_pfd +ok(){ :; }; warn(){ :; } +set_install_dir_owner(){ :; }; real_chmod(){ chmod "\$@"; } +PFOUTER + python3 - "$helper_src" >> "$T/pfrun.sh" <<'PFPY' +import sys +s = open(sys.argv[1]).read() +i = s.index('install_profile_helpers() {') +d, j = 0, i +while True: + if s[j] == '{': d += 1 + elif s[j] == '}': + d -= 1 + if d == 0: break + j += 1 +print(s[i:j+1]); print('install_profile_helpers') +PFPY + bash "$T/pfrun.sh" >/dev/null 2>&1 + [ -f "$_pfd/etc/profile.d/00-path-functions.sh" ] \ + || _p="$_p;the path helpers were not written" + # and it must SAY it wrote them: reporting only what it declined to do + # left the user believing nothing had been installed + _slice_fn "$helper_src" install_profile_helpers 2>/dev/null | grep -q 'wrote .*00-path-functions' \ + || grep -q 'wrote \$d/00-path-functions.sh' "$helper_src" \ + || _p="$_p;installing the helpers is silent" + # an existing /etc/profile that lacks the loop is OFFERED, not imposed + grep -q 'append a loop to /etc/profile' "$helper_src" \ + || _p="$_p;an /etc/profile without the profile.d loop is only complained about" + grep -q 'before-lfs-helper' "$helper_src" \ + || _p="$_p;/etc/profile would be appended to without a backup" + _pv="$(bash -c "PATH=/usr/bin; . $_pfd/etc/profile.d/00-path-functions.sh; \ + pathprepend /opt/r/bin; pathappend /opt/x/bin; echo \$PATH" 2>/dev/null)" + [ "$_pv" = "/opt/r/bin:/usr/bin:/opt/x/bin" ] \ + || _p="$_p;the path helpers do not work ($_pv)" + # THE WHOLE CHAPTER, not just the path helpers. "The Bash Shell Startup + # Files" creates /etc/profile, /etc/bashrc, four profile.d snippets and + # four skeleton dotfiles; we had one of them, so a package user's shell + # had no umask policy, no INPUTRC, no locale and no ~/.bashrc. + grep -q '^install_shell_startup_files()' "$helper_src" \ + || _p="$_p;only the path helpers are installed, not the startup files" + # (written as "$d/umask.sh" and "$skel/.bashrc" in the source, so match + # the leaf name; and check they really appear on disk below) + for _sf in umask.sh readline.sh i18n.sh extrapaths.sh; do + grep -q "/$_sf\"" "$helper_src" \ + || _p="$_p;$_sf from the startup-files chapter is not installed" + done + for _sk in .bash_profile .bashrc .bash_logout .profile; do + grep -q "skel/$_sk\"" "$helper_src" \ + || _p="$_p;the skeleton $_sk is not installed" + done + grep -q 'kept existing' "$helper_src" \ + || _p="$_p;the startup files would overwrite what the administrator has" + + # an existing /etc/profile is never overwritten + printf 'mine\n' > "$_pfd/etc/profile" + bash "$T/pfrun.sh" >/dev/null 2>&1 + [ "$(cat "$_pfd/etc/profile")" = "mine" ] \ + || _p="$_p;an existing /etc/profile was overwritten" + rm -rf "$_pfd" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the path helpers BLFS snippets call are installed, and your profile is not touched" +fi + +# ---- staging a file onto itself is a no-op, not an error --------------------- # +# pm-install copies the script into the account's home, and the phase runner +# stages it again to the same name. Invisible while the home was MISPLACED +# (two different paths); once fix-home repointed the home mid-run they became +# one file and install(1) refused: +# install: '.../install_p_nss' and '.../install_p_nss' are the same file +_p="" +[ "$(_slice_fn "$helper_src" cmd_build | grep -c '\-ef "\$staged"')" -ge 2 ] \ + || _p="$_p;staging does not notice that the source IS the destination" +[ "$(grep -c '\-ef "\$staged"' "$helper_src")" -ge 2 ] \ + || _p="$_p;only one of the two staging paths checks for the same file" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a script already in place is left alone instead of copied onto itself" +fi + +# ---- a shared INDEX is shared, not owned ------------------------------------- # +# Every package that installs an info page runs install-info against +# /usr/share/info/dir. As a package user that is +# install-info: Operation not permitted for /usr/share/info/dir +# and it killed nettle after its libraries were already installed. The file +# is never-claimed (nobody owns it) but nothing made it writable by the +# people who must update it: a shared directory is root:install and +# group-writable, and a shared file is the same rule one level down. +_p="" +_slice_fn "$helper_src" never_claim_list | grep -q '/usr/share/info/dir' \ + || _p="$_p;the info index is claimable by a package" +grep -q '^shared_files_list()' "$helper_src" \ + || _p="$_p;there is no list of files every package must be able to rewrite" +_slice_fn "$helper_src" shared_files_list | grep -q 'info/dir' \ + || _p="$_p;the info index is not treated as a shared file" +# the XML catalogs are the same species, one directory over: docbook-xml +# creates /etc/xml/catalog as its own user and docbook-xsl then cannot write +# it ("could not open /etc/xml/catalog for saving") +_slice_fn "$helper_src" shared_files_list | grep -q '/etc/xml/catalog' \ + || _p="$_p;the XML catalog is not treated as a shared file" +# the whole family of regenerated caches, added at once rather than one +# failure at a time (update-desktop-database, update-mime-database, +# glib-compile-schemas, gtk-update-icon-cache) +for _sf in applications/mimeinfo.cache mime/mime.cache \ + schemas/gschemas.compiled icons/hicolor/icon-theme.cache; do + _slice_fn "$helper_src" shared_files_list | grep -q "$_sf" \ + || _p="$_p;$_sf is not shared -- the package that runs its update tool will fail" +done +# and the GROUP is half the rule: a dir left root:root 775 is group-writable +# for group root, which helps no package user +_slice_fn "$helper_src" share_shared_files | grep -q 'real_chown root:install "\$d"' \ + || _p="$_p;a shared directory gets the mode but not the install group" +# "answer for me" must mean the same thing whoever asked: the group-choice +# prompt honoured LFS_ASSUME_YES while packagemanager --yes sets PM_YES, so +# `stack --run --yes` still stopped to ask which collector group to use +grep -q '^assume_yes()' "$helper_src" \ + || _p="$_p;there is no single meaning for assume-yes" +for _v in LFS_ASSUME_YES PM_YES; do + _ayout="$({ _slice_fn "$helper_src" assume_yes + echo 'assume_yes && echo YES || echo ASK'; } | env "$_v=1" bash 2>/dev/null)" + [ "$_ayout" = YES ] || _p="$_p;$_v=1 does not answer the prompts" +done +_ayout="$({ _slice_fn "$helper_src" assume_yes + echo 'assume_yes && echo YES || echo ASK'; } | env -u LFS_ASSUME_YES -u PM_YES bash 2>/dev/null)" +[ "$_ayout" = ASK ] || _p="$_p;the prompts are skipped even when nobody said yes" +_slice_fn "$helper_src" cmd_init_pkgusr | grep -q 'share_shared_files' \ + || _p="$_p;setting up the package-user system does not share the index" +_slice_fn "$helper_src" cmd_build | grep -q 'share_shared_files' \ + || _p="$_p;a build that cannot write the index does not repair it and retry" +_slice_fn "$helper_src" cmd_verify | grep -q 'share_shared_files' \ + || _p="$_p;verify --fix leaves the index unwritable" +# and it really applies the rule, idempotently +_sfd="$T/sharedfile"; mkdir -p "$_sfd/usr/share/info" +: > "$_sfd/usr/share/info/dir"; chmod 600 "$_sfd/usr/share/info/dir" +if [ "$(id -u)" = 0 ] && getent group install >/dev/null 2>&1; then + _sfout="$({ echo "SNAP_ROOT=$_sfd"; echo 'detail(){ echo "D: $*"; }' + echo 'real_chown(){ chown "$@"; }'; echo 'real_chmod(){ chmod "$@"; }' + _slice_fn "$helper_src" shared_files_list + _slice_fn "$helper_src" share_shared_files + echo 'share_shared_files; share_shared_files' + } | bash 2>/dev/null)" + [ "$(stat -c %G "$_sfd/usr/share/info/dir" 2>/dev/null)" = install ] \ + || _p="$_p;the shared index did not get group install" + [ "$(stat -c %a "$_sfd/usr/share/info/dir" 2>/dev/null)" = 664 ] \ + || _p="$_p;the shared index is not group-writable" + [ "$(echo "$_sfout" | grep -c '^D:')" = 1 ] \ + || _p="$_p;sharing the index is not idempotent (it reports work every run)" + # THE MODE OF THE FILE IS ONLY HALF OF IT. install-info renames a new + # file over the old one, and a STICKY directory allows that only to the + # file's owner -- EPERM, whatever the file's group-write says. Reproduce + # it, then check the rule removes it. + chmod 1775 "$_sfd/usr/share/info" 2>/dev/null + chown root:install "$_sfd/usr/share/info" "$_sfd/usr/share/info/dir" 2>/dev/null + useradd -M -N -g install pmshtest >/dev/null 2>&1 + _renam() { su pmshtest -c ": > $_sfd/usr/share/info/d.new && mv -f $_sfd/usr/share/info/d.new $_sfd/usr/share/info/dir" >/dev/null 2>&1; } + if id pmshtest >/dev/null 2>&1; then + _renam && _p="$_p;the fixture is wrong: a sticky dir should have blocked this" + { echo "SNAP_ROOT=$_sfd"; echo 'detail(){ :; }' + echo 'real_chown(){ chown "$@"; }'; echo 'real_chmod(){ chmod "$@"; }' + _slice_fn "$helper_src" shared_files_list + _slice_fn "$helper_src" shared_file_dirs + _slice_fn "$helper_src" share_shared_files + echo 'share_shared_files'; } | bash >/dev/null 2>&1 + _renam || _p="$_p;a package user still cannot replace the shared index (the directory is sticky)" + userdel pmshtest >/dev/null 2>&1 + fi + # THE INDEX IS USUALLY CREATED MID-BUILD, by the first package that ships + # an info page -- owned by THAT package user. Sharing it only at + # init-pkgusr (when it does not exist yet) or after a failure means one + # guaranteed failure per fresh LFS base build. + if id p_shA >/dev/null 2>&1 || useradd -M -N -g install p_shA >/dev/null 2>&1; then + chown root:install "$_sfd/usr/share/info" 2>/dev/null + chmod 775 "$_sfd/usr/share/info" 2>/dev/null + rm -f "$_sfd/usr/share/info/dir" + su p_shA -c "echo idx > $_sfd/usr/share/info/dir" >/dev/null 2>&1 + { echo "SNAP_ROOT=$_sfd"; echo 'detail(){ :; }' + echo 'real_chown(){ chown "$@"; }'; echo 'real_chmod(){ chmod "$@"; }' + _slice_fn "$helper_src" shared_files_list + _slice_fn "$helper_src" shared_file_dirs + _slice_fn "$helper_src" share_shared_files + echo 'share_shared_files'; } | bash >/dev/null 2>&1 + [ "$(stat -c %U:%G "$_sfd/usr/share/info/dir" 2>/dev/null)" = "root:install" ] \ + || _p="$_p;an index created BY a package user is not taken into shared ownership" + userdel p_shA >/dev/null 2>&1 + fi + _slice_fn "$helper_src" cmd_build | grep -q 'pkgusr_ready && share_shared_files' \ + || _p="$_p;the indexes are only shared AFTER a build fails, not before it runs" + + # and sealing must not put the sticky bit back + _slice_fn "$helper_src" cmd_seal_install_dirs | grep -q 'shared_file_dirs' \ + || _p="$_p;sealing would make the index unreplaceable again" + _slice_fn "$helper_src" _vfy_install_dirs | grep -q 'shared_file_dirs\|_shared_dirs' \ + || _p="$_p;verify would seal the index directory back" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a shared index file gets the shared treatment, everywhere it matters" +fi +rm -rf "$_sfd" + +# ---- downloads follow the commands, not the labels --------------------------- # +# alsa-ucm-conf is labelled "Recommended file:" and is used by a command that +# is NOT conditional, so a label filter ("required" + .patch) did not fetch +# it and the install died on `tar: ../alsa-ucm-conf...: Cannot open`. What a +# file is called does not decide whether it is needed; what runs does. +_p="" +_bt4="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt4" ]; then + python3 - "$_bt4" <<'PYDL' || _p="$_p;additional downloads are chosen by label instead of by what the commands use" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +data = {"additional_links": ["https://x/needed.tar.bz2", "https://x/unused.tar.bz2", + "https://x/fix.patch"], + "build_cmds": "make\n#\ttar -xf ../unused.tar.bz2", + "install_root_cmds": "make install &&\ntar -xf ../needed.tar.bz2", + "install_cmds": "tar -xf ../unused.tar.bz2"} +keep = b._needed_additional_links(data, []) +assert "https://x/needed.tar.bz2" in keep, keep +assert "https://x/fix.patch" in keep, keep # patches always +assert "https://x/unused.tar.bz2" not in keep, keep # only a commented line uses it +PYDL +fi +# an optional artifact that was never built is not a failed install +_slice_fn "$helper_src" make_wrappers | grep -q 'nothing to install' \ + || _p="$_p;install still fails the package when an optional artifact was never built" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "what gets downloaded and what gets installed follow the commands, not the labels" +fi + +# ---- book prose is data, not shell ------------------------------------------- # +# "Installed Directories: $TEXLIVE_PREFIX/bin" is documentation. Emitted in +# double quotes it became an expansion, and pm-install (which sources the +# script for its metadata, under set -u) died before the install started. +_p="" +_bt3="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt3" ]; then + python3 - "$_bt3" <<'PYBS' || _p="$_p;book strings are emitted so the shell can expand them" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b.bash_str('$TEXLIVE_PREFIX/bin, `date`, "x"') +assert out.startswith("'") and out.endswith("'"), out +assert '$TEXLIVE_PREFIX' in out and '`date`' in out +assert b.bash_str("it's") == "'it'\\''s'", b.bash_str("it's") +PYBS +fi +# and our shell options must not govern a file we merely source +_slice_fn "$helper_src" cmd_pm_install | grep -q 'set +u +e' \ + || _p="$_p;sourcing a book script imposes our set -u on it" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "book prose stays data, and our shell options stay ours" +fi + +# ---- a file the installer may not replace ------------------------------------ # +# freetype2 could not install its own freetype2.pc: the file was owned by +# p_cairo (the over-claiming manifests, made real by a verify --fix) and +# /usr/lib/pkgconfig is sticky, so only the owner may replace it -- EPERM, +# which no collector group can grant away. In this model the package that +# installs a file owns it, so the installer takes it over. +_p="" +_tod="$T/takeover"; mkdir -p "$_tod/usr/lib/pkgconfig" "$_tod/src" +if [ "$(id -u)" = 0 ] && command -v useradd >/dev/null 2>&1; then + for _u in p_tkA p_tkB; do id "$_u" >/dev/null 2>&1 || useradd -M -U "$_u" >/dev/null 2>&1; done + : > "$_tod/usr/lib/pkgconfig/ft.pc"; chown p_tkA:p_tkA "$_tod/usr/lib/pkgconfig/ft.pc" 2>/dev/null + : > "$_tod/usr/lib/pkgconfig/root.pc" + : > "$_tod/src/build.o"; chown p_tkA:p_tkA "$_tod/src/build.o" 2>/dev/null + printf "install: cannot remove '%s/usr/lib/pkgconfig/ft.pc': Operation not permitted\ninstall: cannot remove '%s/usr/lib/pkgconfig/root.pc': Operation not permitted\ninstall: cannot remove '%s/src/build.o': x\n" \ + "$_tod" "$_tod" "$_tod" > "$_tod/log" + { echo "SNAP_ROOT=$_tod; SRCROOT=$_tod/src"; echo 'warn(){ :; }' + echo 'real_chown(){ chown "$@"; }'; echo 'user_exists(){ id "$1" >/dev/null 2>&1; }' + _slice_fn "$helper_src" under_dir + _slice_fn "$helper_src" takeover_files_from_log + echo 'takeover_files_from_log "$SNAP_ROOT/log" p_tkB'; } | bash >/dev/null 2>&1 + [ "$(stat -c %U "$_tod/usr/lib/pkgconfig/ft.pc" 2>/dev/null)" = p_tkB ] \ + || _p="$_p;a mis-owned file is not taken over by the package installing it" + [ "$(stat -c %U "$_tod/usr/lib/pkgconfig/root.pc" 2>/dev/null)" = root ] \ + || _p="$_p;a ROOT-owned file was taken over (root's files are not another package's mistake)" + [ "$(stat -c %U "$_tod/src/build.o" 2>/dev/null)" = p_tkA ] \ + || _p="$_p;a file inside a build tree was taken over" + userdel p_tkA >/dev/null 2>&1; userdel p_tkB >/dev/null 2>&1 +fi +_slice_fn "$helper_src" cmd_build | grep -q 'takeover_files_from_log' \ + || _p="$_p;the retry loop never takes over a mis-owned file (granting cannot fix EPERM)" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "ownership follows the installer: a mis-owned file is taken over, root's is not" +fi +rm -rf "$_tod" + +# ---- a directory that is not there cannot be granted ------------------------- # +# cmake installs into /usr/share/vim/vimfiles/indent: vim's tree, a +# subdirectory that never existed. A package user cannot create it, and +# granting cannot fix a directory that is NOT THERE. The model's answer for +# a shared directory (root:install, group-writable) is now applied on demand. +_p="" +_mdd="$T/missingdirs" +mkdir -p "$_mdd/usr/share/vim" "$_mdd/etc" "$_mdd/src/pkgusr/p_x/src/docs" +{ + printf 'file INSTALL cannot make directory "%s/usr/share/vim/vimfiles/indent": No such file or directory.\n' "$_mdd" + # bash failing to redirect -- a DIFFERENT wording for the same problem, + # which the enumerate-the-messages version missed (libpaper, /etc/profile.d) + printf '%s/src/pkgusr/p_libpaper/install_libpaper: line 150: %s/etc/profile.d/libpaper.sh: No such file or directory\n' "$_mdd" "$_mdd" + # a SOURCE the build never produced, inside the build tree: not ours + printf "install: cannot stat '%s/src/pkgusr/p_x/src/docs/man/x.1': No such file or directory\n" "$_mdd" + # a missing file whose directory EXISTS: the error is about the file + printf 'cat: %s/usr/share/vim/gone.txt: No such file or directory\n' "$_mdd" +} > "$_mdd/log" +_mdout="$({ echo "SNAP_ROOT=$_mdd; SRCROOT=$_mdd/src; PM_YES=1" + echo 'ok(){ :; }'; echo 'warn(){ :; }'; echo 'say(){ :; }' + echo 'soft(){ shift 2; "$@"; }' + echo 'set_install_dir_owner(){ :; }'; echo 'real_chmod(){ chmod "$@"; }' + _slice_fn "$helper_src" assume_yes + _slice_fn "$helper_src" confirm_change + _slice_fn "$helper_src" missing_dirs_from_log + _slice_fn "$helper_src" create_missing_dirs_from_log + echo 'missing_dirs_from_log "$SNAP_ROOT/log"' + echo 'create_missing_dirs_from_log "$SNAP_ROOT/log" p_cmake' + } | bash 2>/dev/null)" +echo "$_mdout" | grep -q "vimfiles/indent" \ + || _p="$_p;a missing destination directory is not detected" +echo "$_mdout" | grep -q "etc/profile.d" \ + || _p="$_p;a shell redirection into a missing directory is not recognised (only some wordings are)" +echo "$_mdout" | grep -q "docs/man" \ + && _p="$_p;a missing SOURCE file in the build tree is mistaken for a directory" +[ -d "$_mdd/usr/share/vim/vimfiles/indent" ] \ + || _p="$_p;the missing shared directory was not created" +[ -d "$_mdd/etc/profile.d" ] \ + || _p="$_p;the redirection's missing directory was not created" +[ -d "$_mdd/src/pkgusr/p_x/src/docs/man" ] \ + && _p="$_p;a directory was created inside a package's build tree" +[ -d "$_mdd/usr/share/vim/gone.txt" ] \ + && _p="$_p;a directory was invented where only a FILE was missing" +# A RETRY MUST NOT THROW AWAY A BUILD THAT SUCCEEDED. LLVM compiled 4343 +# targets, failed on a systemd-run line at the very end, and the next run +# wiped the tree and compiled it all again. +_cb10="$(_slice_fn "$helper_src" cmd_build)" +printf '%s' "$_cb10" | grep -q 'keeping the source tree' \ + || _p="$_p;a retry of --phase all recompiles a build that already finished" +printf '%s' "$_cb10" | grep -q '_done_phases' \ + || _p="$_p;the retry does not consult the phase records before cleaning" +# ...and a package with no finished build is still cleaned +_phd="$T/phases"; mkdir -p "$_phd" +printf 'unpack\nbuild\n' > "$_phd/done"; printf 'unpack\n' > "$_phd/partial" +for _pk in done partial; do + _out="$({ echo 'say(){ echo KEPT; }'; echo 'clean_stale_source(){ echo CLEANED; }' + printf 'phase_file(){ echo "%s/$1"; }\n' "$_phd" + echo "name=$_pk; phase=all" + sed -n '/DO NOT THROW AWAY A BUILD/,/^ esac$/p' "$helper_src" \ + | sed 's/^ //' | sed 's/^local /_x=/' + } | bash 2>/dev/null | head -1)" + case "$_pk:$_out" in + done:KEPT|partial:CLEANED) ;; + *) _p="$_p;retry cleaning is wrong for a $_pk build ($_out)" ;; + esac +done +_slice_fn "$helper_src" cmd_build | grep -q 'create_missing_dirs_from_log' \ + || _p="$_p;the retry loop never creates missing directories" +# CREATING A DIRECTORY IS A CHANGE TO THE SYSTEM, and it needs the user's +# word. --yes is that word given in advance; no terminal means NO, because +# silence is not consent. +_slice_fn "$helper_src" create_missing_dirs_from_log | grep -q 'confirm_change' \ + || _p="$_p;directories are created in someone else's tree without asking" +# WHO OWNS A DIRECTORY WE CREATE. root:install is the rule for the system's +# own install directories. Inside ANOTHER PACKAGE's tree it is wrong twice: +# chmod needs ownership (cmake: "cannot set permissions"), and it invents a +# second rule where the collector-group model already answers. +_slice_fn "$helper_src" create_missing_dirs_from_log | grep -q 'share_new_dir_with_tree_owner' \ + || _p="$_p;a directory made inside another package's tree does not go through collector groups" +# THE GROUP BELONGS TO THE TREE, not to whoever installed first: vim's tree +# came out nimgnu_cmake. And naming a NEW group is a decision, not a +# yes/no -- --yes means "do not ask me to confirm", not "pick a name for me". +if [ "$(id -u)" = 0 ] && command -v groupadd >/dev/null 2>&1; then + _gtd="$T/treegroup"; mkdir -p "$_gtd/n1" "$_gtd/n2" + groupadd nimgnu_treet >/dev/null 2>&1 + _grun() { { echo 'COLLECTOR_PREFIX=nimgnu; PKGUSR_PREFIX=p_; CFGUSR_PREFIX=cfg_' + echo 'ok(){ :; }'; echo 'warn(){ :; }'; echo 'say(){ :; }' + echo 'soft(){ shift 2; "$@"; }' + echo 'real_chgrp(){ chgrp "$@"; }'; echo 'real_chmod(){ chmod "$@"; }' + echo 'create_collector_group(){ groupadd "$1" 2>/dev/null; }' + echo 'add_to_group(){ :; }' + echo 'group_exists(){ getent group "$1" >/dev/null 2>&1; }' + _slice_fn "$helper_src" sanitise_group_name + _slice_fn "$helper_src" collector_group_for + _slice_fn "$helper_src" unprefix_pkg_user + _slice_fn "$helper_src" pkgusr_kind + _slice_fn "$helper_src" share_new_dir_with_tree_owner + echo "share_new_dir_with_tree_owner $1 $2 p_makert"; } > "$_gtd/run.sh" + setsid bash "$_gtd/run.sh" < /dev/null >/dev/null 2>&1; } + # THE PARENT DIRECTORY DECIDES: a subdirectory of a directory that + # already carries a collector group inherits it, with no new question. + chgrp nimgnu_treet "$_gtd" 2>/dev/null + _grun "$_gtd/n1" p_treet + [ "$(stat -c %G "$_gtd/n1" 2>/dev/null)" = nimgnu_treet ] \ + || _p="$_p;a subdirectory does not inherit its parent's collector group" + chgrp root "$_gtd" 2>/dev/null + _grun "$_gtd/n2" p_fresht + case "$(stat -c %G "$_gtd/n2" 2>/dev/null)" in + nimgnu_fresht) ;; + nimgnu_makert) _p="$_p;the new group is named after the package that created the directory, not the tree" ;; + *) _p="$_p;no collector group was put on a directory made in another package's tree" ;; + esac + groupdel nimgnu_treet >/dev/null 2>&1; groupdel nimgnu_fresht >/dev/null 2>&1 + rm -rf "$_gtd" +fi +_slice_fn "$helper_src" share_new_dir_with_tree_owner | grep -q '/dev/tty' \ + || _p="$_p;naming a NEW collector group is never asked about" +# STDOUT IS THE RETURN VALUE. choose_collector_group ANSWERS on stdout, and +# 1.13.46 added explanation lines with plain `say` -- so the caller captured +# the paragraph as the group name and `chgrp` failed on it, silently. +_ccg="$(_slice_fn "$helper_src" choose_collector_group)" +printf '%s' "$_ccg" | grep -E '^\s+say ' | grep -v '>&2' | grep -v '^\s*#' \ + | grep -q . && { + # the prompt body is legitimate: it lives inside a `{ ... } > /dev/tty` + printf '%s' "$_ccg" | grep -q '} > /dev/tty' \ + || _p="$_p;informational output in choose_collector_group goes to stdout" +} +_cap="$({ echo 'COLLECTOR_PREFIX=nimgnu' + echo 'say(){ echo "$*"; }'; echo 'warn(){ echo "W:$*" >&2; }' + echo 'sanitise_group_name(){ printf "%s" "$1"; }' + echo 'collector_group_for(){ printf "nimgnu_%s" "${1#p_}"; }' + echo 'group_exists(){ false; }'; echo 'user_in_group(){ false; }' + echo 'stat(){ case "$3" in */gstreamer-1.0) echo nimgnu_gst ;; *) echo root ;; esac; }' + printf '%s\n' "$_ccg" + echo 'choose_collector_group /usr/share/gstreamer-1.0/presets p_good p_bad' + } | bash 2>/dev/null)" +case "$_cap" in + nimgnu_gst) ;; + *) _p="$_p;the captured group name is not a name: [$_cap]" ;; +esac + +# A NAME THAT CANNOT BE ASKED FOR MUST STILL BE CHOSEN. During a build stdin +# is not a terminal, so choose_collector_group's prompt cannot run and it +# returned NOTHING -- then `chgrp "" dir` failed and grant_dir_access returned +# 1 with no message, reported as "nothing to grant". Five rounds of +# debugging on gst-plugins-bad were chasing that silence. +_slice_fn "$helper_src" grant_dir_access | grep -q 'NEVER LEAVE IT EMPTY, AND NEVER FAIL QUIETLY' \ + || _p="$_p;an unaskable group name leaves the grant silently broken" +_gfb="$({ echo 'COLLECTOR_PREFIX=nimgnu; owner=p_other; dir=/tmp/x' + echo 'warn(){ :; }' + echo 'sanitise_group_name(){ printf "%s" "$1"; }' + echo 'collector_group_for(){ printf "nimgnu_%s" "$1"; }' + echo 'choose_collector_group(){ printf ""; }' + echo 'grp="$(choose_collector_group x y z)"' + _slice_fn "$helper_src" grant_dir_access \ + | sed -n '/# NEVER LEAVE IT EMPTY, AND NEVER FAIL QUIETLY/,/^ fi$/p' \ + | sed 's/^ //' + echo 'printf "%s" "$grp"'; } | bash 2>/dev/null)" +[ "$_gfb" = "nimgnu_other" ] \ + || _p="$_p;the fallback group name is wrong ($_gfb)" +_slice_fn "$helper_src" grant_dir_access | grep -q 'chgrp $grp $dir FAILED' \ + || _p="$_p;a failed chgrp is silent" + +# AN ANCESTOR'S GROUP ALREADY ANSWERED. /usr/lib/python3.14/site-packages is +# nimgnu_python, and a package needing .../site-packages/gi/overrides was +# still asked to name a group for it. +_slice_fn "$helper_src" choose_collector_group | grep -q "ANCESTOR'S GROUP ALREADY ANSWERED" \ + || _p="$_p;a directory inside a shared tree still asks for a group name" +_ancs="$T/anc.sh" +python3 - "$helper_src" > "$_ancs" <<'PYANC' +import sys +s = open(sys.argv[1]).read() +i = s.index('choose_collector_group() {') +d, j = 0, i +while True: + if s[j] == '{': d += 1 + elif s[j] == '}': + d -= 1 + if d == 0: break + j += 1 +print('COLLECTOR_PREFIX=nimgnu') +print('say(){ echo "$*"; }; warn(){ :; }') +print('sanitise_group_name(){ printf "%s" "$1"; }') +print('collector_group_for(){ printf "nimgnu_%s" "${1#p_}"; }') +print('group_exists(){ false; }; user_in_group(){ false; }') +print('stat(){ case "$3" in */site-packages) echo nimgnu_python ;; *) echo root ;; esac; }') +print(s[i:j+1]) +print('choose_collector_group /usr/lib/python3.14/site-packages/gi/overrides ' + 'p_pygobject3 p_at-spi2-core') +PYANC +[ "$(bash "$_ancs" 2>/dev/null | tail -1)" = "nimgnu_python" ] \ + || _p="$_p;the ancestor's collector group is not inherited" + +# --yes ANSWERS A QUESTION; IT DOES NOT NAME THINGS. With --yes the grant +# path picked the owner-named group and created it silently -- a name that +# then sits on the system for good. An EXISTING group is a yes/no and --yes +# may take it; a NEW one is still asked about when there is a terminal. +grep -q 'does not name a NEW collector group' "$helper_src" \ + || _p="$_p;--yes silently invents collector group names" +_gp="$(_slice_fn "$helper_src" _collector_group_for_dir 2>/dev/null || true)" +[ -n "$_gp" ] && { printf '%s' "$_gp" | grep -q 'assume_yes && group_exists' \ + || _p="$_p;--yes does not distinguish reusing a group from creating one"; } +# A PACKAGE'S OWN TREE NEEDS NO GROUP. p_rust created a directory inside +# p_rust's tree and was asked which group should own it -- offering the same +# name twice, because "the tree" and "the creator" were one package. +_slice_fn "$helper_src" share_new_dir_with_tree_owner | grep -q 'which created it -- no group needed' \ + || _p="$_p;a package is asked to share a directory in its own tree" +# and the group is named after the PACKAGE, not the account: with an empty +# PKGUSR_PREFIX the name came out nimgnu_p_rust +_slice_fn "$helper_src" share_new_dir_with_tree_owner | grep -q 'powner#p_' \ + || _p="$_p;the collector group keeps the account prefix (nimgnu_p_rust)" +# the three things the answer depends on are highlighted +for _hl in 'package :' 'directory:' 'recommended'; do + _slice_fn "$helper_src" share_new_dir_with_tree_owner | grep -q "$_hl" \ + || _p="$_p;the prompt does not highlight '$_hl'" +done +_slice_fn "$helper_src" create_missing_dirs_from_log | grep -q 'is_install_dir' \ + || _p="$_p;the system's install directories and another package's tree get the same rule" +# A DIRECTORY WE CREATED IS NOT EVIDENCE ABOUT OWNERSHIP. mkdir -p leaves +# intermediates root:root, and the next round read that artifact as "root's +# tree" and applied the system rule inside a package's tree. +if [ "$(id -u)" = 0 ] && command -v useradd >/dev/null 2>&1; then + _vvd="$T/dirrule"; mkdir -p "$_vvd/usr/share/vim" "$_vvd/usr/lib" "$_vvd/src" + for _u in p_vimQ p_cmQ; do id "$_u" >/dev/null 2>&1 || useradd -M -U "$_u" >/dev/null 2>&1; done + chown p_vimQ:p_vimQ "$_vvd/usr/share/vim" 2>/dev/null + _mkrun() { + printf 'cannot make directory "%s": x\n' "$1" > "$_vvd/log" + { echo "SNAP_ROOT=$_vvd; SRCROOT=$_vvd/src; PM_YES=1" + echo 'ok(){ :; }'; echo 'warn(){ :; }'; echo 'say(){ :; }' + echo 'soft(){ shift 2; "$@"; }' + echo 'real_chmod(){ chmod "$@"; }'; echo 'real_chown(){ chown "$@"; }' + echo 'set_install_dir_owner(){ chown root:install "$1"; }' + printf 'is_install_dir(){ case "$1" in %s/usr/lib) return 0;; esac; return 1; }\n' "$_vvd" + echo 'grant_dir_access(){ :; }' + _slice_fn "$helper_src" assume_yes + _slice_fn "$helper_src" confirm_change + _slice_fn "$helper_src" under_dir + _slice_fn "$helper_src" missing_dirs_from_log + _slice_fn "$helper_src" create_missing_dirs_from_log + echo "create_missing_dirs_from_log $_vvd/log p_cmQ"; } > "$_vvd/run.sh" + bash "$_vvd/run.sh" < /dev/null >/dev/null 2>&1 + } + _mkrun "$_vvd/usr/share/vim/vimfiles/indent" + _mkrun "$_vvd/usr/share/vim/vimfiles/syntax" # the second round + _mkrun "$_vvd/usr/lib/sysdir" + [ "$(stat -c %U "$_vvd/usr/share/vim/vimfiles" 2>/dev/null)" = p_cmQ ] \ + || _p="$_p;an intermediate directory is left root-owned and unusable" + [ "$(stat -c %U "$_vvd/usr/share/vim/vimfiles/syntax" 2>/dev/null)" = p_cmQ ] \ + || _p="$_p;the second round mistook our own intermediate for the system's tree" + [ "$(stat -c %G "$_vvd/usr/lib/sysdir" 2>/dev/null)" = install ] \ + || _p="$_p;a real install directory no longer gets the system rule" + userdel p_vimQ >/dev/null 2>&1; userdel p_cmQ >/dev/null 2>&1 + rm -rf "$_vvd" +fi +grep -q '^confirm_change()' "$helper_src" \ + || _p="$_p;there is no way to ask before changing the system" +# A PROMPT BEHIND A PIPE LOOKS LIKE A HANG. The question goes to /dev/tty so +# it survives the pipes, but the build's output goes through tee -- so a run +# waiting for an answer showed nothing at all. +[ "$(grep -c 'WAITING FOR AN ANSWER' "$helper_src")" -ge 3 ] \ + || _p="$_p;a prompt can wait without saying so in the output" +_slice_fn "$helper_src" cmd_build | grep -q 'missing_dirs_from_log "\$log"' \ + || _p="$_p;a missing-directory failure never reaches the retry loop" +# the GATE decides whether the loop runs at all, and cmake's wording is +# "cannot MAKE directory" -- which the regex did not list, so cmake failed +# twice with no attempt and no explanation +_slice_fn "$helper_src" cmd_build | grep -q 'cannot (create|make|remove' \ + || _p="$_p;the retry gate does not recognise \"cannot make directory\"" +_slice_fn "$helper_src" cmd_build | grep -q 'no automatic repair attempted' \ + || _p="$_p;when no repair is attempted, nothing says why" +# AND IT MUST WORK WHEN THE ROOT IS "/" -- which is every real chroot, and +# no sandbox. "${SNAP_ROOT%/}" is EMPTY there, so `case "$d" in +# "${SNAP_ROOT%/}"/*)` matched nothing and `"$SRCROOT"/*` with an unset +# SRCROOT excluded everything: cmake's missing directory was invisible on +# the only system that matters. +printf 'file INSTALL cannot make directory "/usr/share/vim/vimfiles/indent": No such file or directory.\n' > "$T/rootslash.log" +_rs1="$({ echo 'SNAP_ROOT=/; SRCROOT=/usr/src' + _slice_fn "$helper_src" missing_dirs_from_log + echo 'missing_dirs_from_log '"$T"'/rootslash.log'; } | bash 2>/dev/null)" +printf '%s\n' "$_rs1" | grep -q '/usr/share/vim/vimfiles/indent' \ + || _p="$_p;with the root at / (every real chroot) a missing directory is invisible" +_rs2="$({ echo 'SNAP_ROOT=/' + _slice_fn "$helper_src" missing_dirs_from_log + echo 'missing_dirs_from_log '"$T"'/rootslash.log'; } | bash 2>/dev/null)" +printf '%s\n' "$_rs2" | grep -q '/usr/share/vim/vimfiles/indent' \ + || _p="$_p;an unset SRCROOT turns the build-tree exclusion into a wildcard that excludes everything" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a shared directory that does not exist yet is created, whatever the tool called it" +fi +rm -rf "$_mdd" + +# ---- the attribution window opens on a quiet tree ---------------------------- # +# Timestamp attribution assumes nothing else writes during a build. A real +# run disproved it: make-ca was still generating /etc/ssl/certs while libogg +# built, and libogg's manifest claimed 400 certificates. cmd_build now waits +# for the known background writers before dropping its stamp. +_p="" +_slice_fn "$helper_src" cmd_build | grep -q '_settle_tree_before_stamp' \ + || _p="$_p;the attribution window opens without waiting for background writers" +grep -q '^_SETTLE_PROCS=' "$helper_src" \ + || _p="$_p;there is no list of known background writers" +for _w in mandb make-ca fc-cache; do + grep -A4 '^_SETTLE_PROCS=' "$helper_src" | grep -q "$_w" \ + || _p="$_p;$_w is not waited for (it writes during other packages' builds)" +done +# it returns at once on a quiet tree, and gives up (never hangs) on a busy one +_stq="$({ echo 'say(){ :; }'; echo 'warn(){ :; }'; echo '_SETTLE_PROCS="nosuchproc-xyz"' + _slice_fn "$helper_src" _settle_tree_before_stamp + echo 'time _settle_tree_before_stamp'; } | bash 2>&1 | grep real)" +case "$_stq" in *0m0.*|*0m1.*) ;; *) _p="$_p;settling is slow on a quiet tree" ;; esac +_stb="$({ echo 'say(){ echo S; }'; echo 'warn(){ echo GAVEUP; }'; echo '_SETTLE_PROCS="sleep"' + _slice_fn "$helper_src" _settle_tree_before_stamp + echo 'LFS_SETTLE_SECONDS=2 _settle_tree_before_stamp'; } > "$T/st.sh" + sleep 20 & _sp=$! + bash "$T/st.sh" 2>&1; kill $_sp 2>/dev/null)" +echo "$_stb" | grep -q GAVEUP \ + || _p="$_p;a busy tree hangs the build instead of warning and continuing" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a build's file list starts on a quiet tree, and never hangs waiting for one" +fi + +# ---- a partly sealed tree is sealed, not unsealed ---------------------------- # +# The seal probe demanded ALL five probed dirs be sticky, so a mixed tree read +# as "not sealed" and verify proposed stripping the sticky bit off ~50 +# correctly sealed directories (1775 -> 775). Sealing is one-way: verify +# completes it, never undoes it. +_sd="$T/sealdirs" +_seal_probe() { # $1 = how many of three dirs are sticky + rm -rf "$_sd"; mkdir -p "$_sd/a" "$_sd/b" "$_sd/c" + local n=0 d + for d in a b c; do + [ "$n" -lt "$1" ] && chmod 1775 "$_sd/$d" + n=$((n+1)) + done + { echo "SNAP_ROOT=$_sd"; echo 'warn(){ :; }' + printf 'install_dirs_list(){ printf "/a\\n/b\\n/c\\n"; }\n' + _slice_fn "$helper_src" _install_dirs_are_sealed + echo '_install_dirs_are_sealed && echo SEALED || echo UNSEALED' + } | bash 2>/dev/null | tail -1 +} +_p="" +[ "$(_seal_probe 0)" = UNSEALED ] \ + || _p="$_p;a mid-build tree reads as sealed -- the sticky bit would block package users" +[ "$(_seal_probe 3)" = SEALED ] || _p="$_p;a fully sealed tree reads as unsealed" +[ "$(_seal_probe 2)" = SEALED ] \ + || _p="$_p;a PARTLY sealed tree reads as unsealed -- verify would strip sticky bits it should complete" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "sealing is one-way: a partly sealed tree is completed, never undone" +fi +rm -rf "$_sd" + +# ---- a failed build makes no install claim ---------------------------------- # +# A failed libxml2 run recorded its 4 half-built files as pkg.lst; the +# planner read pkg.lst as "installed" and skipped the package forever -- +# docbook then died on the xmlcatalog that was never installed. The +# manifest (what a step TOUCHED) survives failure; the recorder must not. +_p="" +_slice_fn "$helper_src" cmd_build | grep -q '= 0 \] && { cmd_record_install' \ + || _p="$_p;a failed build still records itself as installed" +_slice_fn "$helper_src" cmd_build | grep -q "Program '\[A-Za-z0-9._+-\]+' not found" \ + || _p="$_p;a missing-program failure does not point at the lying record" +# the same species from the LINKER's mouth: "cannot find -lfoo" +_slice_fn "$helper_src" cmd_build | grep -q 'cannot find -l' \ + || _p="$_p;a missing-library link failure does not point at the lying record" +# and it must name only the libraries that are REALLY absent. `ls a* b* c*` +# fails when any one pattern misses, so a library present in the first +# directory was reported missing because the others lacked it. +_lkd="$T/linklibs"; mkdir -p "$_lkd/usr/lib" +printf 'ld: cannot find -lonlymissing: x\nld: cannot find -lhalf: x\nld: cannot find -lpresent: x\n' > "$_lkd/log" +: > "$_lkd/usr/lib/libpresent.so.1" +ln -s libpresent.so.1 "$_lkd/usr/lib/libpresent.so" # complete install +: > "$_lkd/usr/lib/libhalf.so.6.20.1" # real lib, NO dev symlink +_lkout="$({ echo "SNAP_ROOT=$_lkd; log=$_lkd/log"; echo 'fail(){ echo "$*"; }' + echo 'unprefix_pkg_user(){ echo "${1#p_}"; }' + sed -n '/# "cannot find -lfoo" is the linker/,/^ fi$/p' "$helper_src" | sed 's/^ //' + } | bash 2>/dev/null)" +echo "$_lkout" | grep -q 'libonlymissing.so' \ + || _p="$_p;a genuinely missing library is not named" +echo "$_lkout" | grep -q 'libpresent' \ + && _p="$_p;a COMPLETE library is reported missing (one failed glob poisons the check)" +# libfoo.so.N without libfoo.so is what -lfoo actually fails on, and it is a +# DIFFERENT illness: the owner's install never finished, its record is honest +echo "$_lkout" | grep -q 'libhalf.so.6.20.1 exists' \ + || _p="$_p;libfoo.so.N without libfoo.so is treated as present (that IS the failure case)" +echo "$_lkout" | grep -q 'development symlink' \ + || _p="$_p;a missing development symlink is diagnosed as a lying record instead" +# and it must name the package that OWNS the library, not whoever ran +# ldconfig: the SONAME link libfoo.so.1 carries the account that made it, +# which blamed p_libpaper for fontconfig +mkdir -p "$_lkd/man" +: > "$_lkd/usr/lib/libowned.so.1.13.0" +ln -s libowned.so.1.13.0 "$_lkd/usr/lib/libowned.so.1" +printf '%s/usr/lib/libowned.so.1.13.0\n' "$_lkd" > "$_lkd/man/theowner.files" +printf 'ld: cannot find -lowned: x\n' > "$_lkd/log2" +_lkout2="$({ echo "SNAP_ROOT=$_lkd; log=$_lkd/log2; MANIFESTS=$_lkd/man" + echo 'fail(){ echo "$*"; }'; echo 'unprefix_pkg_user(){ echo "${1#p_}"; }' + sed -n '/# "cannot find -lfoo" is the linker/,/^ fi$/p' "$helper_src" | sed 's/^ //' + } | bash 2>/dev/null)" +echo "$_lkout2" | grep -q 'build theowner' \ + || _p="$_p;the rebuild hint names the wrong package (it follows the SONAME link, not the manifest)" +rm -rf "$_lkd" +# a compile error is not the tools' to fix, but the box must name where a +# per-package build flag survives a regenerated script +_slice_fn "$helper_src" cmd_build | grep -q 'machine.conf' \ + || _p="$_p;a compile error does not point at machine.conf for a per-package flag" +# a build that stops for a missing variable must print the command that sets +# it -- "needs XORG_CONFIG to be set" says what, not how +_slice_fn "$helper_src" cmd_build | grep -q 'packagemanager env --set' \ + || _p="$_p;a missing build variable is reported without the command that sets it" +# a failure whose script predates the generator must say so IN THE BOX: the +# resolve-time warning is far out of sight by the time a package fails +# THE RETRY HINT MUST BE A COMMAND THAT WORKS. `lfs-helper build` takes the +# ACCOUNT; the box printed the anchor for a BLFS package, so the suggested +# command fell through to the LFS base-step path and blamed step scripts from +# an old version -- sending the debugging in the wrong direction entirely. +_slice_fn "$helper_src" cmd_build | grep -q 'THE RETRY HINT MUST BE A COMMAND THAT WORKS' \ + || _p="$_p;the failure box suggests a name lfs-helper cannot resolve" +[ "$(_slice_fn "$helper_src" cmd_build | grep -c '\$_retry --phase')" -ge 3 ] \ + || _p="$_p;not every retry hint uses the resolved account name" + +# A FEATURE MISSING FROM AN INSTALLED PACKAGE. pango died on "No Cairo font +# backends found" with cairo installed: cairo had been built before freetype2 +# existed and silently has no ft backend. +_slice_fn "$helper_src" cmd_build | grep -q "is missing from it -- it was probably built" \ + || _p="$_p;a missing feature in an installed package is not explained" +_fl="$T/feature.log" +printf 'Run-time dependency cairo found: YES 1.18.4\n' > "$_fl" +printf 'Run-time dependency cairo-ft found: NO (tried pkgconfig)\n' >> "$_fl" +_fp="$(grep -oE "dependency [a-z0-9]+-[a-z0-9]+ found: NO" "$_fl" | head -n1 | awk '{print $2}' | cut -d- -f1)" +[ "$_fp" = cairo ] || _p="$_p;the feature-provider is not identified (got: $_fp)" + +_slice_fn "$helper_src" cmd_build | grep -q 'the generator is now' \ + || _p="$_p;a failure on a stale script does not mention regenerating it" +# and it must suggest the ANCHOR (mesa), not the account (p_mesa) -- and say +# so when the prefix it read disagrees with the account it was handed +_slice_fn "$helper_src" cmd_build | grep -q 'read an EMPTY package-user prefix' \ + || _p="$_p;a prefix disagreement is papered over instead of reported" +# ...but an EMPTY EXPORT is not a disagreement: a chroot script exports +# LFS_PKGUSR_PREFIX="" when the value was unknown, and `-` let that beat a +# config that says "p" -- so the tool reported files as disagreeing when they +# agreed. The config may mean empty; the environment may not. +grep -q 'THE CONFIG IS CURRENT; THE ENVIRONMENT MAY BE OLD' "$helper_src" \ + || _p="$_p;an empty exported prefix overrides the config file" +for _pfx in "p_" ""; do + _sg="$({ printf "PKGUSR_PREFIX='%s'\n" "$_pfx" + echo 'name=p_mesa'; echo 'fail(){ echo "$*"; }' + sed -n '/^unprefix_pkg_user() {/,/^}/p' "$helper_src" + _slice_fn "$helper_src" cmd_build \ + | sed -n '/# The suggestion needs the ANCHOR/,/esac/p' \ + | sed 's/^ //;s/^local /_x=/' + } | bash 2>/dev/null | head -1)" + case "$_sg" in + *"install mesa "*) ;; + *) _p="$_p;with prefix [$_pfx] the box suggests: $_sg" ;; + esac +done +# (the "install mesa" behaviour is checked above for both prefixes; the +# literal ${name#p_} that used to be here was replaced in 1.13.25 by +# reporting the config disagreement instead of hiding it) +# and verify sweeps for ALL lying records in one pass +_lr="$T/lyingrec"; mkdir -p "$_lr/p_liar" "$_lr/p_honest" "$_lr/real" +printf '%s/real/g1\n%s/real/g2\n%s/real/g3\n%s/real/here\n' "$_lr" "$_lr" "$_lr" "$_lr" > "$_lr/p_liar/pkg.lst" +touch "$_lr/real/here" "$_lr/real/a" +printf '%s/real/a\n%s/real/here\n' "$_lr" "$_lr" > "$_lr/p_honest/pkg.lst" +_lrout="$({ echo "PKGUSR_ROOT=$_lr" + echo 'warn(){ echo "W: $*"; }' + echo 'unprefix_pkg_user(){ echo "${1#p_}"; }' + _slice_fn "$helper_src" _vfy_lying_records + echo '_vfy_lying_records' + } | bash 2>/dev/null)" +echo "$_lrout" | grep -q "lying record: p_liar" \ + || _p="$_p;verify does not sweep for lying records" +echo "$_lrout" | grep -q "p_honest" \ + && _p="$_p;an honest record is accused" +_slice_fn "$helper_src" cmd_verify | grep -q '_vfy_lying_records' \ + || _p="$_p;cmd_verify never runs the lying-record sweep" +# THE ORDINARY BUILD PATH MUST NOT NEED THE SWEEP. The planner decided +# "installed" from ONE surviving path, so a record with 190 of 330 files gone +# still made it skip the package -- and only a manual `verify` would have +# said otherwise. Same rule in both places now. +python3 - "$pm_src" <<'PYMAN' || _p="$_p;the planner believes a record whose files are mostly gone" +import sys, os, tempfile, random, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +C = pm.User +d = tempfile.mkdtemp(); real = os.path.join(d, "real"); os.makedirs(real) +def mk(name, alive, dead, shuffle=True): + lines = [] + for i in range(alive): + p = os.path.join(real, "%s%d" % (name, i)); open(p, "w").close(); lines.append(p) + for i in range(dead): + lines.append(os.path.join(real, "%sgone%d" % (name, i))) + if shuffle: random.shuffle(lines) + lst = os.path.join(d, name + ".lst"); open(lst, "w").write("\n".join(lines) + "\n") + u = C.__new__(C); u.pkg_list = lst; return u +assert mk("liar", 140, 190)._manifest_has_real_paths() is False +# pkg.lst is path-sorted, so sampling the top would call this one healthy +assert mk("sorted", 140, 190, shuffle=False)._manifest_has_real_paths() is False +assert mk("good", 300, 30)._manifest_has_real_paths() is True +PYMAN +# --fix must DELETE the false claim: while it stands the planner skips the +# package and every dependent fails somewhere else entirely. Rebuilding +# stays the user's decision; removing a lie is not. +_lrf="$T/lyingfix"; mkdir -p "$_lrf/p_liar" "$_lrf/p_ok" "$_lrf/real" +printf '%s/real/g1\n%s/real/g2\n%s/real/g3\n' "$_lrf" "$_lrf" "$_lrf" > "$_lrf/p_liar/pkg.lst" +: > "$_lrf/p_liar/VERSION"; touch "$_lrf/real/a" +printf '%s/real/a\n' "$_lrf" > "$_lrf/p_ok/pkg.lst" +{ echo "PKGUSR_ROOT=$_lrf"; echo 'warn(){ :; }'; echo 'ok(){ :; }' + echo 'unprefix_pkg_user(){ echo "${1#p_}"; }'; echo '_vfy_n_fixed=0' + _slice_fn "$helper_src" _vfy_lying_records; echo '_vfy_lying_records 0'; } | bash >/dev/null 2>&1 +[ -f "$_lrf/p_liar/pkg.lst" ] || _p="$_p;a bare verify deleted a record (it must only report)" +{ echo "PKGUSR_ROOT=$_lrf"; echo 'warn(){ :; }'; echo 'ok(){ :; }' + echo 'unprefix_pkg_user(){ echo "${1#p_}"; }'; echo '_vfy_n_fixed=0' + _slice_fn "$helper_src" _vfy_lying_records; echo '_vfy_lying_records 1'; } | bash >/dev/null 2>&1 +[ -f "$_lrf/p_liar/pkg.lst" ] && _p="$_p;verify --fix leaves the false record in place" +[ -f "$_lrf/p_liar/VERSION" ] && _p="$_p;verify --fix leaves the false VERSION in place" +[ -f "$_lrf/p_ok/pkg.lst" ] || _p="$_p;verify --fix deleted an HONEST record" +rm -rf "$_lrf" +python3 - "$pm_src" <<'PYCASE' || _p="$_p;python does not lowercase names like the helper (p_DocBook vs p_docbook)" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm.pkgusr_name('DocBook') == 'p_docbook' +assert pm.pkgusr_name('p_docbook') == 'p_docbook' +PYCASE +# --status answers "where are we", short by default and complete on request: +# entry counts alone said "1 of 39" while hundreds of packages were in flight. +# (checked in the source: running it needs a book, which a bare checkout +# does not have -- see the four book-gated failures) +# THE BUILD RUNS AS A PACKAGE USER through `su -`, which drops the caller's +# environment -- so exporting TEXLIVE_PREFIX in a shell never reached it. +# One file, written by `packagemanager env`, passed to every phase. +grep -q 'def cmd_env' "$pm_src" || _p="$_p;there is no way to set the book's build variables" +# scripts from an OLDER generator ask older questions: after the guard was +# narrowed, the stale files in /tmp still listed EUID and DOCNAME, so the +# fix looked like it had done nothing +grep -q 'stale.add' "$pm_src" \ + || _p="$_p;env reads scripts from an older blfs and repeats their questions" +# ...and an empty list must not read as "nothing to do": a variable only +# becomes visible once its package's script is generated, during the build +grep -q 'becomes visible when its package' "$pm_src" \ + || _p="$_p;an empty env list is presented as if nothing will ever be needed" +grep -q '"--refresh"' "$pm_src" \ + || _p="$_p;stale scripts cannot be cleared to reveal the current questions" +grep -q 'XORG_CONFIG' "$pm_src" \ + || _p="$_p;--ask offers no value for XORG_CONFIG, which the generator cannot lift safely" +# a value that spans continuation lines must never be emitted half-captured: +# "XORG_CONFIG=\"--prefix=$XORG_PREFIX --sysconfdir=/etc \\" is broken bash +python3 - "$_bt4" <<'PYCHAIN' || _p="$_p;a multi-line book value is emitted as a fragment" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +class P: + name = "pre" + def __init__(s, t): s.t = t + def get_text(s): return s.t + def find_all(s, *a, **k): return [s] +class Root(P): + parent = None +sec = Root('export A_PREFIX="/usr"\nexport A_CONFIG="--prefix=$A_PREFIX \\\n --more"\n') +got = b._page_env_definitions(sec, {"A_CONFIG"}) +for v in got.values(): + assert not v.endswith("\\"), got +PYCHAIN +grep -q 'BUILD_ENV_FILE' "$helper_src" \ + || _p="$_p;the build never reads the variables that were set for it" +_slice_fn "$helper_src" run_phase_as | grep -q 'build_env_pairs' \ + || _p="$_p;a phase runs without the build environment" +_bev="$T/build.env" +printf 'A=1\n# c\nB=two words\n' > "$_bev" +_bout="$({ echo "BUILD_ENV_FILE=$_bev"; _slice_fn "$helper_src" build_env_pairs + echo 'build_env_pairs'; } | bash 2>/dev/null)" +# quoted, because these pairs are pasted into a command line +case "$_bout" in *"A='1'"*"B='two words'"*) ;; + *) _p="$_p;the build environment file is not read correctly: $_bout" ;; esac +# A VALUE WITH SPACES IS ONE VALUE. These pairs are pasted into +# `env $envpass bash ...`; unquoted, XORG_CONFIG's flag list split and the +# second word was run as a command: +# env: '--disable-static': No such file or directory +printf "XC=--prefix=/usr --sysconfdir=/etc --disable-static\nAP=it's here\n" > "$_bev" +_bq="$({ echo "BUILD_ENV_FILE=$_bev"; _slice_fn "$helper_src" build_env_pairs + echo 'p="$(build_env_pairs)"' + echo 'eval "env $p bash -c '"'"'printf \"[%s][%s]\" \"\$XC\" \"\$AP\"'"'"'"' + } | bash 2>/dev/null)" +[ "$_bq" = "[--prefix=/usr --sysconfdir=/etc --disable-static][it's here]" ] \ + || _p="$_p;a build variable containing spaces or quotes does not survive: $_bq" +grep -q '"--status"' "$pm_src" || _p="$_p;there is no --status" +grep -q '"--full"' "$pm_src" || _p="$_p;--status has no full listing" +# A GIT ENTRY'S "AM I INSTALLED?" CHECK MUST BE REAL. `prog=false` looked +# like "no program to check" and `false` IS a program, so four entries +# reported themselves installed before they had ever been built. +if [ -f "$_stk" ]; then + grep -q 'prog=false' "$_stk" \ + && _p="$_p;a stack entry checks for the program 'false' and always passes" + # the WebRTC chain must precede the packages that link it + _ln=$(grep -n '^git .*libnice' "$_stk" | head -1 | cut -d: -f1) + _gb=$(grep -n '^book *gst10-plugins-bad' "$_stk" | head -1 | cut -d: -f1) + _wk=$(grep -n '^book *webkitgtk' "$_stk" | head -1 | cut -d: -f1) + if [ -n "$_ln" ] && [ -n "$_gb" ] && [ "$_ln" -gt "$_gb" ]; then + _p="$_p;libnice is built after gst-plugins-bad, so webrtcbin will be missing" + fi + if [ -n "$_ln" ] && [ -n "$_wk" ] && [ "$_ln" -gt "$_wk" ]; then + _p="$_p;libnice is built after webkitgtk" + fi +fi + +# GIT ENTRIES NEED GIT, which is itself a BLFS package: a clone failed with +# "git: command not found". They are grouped at the end of the stack, after +# `book git`, and --no-git skips them entirely. +grep -q '"--no-git"' "$pm_src" \ + || _p="$_p;the cloned entries cannot be skipped" +# a BOOK entry can become a clone (elogind: `book ... url=...`), and its kind +# is only final after the judging -- so --no-git has to decide there too +grep -q 'skipped (--no-git)' "$pm_src" \ + || _p="$_p;a book entry that falls back to git is still cloned under --no-git" +# ...and some cloned entries are DEPENDENCIES: gst-plugins-bad compiles +# webrtcbin only if libnice is there, so --no-git must say what it costs +grep -q 'SOME CLONED ENTRIES ARE DEPENDENCIES, NOT EXTRAS' "$pm_src" \ + || _p="$_p;--no-git silently drops packages that book entries link" +if [ -f "$_stk" ]; then + grep -qE '^book +git' "$_stk" \ + || _p="$_p;the stack clones packages without building git first" + # The invariant is that the EXTRAS BLOCK is last, not that no git entry + # may precede a book one: 1.13.67 added libnice and friends as git + # entries BEFORE gst-plugins-bad, because webrtcbin is compiled only if + # libnice is there. Those are dependencies, not extras. What must hold + # is that nothing after the extras marker is a book entry. + _marker=$(grep -n 'Everything from here needs GIT' "$_stk" | head -1 | cut -d: -f1) + if [ -n "$_marker" ]; then + # `book git` belongs there: the clones cannot run before git exists. + awk -v m="$_marker" 'NR > m && /^book / && $2 != "git"' "$_stk" \ + | grep -q . \ + && _p="$_p;a book entry other than git sits after the extras marker" + fi +fi +grep -q 'packages: %d of %d installed' "$pm_src" \ + || _p="$_p;--status counts entries but not the packages they pull in" +grep -q 'if not getattr(args, "status", False):' "$pm_src" \ + || _p="$_p;--status still prints the whole entry table" +grep -q 'install_tag' "$pm_src" \ + || _p="$_p;status invents its own idea of what is installed" +LFS_PKGUSR_DIR="$T/stackcache" python3 - "$pm_src" <<'PYSC' || _p="$_p;the stack cache does not round-trip" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +sp = os.path.join(os.environ["LFS_PKGUSR_DIR"], "y.stack") +os.makedirs(os.path.dirname(sp), exist_ok=True); open(sp, "w").write("book x\n") +pm._stack_mark_done(sp, "book dbus") +assert pm._stack_done_set(sp) == {"book dbus"} +PYSC +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a failed build makes no claim; one casing rule; the stack cache holds" +fi + +# ---- one command, one environment: the stack replayer ----------------------- # +# `packagemanager stack <name> --run` replays a declarative plan through the +# proven machinery: book targets via the install planner, git targets via an +# auto-generated template (make all && make PREFIX=/usr install), check/cfg +# root scripts beside the stack. Dry run by default; a malformed plan is +# refused with a line number, never half-understood. +_p="" +_st="$(dirname "$LFS_TOOL")/stacks/sway.stack" +if [ -f "$_st" ]; then + python3 "$pm_src" stack "$_st" >/dev/null 2>&1 \ + || _p="$_p;the shipped sway stack does not survive its own dry run" + for _sh in kernel-sway sway-session; do + bash -n "$(dirname "$LFS_TOOL")/stacks/$_sh.sh" 2>/dev/null \ + || _p="$_p;stacks/$_sh.sh does not parse" + done +else + _p="$_p;no sway.stack ships with the tools" +fi +grep -q 'git fallback' "$pm_src" \ + || _p="$_p;a book entry with url= cannot fall back to git where the book lacks it" +printf 'book\n' > "$T/bad.stack" +python3 "$pm_src" stack "$T/bad.stack" >/dev/null 2>&1 \ + && _p="$_p;a malformed stack file is accepted instead of refused" +python3 - "$pm_src" <<'PYSTK' || _p="$_p;the git template is wrong (see stderr)" +import sys, tempfile, subprocess, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +e = {"kind":"git","target":"x","opts":{"url":"https://example.com/x","prog":"x"}} +p, _ = pm._stack_git_script(e, tempfile.mkdtemp()) +t = open(p).read() +assert 'is_git="1"' in t and 'installed_program="x"' in t +assert 'make all' in t and 'make PREFIX=/usr install' in t +assert subprocess.run(["bash","-n",p]).returncode == 0 +PYSTK +# the machine's options are merged the moment a script is born +_mo="$T/machineopts"; mkdir -p "$_mo" +printf '[mesa]\ngallium-drivers = radeonsi\n' > "$_mo/m.conf" +printf '#!/bin/bash\nmeson setup .. -Dgallium-drivers=auto\n' > "$_mo/install_Mesa-1" +# AUTOTOOLS TOO: libassuan's `make` builds a PDF manual needing TeX, which +# this system does not build -- and machine.conf only knew `meson setup`. +printf '[libassuan]\nconfigure_args = --disable-doc\n' > "$_mo/auto.conf" +printf '#!/bin/bash\n./configure --prefix=/usr &&\nmake\n' > "$_mo/install_libassuan-1" +# A FIX MUST BE ABLE TO REACH AN EXISTING SYSTEM. machine.conf is never +# overwritten (it is the user's), so a workaround shipped in it never +# arrived: libassuan kept failing on a fix that was in the release. +_fx="$(dirname "$LFS_TOOL")/stacks/package-fixes.conf" +if [ -f "$_fx" ]; then + grep -q 'libassuan' "$_fx" \ + || _p="$_p;the shipped package workarounds are not in the refreshed file" + grep -q 'libassuan' "$(dirname "$LFS_TOOL")/stacks/machine.conf" \ + && _p="$_p;a shipped workaround still lives in the file that is never overwritten" +else + _p="$_p;there is no package-fixes.conf for the workarounds the tools ship" +fi +grep -q 'PACKAGE_FIXES' "$pm_src" \ + || _p="$_p;the shipped workarounds are not read at all" +# ...and they must reach a script that ALREADY EXISTS: the options were +# merged only at generation, so every package that had failed once kept its +# untouched script (libassuan built its TeX manual three releases running). +grep -q 'THE OPTIONS APPLY TO A LOCAL SCRIPT TOO' "$pm_src" \ + || _p="$_p;a package's existing script never gets the machine options" +# and the merge must be idempotent, or a re-run appends the flag again +printf '#!/bin/bash\n./configure --prefix=/usr\n' > "$_mo/idem" +printf '[libassuan]\nconfigure_args = --disable-doc\n' > "$_mo/idem.conf" +for _i in 1 2 3; do + PM_PACKAGE_FIXES="$_mo/idem.conf" PM_MACHINE_CONF=/nonexistent \ + python3 - "$pm_src" "$_mo/idem" <<'PYIDEM' >/dev/null 2>&1 +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.apply_machine_opts('libassuan', sys.argv[2]) +PYIDEM +done +[ "$(grep -c -- '--disable-doc' "$_mo/idem")" = 1 ] \ + || _p="$_p;re-applying the machine options duplicates the flag" +PM_MACHINE_CONF="$_mo/auto.conf" python3 - "$pm_src" "$_mo/install_libassuan-1" <<'PYAUTO' \ + || _p="$_p;machine.conf cannot pass an option to an autotools package" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.apply_machine_opts('libassuan', sys.argv[2]) +body = open(sys.argv[2]).read() +assert "./configure --disable-doc --prefix=/usr" in body, body +PYAUTO +# THE BOOK WRITES `-D key=value` WITH A SPACE. Matching only "-Dkey=" left +# the book's own `-D gallium-drivers=auto` in place and merely prepended +# ours -- meson takes the LAST one, so mesa built for every driver and died +# on `Dependency "libdrm_intel" not found`. +printf '[mesa]\ngallium-drivers = radeonsi\n' > "$_mo/spaced.conf" +printf '#!/bin/bash\nmeson setup .. \\\n -D gallium-drivers=auto \\\n -D valgrind=disabled\n' \ + > "$_mo/install_Mesa-2" +PM_MACHINE_CONF="$_mo/spaced.conf" PM_PACKAGE_FIXES=/nonexistent \ +python3 - "$pm_src" "$_mo/install_Mesa-2" <<'PYSPACED' \ + || _p="$_p;a book option written with a space survives the machine.conf merge" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.apply_machine_opts('mesa', sys.argv[2]) +body = open(sys.argv[2]).read() +assert "gallium-drivers=auto" not in body, body +assert body.count("gallium-drivers=radeonsi") == 1, body +assert "-D valgrind=disabled" in body, body # untouched keys stay as they were +PYSPACED + +PM_MACHINE_CONF="$_mo/m.conf" python3 - "$pm_src" "$_mo/install_Mesa-1" <<'PYMO' \ + || _p="$_p;machine.conf options are not merged into a fresh script" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.apply_machine_opts('mesa', sys.argv[2]) +t = open(sys.argv[2]).read() +assert '-Dgallium-drivers=radeonsi' in t and 'auto' not in t +PYMO +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "one command, one environment: the stack replays through the proven machinery" +fi + +# ---- the book flavour follows the system ------------------------------------ # +# A hardcoded stable-systemd default handed the SysV system the systemd book: +# elogind read NOT IN BOOK and every generated script leaned systemd. +_bt2="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bt2" ]; then + grep -q '_default_selector' "$_bt2" \ + && grep -q '/run/systemd/system' "$_bt2" \ + && ok "the default book flavour follows the running init" \ + || bad "a SysV system still defaults to the systemd book" +fi + +# ---- pm-install end to end (item 6, step 7) -------------------------------- # +# The retired engine's whole job, through the one door: confirm skipped +# (PM_YES), records kept, script handed to the one loop, files installed. +if command -v useradd >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then + _d7="$T/step7"; mkdir -p "$_d7/root/usr/bin" "$_d7/src/pkgusr" "$_d7/state" "$_d7/scripts" + userdel p_dmy7 >/dev/null 2>&1; groupdel p_dmy7 >/dev/null 2>&1 + getent group install >/dev/null || groupadd -g 9999 install + _d7_run() { env LFS_PKGUSR_DIR="$_d7/state" LFS_SRC_ROOT="$_d7/src" \ + LFS_SNAP_ROOT="$_d7/root" LFS_COLLECTOR_PREFIX=nimgnu \ + LFS_PKGUSR_PREFIX=p PM_YES=1 bash "$helper_src" "$@" ; } + _d7_run add-user dmy7 >/dev/null 2>&1 + chgrp install "$_d7/root/usr/bin" && chmod g+w "$_d7/root/usr/bin" + printf '#!/bin/bash\npkg="dmy7-1.0.tar.gz"\ninstall_pkg() { :; }\ncase "${1:-all}" in all) echo hi > %s/root/usr/bin/dmy7-tool ;; esac\n' \ + "$_d7" > "$_d7/scripts/install_dmy7-1.0" + _d7_run pm-install dmy7 "$_d7/scripts/install_dmy7-1.0" >/dev/null 2>&1 + _rc7=$? + _p="" + [ "$_rc7" = 0 ] || _p="$_p;pm-install exited $_rc7" + [ -f "$_d7/root/usr/bin/dmy7-tool" ] || _p="$_p;nothing was installed" + _h7="$(_d7_run pkgusr-home dmy7)" + [ -f "$_h7/install_last" ] || _p="$_p;no install_last record was kept" + # ONE file to edit and run (1.14.16): the canonical install_<nv>, not a + # second copy under the account's name + [ -f "$_h7/install_dmy7-1.0" ] || _p="$_p;the editable script is not in the home" + [ -x "$_h7/install_dmy7-1.0" ] || _p="$_p;the editable script is not runnable" + [ -f "$_h7/install_p_dmy7" ] && _p="$_p;the redundant install_p_<user> copy is back" + [ -f "$_d7/state/manifests/dmy7.files" ] \ + || _p="$_p;the install left no manifest under the step name" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "pm-install does the retired engine's whole job through the one door" + fi + userdel p_dmy7 >/dev/null 2>&1; groupdel p_dmy7 >/dev/null 2>&1 + rm -rf "$_d7" +fi + +# verify must run the same check for every package account (rule 6b) -- the +# pass exists and calls the one implementation, not a second copy. +case "$(_slice_fn "$helper_src" _vfy_misplaced_homes)" in + *_repair_misplaced_home*) ok "verify's misplaced-home pass calls the one repair implementation" ;; + *) bad "verify has no misplaced-home pass, or it re-implements the repair" ;; +esac +case "$(_slice_fn "$helper_src" cmd_verify)" in + *_vfy_misplaced_homes*) ok "cmd_verify runs the misplaced-home pass" ;; + *) bad "cmd_verify never runs the misplaced-home pass" ;; +esac + +# and account creation everywhere runs the one repair -- python's own path +# delegates to fix-home, pm-install's runs it in-process. +_pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ -f "$_pm_src" ]; then + grep -q '"fix-home"' "$_pm_src" \ + && ok "packagemanager delegates the home repair to lfs-helper fix-home" \ + || bad "packagemanager still repairs homes on its own (usermod-only leaves the husk)" +fi +_slice_fn "$helper_src" _pm_add_account | grep -q '_repair_misplaced_home' \ + && ok "pm-install account creation runs the one home repair" \ + || bad "pm-install creates accounts without the home repair" +rm -rf "$_fh" + +# ---- the runner moved out of every script (1.14.0) -------------------------- # +# A generated script was ~230 lines for 5 lines of book commands: source +# lookup, the unpack, the heredoc-per-phase trick, the dispatcher, the usage +# text -- copied into every script by four generators in three dialects. +# Now a script is its variables and one function per phase; everything shared +# is lfs-phases, sourced by the last line. What must hold: +_p="" +_bt14="$(dirname "$LFS_TOOL")/blfs" +_pm14="$(dirname "$LFS_TOOL")/packagemanager" +_mk14="$(dirname "$LFS_TOOL")/Makefile" +# the runner ships everywhere a script can run +[ -f "$PHASES_LIB" ] || _p="$_p;lfs-phases is not beside the tools" +[ -f "$_mk14" ] && { grep -qE '^TOOLS\s*:=.*\blfs-phases\b' "$_mk14" \ + || _p="$_p;make install does not install lfs-phases"; } +grep -q '"lfs-phases"' "$LFS_TOOL" || _p="$_p;lfs does not copy lfs-phases into the chroot" +grep -qE '^PKGUSR_TOOLS=.*\blfs-phases\b' "$helper_src" \ + || _p="$_p;lfs-phases is not claimed by the pkgusr package user" +# it identifies itself like the other tools, and does nothing else when run +bash "$PHASES_LIB" --version | grep -qE '^lfs-phases [0-9.]+ \(build [0-9a-f]{7}\)' \ + || _p="$_p;lfs-phases --version does not print a version and build id" +# the phase names must not shadow a program or a builtin: install(1) is used +# INSIDE install phases, and test is a builtin +grep -qE '^(install|test|build)\(\)' "$PHASES_LIB" \ + && _p="$_p;the runner defines a function named like a command the book uses" +# a script is small and carries none of the scaffolding +python3 - "$LFS_TOOL" <<'PY14' || _p="$_p;a generated LFS script still carries scaffolding" +import sys, importlib.machinery as m +lfs = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +cmds = ["mkdir -v build\ncd build", "../configure", "make", "make check", "make install", + "ln -sv x /usr/lib/y"] +s = lfs._crosschain_script_body("demo", "ch-tools-demo", "demo-1.0", "demo-1.0.tar.*", cmds) +assert len(s.splitlines()) < 50, len(s.splitlines()) +for scaffold in ("_enter_build", "_phase_body", "__LFS_PHASE__", "case \"${LFS_CC_PHASE", + "mktemp", "tar tf", "usage:"): + assert scaffold not in s, scaffold +assert s.rstrip().endswith('. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"'), s[-120:] +assert 'pkgusr_stage="cross"' in s +# the commands are NOT indented: an indented heredoc terminator is no terminator +assert "\n../configure\n" in s, s +# the book's test block is the test phase, and the install block the install phase +assert "test_pkg() {\nmake check\n}" in s, s +assert "install_pkg() {\nmake install\n}" in s, s +PY14 +python3 - "$_bt14" <<'PY14B' || _p="$_p;a generated BLFS script still carries scaffolding" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +data = dict(name_version="demo-1.0", pkg="demo-1.0.tar.gz", name="demo", link="https://x/demo-1.0.tar.gz", + md5="abc", info="", required=[], recommended=[], optional=[], installed_content={}, + is_module=False, group=None, install_cmds="./configure &&\nmake", build_cmds="./configure &&\nmake", + install_root_cmds="make install", config_cmds="", between_cmds=["wget -i list"], page_env={}, + additional_downloads=[]) +fn, s = b.render_script(data, "#demo", "13.0") +assert len(s.splitlines()) < 50, len(s.splitlines()) +for scaffold in ("_enter_build", "_fetch() {", "_mirrors_for", "rm -rf tmp && mkdir tmp", "usage:", "systemctl()"): + assert scaffold not in s, scaffold +assert "prepare_pkg() {\nwget -i list\n}" in s, s +# the mirror of last resort names the BOOK's release, not a literal +assert "pkgusr_book_mirror='https://www.linuxfromscratch.org/blfs/downloads/13.0'" in s, s +src = open(sys.argv[1]).read() +assert "blfs/downloads/13.0/" not in src, "a release number is hardcoded in the generator" +# a phase of comments alone is still a function bash accepts +data["install_root_cmds"] = "# nothing here\n" +fn, s = b.render_script(data, "#demo", "13.0") +assert "install_pkg() {\n# nothing here\n:\n}" in s, s +# and a command left ending in && by a LATER commenting filter is repaired: +# it ran before five of them, and cachecontrol's install failed `bash -n` +data["install_root_cmds"] = "pip3 install --no-index x &&\ntestenv/bin/python3 -m pytest" +fn, s = b.render_script(data, "#demo", "13.0") +import subprocess, tempfile, os +p = os.path.join(tempfile.mkdtemp(), fn); open(p, "w").write(s) +assert subprocess.run(["bash", "-n", p]).returncode == 0, s +PY14B +# the template packagemanager writes for a package not in the book, too +python3 - "$_pm14" <<'PY14C' || _p="$_p;the packagemanager template still carries its own (older) scaffolding" +import sys, importlib.machinery as m, tempfile, subprocess +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +t = pm._TEMPLATE +for stale in ("unpack_pkg && build_pkg", "mv tmp/* .", 'wget -4 "$link"', "BUILD_ROOT:-$PWD"): + assert stale not in t, stale +assert 'pkgusr_run "$@"' in t +e = {"target": "foo", "kind": "git", "opts": {"url": "https://g/foo.git", "build": "make all;make doc"}} +p, _ = pm._stack_git_script(e, tempfile.mkdtemp()) +s = open(p).read() +assert "build_pkg() {\nmake all\nmake doc\n}" in s, s +assert subprocess.run(["bash", "-n", p]).returncode == 0 +assert pm.extract_phase_cmds(s, "build") == "make all\nmake doc" +PY14C +# the version comes from the header line every generator writes. This read +# a "# package:" comment no generator ever wrote, so every pkgusr.info said +# version= and nothing more. +_vf="$T/vfs.sh"; printf 'name_version="xmlto-0.0.29"\n' > "$_vf" +[ "$(_slice_fn "$helper_src" _version_from_script > "$T/vfn.sh"; . "$T/vfn.sh"; _version_from_script "$_vf")" = "0.0.29" ] \ + || _p="$_p;_version_from_script cannot read the version from a generated script" +printf "name_version='wget-1.25.0'\n" > "$_vf" +[ "$(. "$T/vfn.sh"; _version_from_script "$_vf")" = "1.25.0" ] \ + || _p="$_p;_version_from_script does not accept the single-quoted header blfs writes" +# the runner refuses what it cannot re-execute, and does nothing when only +# sourced for its metadata +printf 'name_version="d-1"\nbuild_pkg(){ :; }\ninstall_pkg(){ :; }\n. "$PKGUSR_LIB" && pkgusr_run "$@"\n' > "$T/rr.sh" +bash -s < "$T/rr.sh" >/dev/null 2>&1 && _p="$_p;a script piped on stdin is not refused" +[ "$(bash -c '. '"$T"'/rr.sh; echo "$name_version"')" = "d-1" ] \ + || _p="$_p;sourcing a script for its metadata runs it" +# the runner must not know either of the old test switches alone +grep -q 'PKGUSR_TESTS' "$PHASES_LIB" || _p="$_p;the runner has no test switch" +grep -q 'PKGUSR_TESTS=1' "$helper_src" || _p="$_p;lfs-helper does not pass the test switch" +grep -q '"PKGUSR_TESTS"' "$_pm14" || _p="$_p;packagemanager does not pass the test switch" +# a missing runner is one clear message from lfs-helper, not a shell error +# from inside su +_slice_fn "$helper_src" run_phase_as | grep -q "needs the phase runner 'lfs-phases'" \ + || _p="$_p;lfs-helper does not explain a missing lfs-phases" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a script is its variables and its commands; everything shared is lfs-phases" +fi + +# ---- silence is not progress (1.14.2) -------------------------------------- # +# Reported from a real build: it printed +# ===== [37/106] gcc ===== +# # user : p_gcc (package user) phase: all +# and then nothing for ten minutes, with one bash process on a core and no +# child. Between those two lines sits _claim_earlier_stages, which forked +# three or four times PER PATH (a command substitution for +# strip_host_prefix, a stat, a chown) over every earlier stage's manifest. +# For GCC that is gcc-pass1 + gcc-pass2 + libstdcpp. Nothing is printed +# while it runs, so the build looks hung. +_cl="$T/claim"; mkdir -p "$_cl/manifests" "$_cl/tree/d1" +_p="" +for _i in $(seq 1 400); do : > "$_cl/tree/f$_i"; done +ls -1 "$_cl/tree"/f* > "$_cl/manifests/demo-pass1.files" +printf '%s\n' "$_cl/tree/d1" > "$_cl/manifests/demo-pass1.dirs" +# one file already belongs to another package: it must be left alone +chown bin:bin "$_cl/tree/f7" 2>/dev/null +{ + echo 'SNAP_ROOT="/"; LFS_HOST_MOUNT=""; MANIFESTS="'"$_cl"'/manifests"' + echo 'user_exists() { return 0; }' + echo 'is_install_dir() { return 1; }' + echo 'count_stdin_lines() { wc -l; }' + echo 'say() { echo "$*"; }' + echo '_real_tool() { command -v "$1"; }' + _slice_fn "$helper_src" _claim_candidates + _slice_fn "$helper_src" _claim_earlier_stages + echo '_claim_earlier_stages demo daemon' +} > "$_cl/drv.sh" +_t0=$(date +%s) +_out="$(bash "$_cl/drv.sh" 2>&1)" +_t1=$(date +%s) +case "$_out" in + *"400 path(s)"*) ;; # 400 files less the one another package owns, plus the dir + *) _p="$_p;the claim did not take the 400 paths it should have: $_out" ;; +esac +[ "$(stat -c %U "$_cl/tree/f7" 2>/dev/null)" = bin ] \ + || _p="$_p;the claim took a file that belongs to another package" +[ "$(stat -c %U "$_cl/tree/f1" 2>/dev/null)" = daemon ] \ + || _p="$_p;the claim did not take a file its earlier stage left behind" +[ "$(stat -c %U "$_cl/tree/d1" 2>/dev/null)" = daemon ] \ + || _p="$_p;the claim did not take the directories" +[ $((_t1 - _t0)) -le 5 ] || _p="$_p;400 paths took $((_t1 - _t0))s -- it is forking per path again" +# it must not go back to a fork per path +_slice_fn "$helper_src" _claim_earlier_stages | grep -q 'stat -c %U "\$p"' \ + && _p="$_p;the claim stats one path at a time again" +# and GCC's first stage is not spelled gcc +_slice_fn "$helper_src" _claim_earlier_stages | grep -q 'libstdcpp' \ + || _p="$_p;libstdcpp's files are never claimed by the gcc account" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package claims its earlier stages in two passes, not four forks per file" +fi + +# ---- the bootstrap circle, explained from where you are (1.14.3) ----------- # +# Reported from a real chroot: `packagemanager bootstrap --run` reached stage +# 2 and said +# wget: there is no BLFS book on this system to look it up in. +# Download it (needs working networking): blfs fetch +# ...on a system whose whole problem was that it has no download tool. Worse, +# the host resolves wget's own URL from that same book (blfs sources wget, via +# lfs's bootstrap_sources), so a host with no cached book staged no wget +# tarball either: BOTH halves are missing, and the advice named neither. +_p="" +_pm14c="$(dirname "$LFS_TOOL")/packagemanager" +_bsdir="$T/nobook-src"; mkdir -p "$_bsdir" +_out="$(LFS_SOURCES_DIR="$_bsdir" python3 - "$_pm14c" <<'PYBS' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._blfs_book_available = lambda: False +pm._download_tool_available = lambda: False +pm._explain_not_in_book("wget") +PYBS +)" +case "$_out" in + *"blfs fetch"*"cannot work"*) ;; + *) _p="$_p;a system with no download tool is still told to fetch the book" ;; +esac +case "$_out" in + *"no wget tarball"*) ;; + *) _p="$_p;the missing wget tarball is not mentioned, so it is found out later" ;; +esac +case "$_out" in + *"on the host"*) ;; + *) _p="$_p;the recovery does not say where it has to be done" ;; +esac +# with a download tool present the short advice is right and must stay short +_out2="$(LFS_SOURCES_DIR="$_bsdir" python3 - "$_pm14c" <<'PYBS2' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._blfs_book_available = lambda: False +pm._download_tool_available = lambda: True +pm._explain_not_in_book("wget") +PYBS2 +)" +case "$_out2" in + *"on the host"*) _p="$_p;a system that CAN fetch is sent to the host anyway" ;; +esac +case "$_out2" in + *"blfs fetch"*) ;; + *) _p="$_p;a system that can fetch is not told to" ;; +esac +# and the host says so loudly when it has no book to copy in, because that is +# the same failure one step earlier +python3 - "$LFS_TOOL" <<'PYHOST' || _p="$_p;the host reports a missing BLFS book as an aside, not as the failure it causes" +import sys +src = open(sys.argv[1]).read() +i = src.index("(none cached here") if "(none cached here" in src else -1 +assert i == -1, "install-tools still prints the missing book as a plain note" +j = src.index("BLFS book: none cached on this host") +assert 'warn("! BLFS book: none cached' in src[j - 40:j + 60], src[j - 60:j + 60] +PYHOST +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the bootstrap circle is explained from the side of it you are standing on" +fi + +# ---- the error reporter must not fail (1.14.4) ----------------------------- # +# Reported from a real build. util-macros stopped for a missing XORG_CONFIG, +# the box explained it correctly, and then: +# ! Edit the script, then retry just this phase: +# /usr/bin/lfs-helper: line 5596: _retry: unbound variable +# $_retry was computed INSIDE the permission-failure branch and used by three +# hints outside it, so under `set -u` every OTHER kind of failure crashed the +# reporter on the line that tells you how to retry. The last thing a person +# sees when something breaks cannot itself break. +_p="" +_rp="$T/reporter"; mkdir -p "$_rp" +{ + echo 'set -u' + echo 'fail() { echo "$*"; }' + echo 'say() { echo "$*"; }; warn() { echo "$*" >&2; }' + echo 'detail() { :; }; hint() { :; }; show_build_errors() { :; }' + echo 'pkg_owner_name() { echo "p_$1"; }' + echo 'unprefix_pkg_user() { echo "${1#p_}"; }' + echo 'phase_done() { return 1; }' + echo 'SNAP_ROOT="/"' + # deliberately NOT setting SRCROOT: the report must survive a caller that + # has not set every variable it happens to read (1.14.33) + python3 - "$helper_src" <<'PYREP' +import sys +s = open(sys.argv[1]).read() +i = s.index(" # A FEATURE MISSING FROM AN INSTALLED PACKAGE.") +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('report() {\n local name="$1" phase="$2" log="$3" jobs=""\n' + body + '\n}') +PYREP +} > "$_rp/rep.sh" +bash -n "$_rp/rep.sh" || _p="$_p;the failure report does not even parse on its own" +# every kind of failure the reporter recognises, and one it does not +while IFS= read -r _kind; do + printf '%s\n' "$_kind" > "$_rp/log" + _o="$(bash -c '. "'"$_rp"'/rep.sh"; report util-macros all "'"$_rp"'/log"' 2>&1)" + case "$_o" in + *"unbound variable"*|*"bad substitution"*) + _p="$_p;the failure report crashes on: $_kind" ;; + esac + # and it must still end with a usable retry command + # (the account name resolves to whichever of the two exists; in this + # harness neither does, so only the shape is checked) + case "$_o" in + *"lfs-helper build "*" --phase all --force"*) ;; + *) _p="$_p;no retry command printed after: $_kind" ;; + esac +done <<'KINDS' +this package needs XORG_CONFIG to be set +permission denied +foo.c:12:3: error: bad thing +Program 'gperf' not found +/usr/bin/ld: cannot find -lfoo +nothing familiar at all +KINDS +# a fresh account's home is not news: add_package_user takes no home +# argument, so EVERY new account is briefly misplaced and instantly repaired. +# Reporting that as a warning opened every install with two lines of alarm. +_slice_fn "$helper_src" _repair_misplaced_home > "$_rp/rmh.sh" +grep -q 'fresh="${3:-0}"' "$_rp/rmh.sh" \ + || _p="$_p;the repair cannot tell a fresh account from a long-wrong one" +grep -q '\[ "$fresh" = 1 \] && \[ "$got" = "$husk" \]' "$_rp/rmh.sh" \ + || _p="$_p;the quiet path is not restricted to add_package_user's own home" +grep -q 'misplaced home: $acct is recorded at $got' "$_rp/rmh.sh" \ + || _p="$_p;a genuinely misplaced home is no longer reported at all" +_slice_fn "$helper_src" _pm_add_account | grep -q '_repair_misplaced_home "$1" 1 1' \ + || _p="$_p;account creation does not tell the repair the home is fresh" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the failure report survives every failure, and a new account is not an alarm" +fi + +# ---- one package, several filenames (1.14.5) ------------------------------- # +# Reported from a real chroot. util-macros unpacked and built, then +# packagemanager script install util-macros +# !! no unpacked source in /usr/src/pkgusr/p_util-macros/src +# with the source sitting right there. The build markers were named from $0, +# and this project runs ONE script under three names by design: the book copy +# (install_util-macros-1.20.2), the staged copy (install_util-macros) and the +# account's copy (install_p_util-macros). Unpack recorded one, install looked +# for another. The old scaffolding keyed them to the package (.cc-dir-<name>, +# .pm_build_cwd); keying them to the file was a 1.14.0 regression. +_mk="$T/marker"; mkdir -p "$_mk/sources" "$_mk/src" "$_mk/stage/demo-3.0" +_p="" +printf '#!/bin/sh\nprintf "all:\\n\\techo ok > built\\ninstall:\\n\\tcp built %s/out\\n" > Makefile\n' "$_mk" \ + > "$_mk/stage/demo-3.0/configure" +chmod +x "$_mk/stage/demo-3.0/configure" +( cd "$_mk/stage" && tar czf ../sources/demo-3.0.tar.gz demo-3.0 ) 2>/dev/null +{ + echo '#!/bin/bash' + echo 'name_version="demo-3.0"' + echo 'pkg_glob="demo-3.0.tar.*"' + echo 'build_pkg() {' + echo './configure' + echo 'make' + echo '}' + echo 'install_pkg() {' + echo 'make install' + echo '}' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_mk/install_demo-3.0" +cp "$_mk/install_demo-3.0" "$_mk/install_p_demo" # the account's copy +_env="PKGUSR_LIB=$PHASES_LIB LFS_BUILD_ROOT=$_mk/src LFS_SOURCES_DIR=$_mk/sources" +if env $_env bash "$_mk/install_demo-3.0" unpack >/dev/null 2>&1 \ + && env $_env bash "$_mk/install_demo-3.0" build >/dev/null 2>&1; then + # the marker names the PACKAGE, so any copy of the script finds it + ls "$_mk/src"/.pkgusr-demo-3.0.dir >/dev/null 2>&1 \ + || _p="$_p;the build marker is not named for the package" + env $_env bash "$_mk/install_p_demo" install >/dev/null 2>&1 \ + || _p="$_p;a second copy of the same script cannot find the unpacked source" + [ "$(cat "$_mk/out" 2>/dev/null)" = ok ] \ + || _p="$_p;the install phase did not run from the build directory" + # and with no marker at all the tree is still found, not thrown away + rm -f "$_mk/src"/.pkgusr-* "$_mk/out" + _o="$(env $_env bash "$_mk/install_p_demo" install 2>&1)" + case "$_o" in + *"no marker"*) ;; + *) _p="$_p;an unpacked tree with no marker is not recovered: $_o" ;; + esac + [ "$(cat "$_mk/out" 2>/dev/null)" = ok ] \ + || _p="$_p;the recovered tree was not usable" +else + _p="$_p;the fixture package would not unpack and build at all" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "build markers name the package, so every copy of a script finds the source" +fi + +# ---- a failure has to say WHERE (1.14.6) ----------------------------------- # +# Reported from a real chroot. Running one phase on a package whose build had +# never finished gave, in full: +# make: *** No rule to make target 'install'. Stop. +# !! phase 'install' FAILED (exit 2) +# phase 'install' failed for p_util-macros (exit 2). +# logs: /usr/src/pkgusr/p_util-macros/log +# -- the package's own words for "there is no Makefile", and not one of the +# three script paths, the source directory, or the reason. +_p="" +_wu="$T/unbuilt"; mkdir -p "$_wu/sources" "$_wu/src" "$_wu/stage/demo-4.0" +printf '#!/bin/sh\ntouch configured\n' > "$_wu/stage/demo-4.0/configure" +chmod +x "$_wu/stage/demo-4.0/configure" +( cd "$_wu/stage" && tar czf ../sources/demo-4.0.tar.gz demo-4.0 ) 2>/dev/null +{ + echo '#!/bin/bash' + echo 'name_version="demo-4.0"' + echo 'pkg_glob="demo-4.0.tar.*"' + echo 'build_pkg() {' + echo './configure' + echo '}' + echo 'install_pkg() {' + echo 'make install' + echo '}' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_wu/install_demo-4.0" +_env="PKGUSR_LIB=$PHASES_LIB LFS_BUILD_ROOT=$_wu/src LFS_SOURCES_DIR=$_wu/sources" +env $_env bash "$_wu/install_demo-4.0" unpack >/dev/null 2>&1 \ + || _p="$_p;the fixture would not unpack" +# install with no build ever run: say so BEFORE the package's own tools do +_o="$(env $_env bash "$_wu/install_demo-4.0" install 2>&1)" +case "$_o" in + *"recorded a finished build phase"*) ;; + *) _p="$_p;a phase run on an unbuilt tree does not say the build never finished" ;; +esac +# after a real build the record exists and the warning is gone +env $_env bash "$_wu/install_demo-4.0" build >/dev/null 2>&1 +[ -f "$_wu/src/.pkgusr-demo-4.0.built" ] \ + || _p="$_p;a finished build records nothing, so no later phase can tell" +_o2="$(env $_env bash "$_wu/install_demo-4.0" install 2>&1)" +case "$_o2" in + *"recorded a finished build phase"*) + _p="$_p;the warning still fires after the build finished" ;; +esac +# and a build that FAILS must not claim it finished +{ + echo '#!/bin/bash' + echo 'name_version="demo-4.0"' + echo 'pkg_glob="demo-4.0.tar.*"' + echo 'build_pkg() { false; }' + echo 'install_pkg() { :; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_wu/install_bad" +rm -f "$_wu/src/.pkgusr-demo-4.0.built" +env $_env bash "$_wu/install_bad" build >/dev/null 2>&1 +[ -f "$_wu/src/.pkgusr-demo-4.0.built" ] \ + && _p="$_p;a failed build recorded itself as finished" +# packagemanager's own report names the files it ran and the one you edit +python3 - "$(dirname "$LFS_TOOL")/packagemanager" "$T" <<'PYPATHS' || _p="$_p;the failure report cannot name the script, the edit file or the source" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +h = os.path.join(sys.argv[2], "pmhome"); os.makedirs(os.path.join(h, "src", "util-macros-1.20.2")) +for f in ("install_util-macros-1.20.2", "install_p_util-macros", + "install_p_util-macros.edited"): + open(os.path.join(h, f), "w").write("x") +# the editable one is the versioned one -- not the copy, not the .edited backup +assert pm._versioned_script_in(h).endswith("install_util-macros-1.20.2"), \ + pm._versioned_script_in(h) +assert pm._unpacked_source_dir(h).endswith("src/util-macros-1.20.2") +src = open(sys.argv[1]).read() +i = src.index("phase '{phase}' failed for {name}") +blk = src[i:i + 1600] +for need in ("ran :", "edit :", "source :", "logs :", "retry :"): + assert need in blk, need +assert "packagemanager script build" in blk, "no hint to run the build phase first" +PYPATHS +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a failed phase names its files, and an unbuilt tree says so first" +fi + +# ---- the build environment has to reach the build (1.14.7) ----------------- # +# Reported from a real chroot. The person ran +# packagemanager env --set XORG_CONFIG=... +# was shown it in the table, and the very next command still failed with +# this package needs XORG_CONFIG to be set +# lfs-helper's run_phase_as pastes /etc/pkgusr/build.env in front of the +# command it runs; packagemanager's `script <phase>` built its own `su` and +# passed nothing. `su -` is a login shell and drops the caller's environment, +# so the variables have to travel INSIDE the command. +_p="" +_be="$T/buildenv"; mkdir -p "$_be" +printf '# comment\nXORG_CONFIG=--prefix=/usr --sysconfdir=/etc --disable-static\n' \ + > "$_be/build.env" +_o="$(LFS_BUILD_ENV="$_be/build.env" python3 - "$(dirname "$LFS_TOOL")/packagemanager" <<'PYENV' +import sys, shlex, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +be = pm._read_build_env() +assert "XORG_CONFIG" in be, be +# the pairs, exactly as cmd_script builds them +pairs = "".join("%s=%s " % (k, shlex.quote(v)) for k, v in sorted(be.items())) +print(pairs) +PYENV +)" +case "$_o" in + *"XORG_CONFIG='--prefix=/usr --sysconfdir=/etc --disable-static'"*) ;; + *) _p="$_p;a value with spaces is not passed as one value: $_o" ;; +esac +# and it must actually arrive: the same guard the book's scripts carry +_g="$T/guard.sh" +{ + echo 'name_version="demo-9.0"' + echo 'build_pkg() {' + echo '[ -n "${XORG_CONFIG:-}" ] || { echo "needs XORG_CONFIG" >&2; exit 1; }' + echo 'echo "got [$XORG_CONFIG]"' + echo '}' + echo 'install_pkg() { :; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_g" +mkdir -p "$T/genv-build" +_r="$(env LFS_BUILD_ROOT="$T/genv-build" PKGUSR_LIB="$PHASES_LIB" \ + sh -c "env $_o bash $_g build" 2>&1)" +case "$_r" in + *"got [--prefix=/usr --sysconfdir=/etc --disable-static]"*) ;; + *) _p="$_p;the variable does not reach the phase: $_r" ;; +esac +# the command line pm prints must be the one it runs +_src14="$(sed -n '/THE BUILD ENVIRONMENT HAS TO REACH THE BUILD/,/_records = phase/p' \ + "$(dirname "$LFS_TOOL")/packagemanager")" +printf '%s' "$_src14" | grep -q '_read_build_env()' \ + || _p="$_p;the script path does not read the build environment file" +printf '%s' "$_src14" | grep -q 'shlex.quote(v)' \ + || _p="$_p;the pairs are not quoted, so a value with spaces splits" +printf '%s' "$_src14" | grep -q 'cd ~ &&' \ + || _p="$_p;the phase no longer runs from the package user's home" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "what 'packagemanager env' sets reaches the build, spaces and all" +fi + +# ---- one book load, not one per package (1.14.8) --------------------------- # +# Reported from a real chroot: a 151-package plan printed +# cached script is from blfs 1.14.3, this is 1.14.7 -- regenerating +# twice and then sat silent for five minutes, and the person killed it. +# resolve_script generated ONE PER PACKAGE, each a subprocess that loaded and +# indexed the whole book again. `blfs script` has always taken a list of +# targets and used only the first. +_p="" +_pmw="$(dirname "$LFS_TOOL")/packagemanager" +_bw="$(dirname "$LFS_TOOL")/blfs" +_bookf="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bookf="$_c" && break +done +if [ -n "$_bookf" ]; then + _wd="$T/prewarm"; mkdir -p "$_wd" + _n="$(python3 "$_bw" --book-file "$_bookf" script xcb-proto dbus libxau \ + -o "$_wd" 2>/dev/null | grep -c '^Created ')" + [ "$_n" = 3 ] || _p="$_p;blfs script writes $_n of 3 packages asked for in one call" + # and packagemanager uses that: a plan pre-generates in one pass + LFS_BUILD_ENV=/nonexistent BLFS_BOOK_FILE="$_bookf" \ + python3 - "$_pmw" "$T/prewarm2" <<'PYPW' || _p="$_p;the plan does not pre-generate its scripts in one pass" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.pkgusr_home = lambda n: "/nonexistent/" + n # no local copies to prefer +out = sys.argv[2]; os.makedirs(out, exist_ok=True) +plan = [("xcb-proto", "xcb-proto-1.17.0", "new"), ("dbus", "dbus-1.16.2", "new"), + ("libxau", "libXau-1.0.12", "new")] +pm._prewarm_scripts(plan, out) +assert len(os.listdir(out)) == 3, os.listdir(out) +# run again: everything is current, so nothing is regenerated +before = sorted((f, os.path.getmtime(os.path.join(out, f))) for f in os.listdir(out)) +pm._prewarm_scripts(plan, out) +assert before == sorted((f, os.path.getmtime(os.path.join(out, f))) for f in os.listdir(out)), \ + "current scripts were regenerated anyway" +PYPW + grep -q '_prewarm_scripts(plan, script_dir' "$_pmw" \ + || _p="$_p;the install plan never calls the pre-generation pass" +fi +# and the pre-build env check must read WHERE THE BUILD READS. It asked +# os.environ only, so after `packagemanager env --set XORG_CONFIG=...` every +# Xorg package was still announced as missing it -- and the cure it printed, +# `export`, is the one that cannot work through `su -`. +_ev="$T/envwarn"; mkdir -p "$_ev" +printf 'XORG_CONFIG=--prefix=/usr\n' > "$_ev/build.env" +printf '#!/bin/bash\nbuild_pkg() {\n[ -n "${XORG_CONFIG:-}" ] || { echo "this package needs XORG_CONFIG to be set" >&2; exit 1; }\n}\n' \ + > "$_ev/install_demo-1.0" +_o="$(LFS_BUILD_ENV="$_ev/build.env" python3 - "$_pmw" "$_ev/install_demo-1.0" 2>&1 <<'PYEW' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._warn_unset_env("demo", sys.argv[2]) +PYEW +)" +[ -z "$_o" ] || _p="$_p;a variable set in build.env is still reported as missing: $_o" +_o2="$(LFS_BUILD_ENV="$_ev/empty.env" python3 - "$_pmw" "$_ev/install_demo-1.0" 2>&1 <<'PYEW2' +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +os.environ.pop("XORG_CONFIG", None) +pm._warn_unset_env("demo", sys.argv[2]) +PYEW2 +)" +case "$_o2" in + *"packagemanager env --set XORG_CONFIG"*) ;; + *) _p="$_p;an unset variable is not reported, or not with a cure that works" ;; +esac +case "$_o2" in + *"export XORG_CONFIG=..."*) + _p="$_p;it still tells you to export, which su - discards" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a plan reads the book once, and the env check reads where the build reads" fi -# ---- /sources is normalised on the way into the chroot ---------------------- # -# Book 3.1 wants $LFS/sources root:root 1777. The host `lfs` account comes from -# book 4.3's own useradd, which pins no uid, so it lands wherever the host had a -# gap -- 10753 on a Debian host. Package users start at 10000, so that uid does -# not merely read as a number inside the chroot, it COLLIDES with a real package -# user and tarballs report as owned by something unrelated. -# -# _normalize_sources was only called from get-sources and restart, so a tree set -# up before it existed kept the host uid forever. Entry is the one path every -# build goes through, whatever it did before. -python3 - "$LFS_TOOL" <<'PYSRCNORM' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) -problems = [] -if not m: - problems.append("cmd_bs_chroot is gone") +# ---- the commands are the fact, the dep list is an opinion (1.14.9) -------- # +# Reported from a real chroot: three packages into a 151-package plan, +# /usr/src/pkgusr/p_libheif/install_libheif: line 33: cmake: command not found +# libheif's BLFS page has NO Required section: CMake reaches it only through +# x265, and x265 was already installed, so nothing pulled cmake into the plan. +# The dependency list is what the book says; the script's own commands are +# what will actually run, and they are readable before anything is built. +_p="" +_pm9="$(dirname "$LFS_TOOL")/packagemanager" +_b9="$(dirname "$LFS_TOOL")/blfs" +_bk9="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk9="$_c" && break +done +if [ -n "$_bk9" ]; then + _sd9="$T/tools"; mkdir -p "$_sd9" + python3 "$_b9" --book-file "$_bk9" script libheif libtasn1 -o "$_sd9" >/dev/null 2>&1 + python3 - "$_pm9" "$_sd9" <<'PYT' || _p="$_p;the missing build tool is not found from the script's own commands" +import sys, os, glob, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2] +heif = glob.glob(os.path.join(d, "install_libheif*"))[0] +tas = glob.glob(os.path.join(d, "install_libtasn1*"))[0] +# pretend nothing is installed, so the check is about the parsing +pm._tool_available = lambda t: False # nothing installed anywhere +got = pm._warn_missing_build_tools("libheif", heif) +assert "cmake" in got, got +# a package that needs no build-system driver stays silent -- the check must +# not turn into noise on every package +assert pm._warn_missing_build_tools("libtasn1", tas) == [], "false positive" +# and a tool that IS present is not reported +pm._MISSING_TOOLS.clear() +pm._tool_available = lambda t: True # everything installed +assert pm._warn_missing_build_tools("libheif", heif) == [] +# warned once per tool, not once per package +pm._MISSING_TOOLS.clear() +pm._tool_available = lambda t: False +import io, contextlib +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm._warn_missing_build_tools("libheif", heif) + pm._warn_missing_build_tools("other", heif) +assert buf.getvalue().count("builds with `cmake`") == 1, buf.getvalue() +assert pm._MISSING_TOOLS["cmake"] == ["libheif", "other"], pm._MISSING_TOOLS +PYT +fi +# the checks run for a script taken from the cache too -- pre-generating a +# whole plan in one pass (1.14.8) must not skip them +_rs="$(sed -n '/^def resolve_script(/,/^def /p' "$_pm9")" +[ "$(printf '%s' "$_rs" | grep -c '_check_script_prereqs')" -ge 2 ] \ + || _p="$_p;a cached script is used without checking its prerequisites" +grep -q '_report_missing_tools()' "$_pm9" \ + || _p="$_p;the plan does not summarise the tools it is missing" +# EVERY SUGGESTED INSTALL CARRIES ITS DEPENDENCIES. `packagemanager install +# cmake --run` builds cmake alone, and cmake then fails on +# CMAKE_USE_SYSTEM_CURL is ON but a curl is not found +# because cURL is one of its four RECOMMENDED deps -- so the tool suggested +# to fix a missing tool has to be suggested with --recursive. +_bad="$(grep -n 'packagemanager install %s --run"' "$_pm9" || true)" +[ -z "$_bad" ] || _p="$_p;a suggested install still omits --recursive: $_bad" +# (the command is built by _install_cmd_for now, which checks the book first) +sed -n '/def _install_cmd_for/,/^def /p' "$_pm9" \ + | grep -q 'install %s --run --recursive' \ + || _p="$_p;the missing-tool hint does not build the tool's dependencies" +sed -n '/def _report_missing_tools/,/^def /p' "$_pm9" \ + | grep -q 'install %s --run --recursive' \ + || _p="$_p;the summary's install command does not build dependencies" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a build tool the book forgot is found in the script, before the build" +fi + +# ---- the book sets them together (1.14.11) --------------------------------- # +# Reported from a real chroot, seventeen packages into a plan: +# this package needs XORG_PREFIX to be set +# The person had set XORG_CONFIG hours earlier, from the same BLFS page that +# sets both: +# export XORG_PREFIX="/usr" +# export XORG_CONFIG="--prefix=$XORG_PREFIX --sysconfdir=/etc ..." +# and nothing said the other half was missing -- because `env --set` looks at +# the cached scripts to know what to ask, and every one of them was from an +# older blfs, so it asked nothing at all. +_p="" +_pmA="$(dirname "$LFS_TOOL")/packagemanager" +_ea="$T/envcomp"; mkdir -p "$_ea" +python3 - "$_pmA" "$_ea" <<'PYC' || _p="$_p;setting one half of a book pair does not mention the other" +import sys, os, io, contextlib, importlib.machinery as m +os.environ["LFS_BUILD_ENV"] = os.path.join(sys.argv[2], "build.env") +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm._warn_env_companions({"XORG_CONFIG": "--prefix=/usr"}) +out = buf.getvalue() +assert "XORG_PREFIX" in out, out +assert "env --set XORG_PREFIX=/usr" in out, out # with the book's value +# both ways round, and silent once both are set +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm._warn_env_companions({"XORG_CONFIG": "x", "XORG_PREFIX": "/usr"}) +assert buf.getvalue() == "", buf.getvalue() +PYC +# and in a plan: once per VARIABLE, plus a summary naming how many want it +python3 - "$_pmA" "$_ea" <<'PYS' || _p="$_p;the plan repeats the same variable per package, or never summarises it" +import sys, os, io, contextlib, importlib.machinery as m +os.environ["LFS_BUILD_ENV"] = os.path.join(sys.argv[2], "empty.env") +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +os.environ.pop("XORG_PREFIX", None) +sc = os.path.join(sys.argv[2], "install_demo-1.0") +open(sc, "w").write('build_pkg() {\n[ -n "${XORG_PREFIX:-}" ] || ' + '{ echo "this package needs XORG_PREFIX to be set" >&2; exit 1; }\n}\n') +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + for pkg in ("libevdev", "libinput", "xorg-server"): + pm._warn_unset_env(pkg, sc) + pm._report_missing_env() +out = buf.getvalue() +assert out.count("needs XORG_PREFIX set before") == 1, out +assert "needed by 3 package(s), first: libevdev" in out, out +assert pm._MISSING_ENV["XORG_PREFIX"] == ["libevdev", "libinput", "xorg-server"] +PYS +# the summary runs with the plan's other prerequisite report +sed -n '/def _report_missing_tools/,/^def /p' "$_pmA" | grep -q '_report_missing_env()' \ + || _p="$_p;the plan prints missing tools but not missing variables" +# a run where EVERY cached script is stale must not silently ask nothing +sed -n '/def _env_wanted_by_scripts/,/^def cmd_env/p' "$_pmA" \ + | grep -q 'nothing here can say which variables' \ + || _p="$_p;when every script is stale, env --set still says nothing useful" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a variable the book pairs with another is asked for at the same time" +fi + +# ---- a plan that cannot finish should not start (1.14.12) ------------------ # +# Reported from a real chroot. 1.14.9's check knew libaom needs a cmake that +# is not installed -- and said nothing, because resolve_script returns a +# script already sitting in the package user's home BEFORE either place the +# checks were wired in, and that is every package attempted once. So the run +# built twenty-two packages and then stopped exactly where the check would +# have pointed. Warning and building anyway was the other half of the +# mistake: a plan known to be unable to finish should not start. +_p="" +_pmB="$(dirname "$LFS_TOOL")/packagemanager" +# the checks are on EVERY path out of resolve_script, home copy included +_rs="$(sed -n '/^def resolve_script(/,/^def /p' "$_pmB")" +[ "$(printf '%s' "$_rs" | grep -c '_check_script_prereqs')" -ge 3 ] \ + || _p="$_p;a script from the package user's home is used without its checks" +printf '%s' "$_rs" | sed -n '/return local/!d;=' >/dev/null +printf '%s' "$_rs" | grep -B4 'return local' | grep -q '_check_script_prereqs' \ + || _p="$_p;the home-copy path returns before the checks run" +python3 - "$_pmB" <<'PYSTOP' || _p="$_p;a plan missing what it needs still starts" +import sys, os, tempfile, types, io, contextlib, importlib.machinery as m +os.environ["LFS_BUILD_ENV"] = tempfile.mktemp() +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm.shutil.which = lambda *a, **k: None +d = tempfile.mkdtemp(); sc = os.path.join(d, "install_libaom-3.13.1") +open(sc, "w").write("build_pkg() {\ncmake -B build\n}\n") +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm._warn_missing_build_tools("libaom", sc) + # a DRY RUN still shows the plan: nothing is being built, so nothing to stop + pm._stop_if_prereqs_missing(types.SimpleNamespace(run=False, ignore_prereqs=False)) + # and an explicit override starts anyway + pm._stop_if_prereqs_missing(types.SimpleNamespace(run=True, ignore_prereqs=True)) +# but a real run stops before the first package +try: + with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm._stop_if_prereqs_missing(types.SimpleNamespace(run=True, ignore_prereqs=False)) +except SystemExit as e: + assert e.code == 1, e.code else: - body = m.group(0) - enter = body.split('if action == "enter"') - if len(enter) < 2: - problems.append("chroot enter is gone") - elif "_normalize_sources" not in enter[1]: - problems.append("nothing normalises /sources on chroot entry") - prep = body.split('if action == "prepare"') - if len(prep) > 1 and "_normalize_sources" not in prep[1].split('if action ==')[0]: - problems.append("nothing normalises /sources after the 7.2 handover") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS /sources is given back to root on the way into the chroot") -PYSRCNORM -_count_rc $? + raise AssertionError("a run with a missing build tool was allowed to start") +assert "--ignore-prereqs" in buf.getvalue() +PYSTOP +# and the escape hatch is a real flag on the commands that build +for _cmd in install update; do + python3 "$_pmB" $_cmd --help 2>&1 | grep -q -- "--ignore-prereqs" \ + || _p="$_p;$_cmd has no --ignore-prereqs, so the stop cannot be overridden" +done +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a plan that cannot finish stops before it builds anything" +fi -# ---- a wrapper must never stand in for a privileged tool -------------------- # -# make_wrappers ships a `chgrp` and a `chown` that print a note and exit 0, so a -# PACKAGE USER cannot change ownership. They are ordinary files on $PATH. If -# that directory is ever ahead of /usr/bin for root, root's own privileged calls -# resolve to them too -- and they report success while doing nothing. +# ---- build the tool before the package that calls it (1.14.12) ------------- # +# Reported from a real chroot: a plan that CONTAINED CMake stopped at libaom +# with `cmake: command not found`. --recursive had worked -- CMake was in the +# plan at position 151 and libaom at 89. The resolver ordered on the book's +# dependency list, and libaom's page lists no Required at all, so nothing said +# libaom must wait for CMake. Right answer, wrong facts. +_p="" +_pmB="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmB" "$T/order" <<'PYORD' || _p="$_p;a package is not ordered after the tool its commands call" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +pm._tool_available = lambda t: False # nothing installed +def sc(n, body): + p = os.path.join(d, "install_" + n) + open(p, "w").write("build_pkg() {\n%s\n}\n" % body) + return p +steps = [("libaom", "libaom-3.13.1", "new", sc("libaom", "cmake -B build")), + ("curl", "cURL-8.18.0", "new", sc("curl", "./configure --prefix=/usr")), + ("cmake", "CMake-4.2.3", "new", sc("cmake", "./bootstrap --prefix=/usr")), + ("libheif", "libheif-1.21.2", "new", sc("libheif", "cmake --preset=rel"))] +out, moved = pm._order_by_build_tools(list(steps)) +order = [nv for _a, nv, _r, _p in out] +assert order.index("CMake-4.2.3") < order.index("libaom-3.13.1"), order +assert order.index("CMake-4.2.3") < order.index("libheif-1.21.2"), order +# the CONSUMER moves down, never the provider up: cmake's own dep stays ahead +assert order.index("cURL-8.18.0") < order.index("CMake-4.2.3"), order +assert moved, "nothing was reported as reordered" +# a tool already on the system needs no reordering at all +pm._tool_available = lambda t: True +out2, moved2 = pm._order_by_build_tools(list(steps)) +assert [nv for _a, nv, _r, _p in out2] == [nv for _a, nv, _r, _p in steps], out2 +assert moved2 == [], moved2 +# and a tool the plan BUILDS is not also reported as missing +pm._tool_available = lambda t: False +pm._MISSING_TOOLS.clear() +pm._warn_missing_build_tools("libaom", steps[0][3]) +assert "cmake" in pm._MISSING_TOOLS +in_plan = {a.lower() for a, _n, _r, _p in out} +for t in list(pm._MISSING_TOOLS): + if pm._BUILD_TOOLS[t].lower() in in_plan: + pm._MISSING_TOOLS.pop(t) +assert not pm._MISSING_TOOLS, pm._MISSING_TOOLS +PYORD +# the printed plan is the order that will run +_ci="$(sed -n '/Install plan ({len(plan)} to build)/,/for nv, after, tool in _moved/p' "$_pmB")" +_ord="$(printf '%s' "$_ci" | grep -n '_order_by_build_tools' | head -1 | cut -d: -f1)" +_prt="$(printf '%s' "$_ci" | grep -n 'reason:<10' | head -1 | cut -d: -f1)" +[ -n "$_ord" ] && [ -n "$_prt" ] && [ "$_ord" -lt "$_prt" ] \ + || _p="$_p;the plan is printed before it is ordered, so the list lies about what runs" +# and a plan that cannot finish does not start +grep -q '_stop_if_prereqs_missing(args)' "$_pmB" \ + || _p="$_p;a plan missing its prerequisites still starts" +grep -q 'ignore-prereqs' "$_pmB" \ + || _p="$_p;there is no way to start anyway" +# EVERY exit from resolve_script checks prerequisites -- the local copy in the +# package user's home returned before either check, which is every package +# that has been attempted once +[ "$(sed -n '/^def resolve_script(/,/^def _prewarm\|^# ------/p' "$_pmB" \ + | grep -c '_check_script_prereqs')" -ge 3 ] \ + || _p="$_p;a script taken from the package user's home is used unchecked" +# the stack delegates to the SAME install command, so everything above applies +# to `packagemanager stack <name> --run` too -- and a decision to push past a +# missing tool has to travel with it +_ivs="$(sed -n '/^def _install_via_self/,/^def /p' "$_pmB")" +printf '%s' "$_ivs" | grep -q '"install", pkg, "--run"' \ + || _p="$_p;a stack entry no longer goes through the one install command" +printf '%s' "$_ivs" | grep -q 'ignore_prereqs' \ + || _p="$_p;a stack cannot pass --ignore-prereqs to the install it delegates to" +sed -n '/kind == "book"/,/elif kind in/p' "$_pmB" | grep -q 'ignore_prereqs=getattr' \ + || _p="$_p;the stack does not forward its own --ignore-prereqs" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package waits for the tool its commands call, even when the book is silent" +fi + +# ---- name a package the book actually has (1.14.14) ------------------------ # +# Reported from a real chroot. The 1.14.9 hint said +# cargo needed by 3 package(s), first: rust-bindgen +# packagemanager install rustc --run --recursive +# and `rustc` is not an anchor in the book -- `rust` is. The command would +# have answered "not in the BLFS book". Same mistake as 1.14.10, one layer +# down: the tool->package map was written from memory instead of checked. +_p="" +_pmC="$(dirname "$LFS_TOOL")/packagemanager" +_bk14="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk14="$_c" && break +done +if [ -n "$_bk14" ]; then + BLFS_BOOK_FILE="$_bk14" python3 - "$_pmC" <<'PYPROV' || _p="$_p;a tool is mapped to a package the book does not have" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +# EVERY named provider must resolve in the book -- that is the whole point +bad = [] +for tool, prov in pm._BUILD_TOOLS.items(): + if not prov: + continue # deliberately none: an LFS tool + try: + if not pm.book_version(prov): + bad.append((tool, prov)) + except SystemExit: + bad.append((tool, prov)) +assert not bad, "these providers are not in the book: %r" % bad +# the one from the report +assert pm._provider_of("cargo") == "rust", pm._provider_of("cargo") +assert "install rust --run --recursive" in pm._install_cmd_for("cargo") +# a tool with no BLFS page gets NO command rather than a wrong one +assert pm._provider_of("meson") == "" +assert pm._install_cmd_for("meson") == "" +PYPROV +fi +# and a tool whose provider is not in the plan at all is ADDED, not just +# reported: reordering cannot help when the package is absent +grep -q 'return cmd_install(args, _replanned=True)' "$_pmC" \ + || _p="$_p;a missing provider is never added to the plan" +sed -n '/A TOOL THE PLAN DOES NOT HAVE AT ALL/,/return cmd_install/p' "$_pmC" \ + | grep -q 'not _replanned' \ + || _p="$_p;the re-plan is not bounded, so a missing provider could loop" +sed -n '/A TOOL THE PLAN DOES NOT HAVE AT ALL/,/return cmd_install/p' "$_pmC" \ + | grep -q 'getattr(args, "recursive", False)' \ + || _p="$_p;a non-recursive install silently grows extra packages" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every tool names a package the book has, or no command at all" +fi + +# ---- progress that is real should be visible (1.14.15) --------------------- # +# The user, after three runs of the same stack: "its like always starting from +# fresh". It was not -- the plan had gone 181 -> 151 -> 135, forty-six +# packages built -- but each run recomputes the whole tree and prints only +# what is LEFT, so three real advances read as three identical fresh starts. +_p="" +_pmD="$(dirname "$LFS_TOOL")/packagemanager" +_bkD="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkD="$_c" && break +done +if [ -n "$_bkD" ]; then + _o="$(BLFS_BOOK_FILE="$_bkD" timeout 600 python3 - "$_pmD" 2>&1 <<'PYPROG' +import sys, types, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +real = pm.gather_user +class Installed: + state = "installed"; version = None + def __init__(self, a): self.name = a +done = {"icu", "libuv", "libarchive", "nghttp2"} +pm.gather_user = lambda a: Installed(a) if a in done else real(a) +args = types.SimpleNamespace(packages=["libheif"], recursive=True, ignore="", + reinstall=False, yes=True, run=False, regenerate=False, local=None, + test=False, clean=False, ignore_recommended=False, optional=False, + ignore_prereqs=False, outdir=None, select=False) +try: + pm.cmd_install(args) +except SystemExit: + pass +PYPROG +)" + case "$_o" in + *"already installed -- they are skipped"*) ;; + *) _p="$_p;a re-run never says how much of the tree is already done" ;; + esac + # the count has to be the truth, not a guess: 4 of the 19 considered + case "$_o" in + *"4 of the 19 packages"*) ;; + *) _p="$_p;the already-installed count does not match the tree" ;; + esac + # and the plan itself shrinks by exactly those four + case "$_o" in + *"Install plan (15 to build)"*) ;; + *) _p="$_p;installed packages are not dropped from the plan" ;; + esac +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a re-run says how much of the tree it is skipping" +fi + +# ---- one file to edit, one to run (1.14.16) -------------------------------- # +# The user, reading a failure box: "im also irritated cause of the different +# files in script and staged, is it wanted that we have the p prefix?" +# No. One script had FOUR names in one home -- install_Hatchling-1.28.0 (the +# one you are told to edit), install_p_hatchling (the one that actually ran), +# install_hatchling (a staged copy of it) and install_last (the record) -- and +# every failure spent three lines explaining which was which. +_p="" +_pmE="$(dirname "$LFS_TOOL")/lfs-helper" +_stg="$(sed -n '/ONE FILE TO EDIT/,/^ local rc=0/p' "$_pmE")" +printf '%s' "$_stg" | grep -q 'cp "$install_script" "$home/install_$user_name"' \ + && _p="$_p;the redundant install_p_<user> copy is still written" +printf '%s' "$_stg" | grep -q 'rm -f "$home/install_$user_name"' \ + || _p="$_p;an old install_p_<user> copy is never cleaned up" +# ...but only when it is a copy: someone may have edited it +printf '%s' "$_stg" | grep -q 'cmp -s' \ + || _p="$_p;an edited install_p_<user> would be deleted without a word" +# the canonical file is the one that runs +grep -q 'cmd_build "$user_name" --script "$home/\$base"' "$_pmE" \ + || _p="$_p;the phased path still runs a copy instead of the editable script" +grep -q 'run_phase_as "$user_name" "$home/\$base"' "$_pmE" \ + || _p="$_p;the unphased path still runs a copy" +# and a script already in the owner's home is not staged a second time +sed -n '/ALREADY THERE IS ALREADY STAGED/,/elif ! /p' "$_pmE" | grep -q 'staged="$s"' \ + || _p="$_p;a script in the package user's home is still copied to a second name" +sed -n '/! \$name FAILED/,/auto-repair/p' "$_pmE" | grep -q '\[ "$staged" = "$s" \] ||' \ + || _p="$_p;the failure box prints a staged line even when there is no second file" +# a missing Python module is named, not left inside pip's traceback +_pl="$T/pylog"; mkdir -p "$_pl" +printf " import pathspec\nModuleNotFoundError: No module named 'pathspec'\nerror: metadata-generation-failed\n" \ + > "$_pl/log" +{ + echo 'set -u' + echo 'fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }' + echo 'detail() { :; }; hint() { :; }; show_build_errors() { :; }' + echo 'pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }' + echo 'phase_done() { return 1; }; SNAP_ROOT="/"' + python3 - "$_pmE" <<'PYREP2' +import sys +s = open(sys.argv[1]).read() +i = s.index(" # A FEATURE MISSING FROM AN INSTALLED PACKAGE.") +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('report() {\n local name="$1" phase="$2" log="$3" jobs=""\n' + body + '\n}') +PYREP2 +} > "$_pl/rep.sh" +_o="$(bash -c '. "'"$_pl"'/rep.sh"; report hatchling all "'"$_pl"'/log"' 2>&1)" +case "$_o" in + *"Python module 'pathspec'"*) ;; + *) _p="$_p;a ModuleNotFoundError does not name the module: $_o" ;; +esac +case "$_o" in + *"install pathspec --run --recursive"*) ;; + *) _p="$_p;no command to install the missing module" ;; +esac +case "$_o" in + *"built for an"*) ;; + *) _p="$_p;it does not mention the module may exist for an older Python" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "one script, one name in its home, and a missing module is named" +fi + +# ---- build a wheel, install nothing (1.14.16) ------------------------------ # +# Found from a user's `ls`: p_pathspec's manifest held ONE path, +# /var/mail/p_pathspec +# and nothing else. Two faults met. # -# A tree ran 34 packages that way: init-pkgusr said -# # 46 install directories now belong to group 'install' -# with an empty failure list, while every one was still root:root. No package -# user could write to /usr/bin, `verify --fix` could not repair it (its chown -# went through the same wrapper), and nothing said why. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - # the wrappers really do exit 0 without acting -- that is the hazard - _slice_fn "$helper_src" make_wrappers | grep -q 'package users cannot change ownership' \ - || _p="$_p;the chown wrapper no longer explains itself" - # so ownership and mode must go through the binary, by absolute path - for _fn in real_chown real_chgrp real_chmod; do - _slice_fn "$helper_src" "$_fn" | grep -q . \ - || _p="$_p;there is no $_fn -- a wrapper can shadow root's own calls" - done - _slice_fn "$helper_src" _real_tool | grep -q '/usr/bin/' \ - || _p="$_p;_real_tool does not resolve an absolute path" - # and no privileged pass may call the bare command any more - for _fn in cmd_init_pkgusr init_package_user_home _vfy_install_dirs ensure_pkgusr_roots; do - # only where the tool is INVOKED -- start of a command, or after - # &&/||/;/!. Matching it anywhere caught the warning text that explains - # the hazard, which would have made this test impossible to satisfy. - if _slice_fn "$helper_src" "$_fn" \ - | grep -vE '^[[:space:]]*#' \ - | grep -qE '(^[[:space:]]*|&&[[:space:]]*|\|\|[[:space:]]*|;[[:space:]]*|![[:space:]]*)(chown|chgrp|chmod) '; then - _p="$_p;$_fn still calls a bare chown/chgrp/chmod" - fi - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "root's ownership calls cannot resolve to a package-user wrapper" - fi +# (1) `_comment_orphan_pip_installs` comments a pip install whose --find-links +# directory nothing builds -- right for brotli, whose wheel step is +# optional and commented out. It judged each block ALONE, and for a +# Python module the wheel is built in the BUILD phase and installed in the +# INSTALL phase, so it never saw the producer and commented the only line +# that installs anything. All 76 module pages: build a wheel, install +# nothing. +# (2) useradd creates /var/mail/<user>, owned by the package user, so the +# manifest scan picked it up -- and one existing path is the evidence +# `state` accepts for "installed". A package that installed NOTHING +# looked installed, the planner skipped it, and Hatchling died on +# `No module named 'pathspec'` twenty-two packages later. +_p="" +_bkF="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkF="$_c" && break +done +if [ -n "$_bkF" ]; then + _wd="$T/wheels"; mkdir -p "$_wd" + python3 "$(dirname "$LFS_TOOL")/blfs" --book-file "$_bkF" \ + script pathspec brotli -o "$_wd" >/dev/null 2>&1 + _ps="$(ls "$_wd"/install_[Pp]athspec* 2>/dev/null | head -1)" + if [ -n "$_ps" ]; then + sed -n '/^install_pkg() {/,/^}/p' "$_ps" | grep -q '^pip3 install' \ + || _p="$_p;a python module page still installs nothing" + sed -n '/^build_pkg() {/,/^}/p' "$_ps" | grep -q 'pip3 wheel' \ + || _p="$_p;the wheel is no longer built" + else + _p="$_p;could not generate the pathspec script" + fi + _br="$(ls "$_wd"/install_[Bb]rotli* 2>/dev/null | head -1)" + if [ -n "$_br" ]; then + grep -q "SKIPPED -- installs from 'dist'" "$_br" \ + || _p="$_p;brotli's orphan pip install is no longer guarded" + fi +fi +_pmF="$(dirname "$LFS_TOOL")/packagemanager" +sed -n '/^_write_pkg_lst() {/,/^}/p' "$helper_src" | grep -q 'var/mail/\*' \ + || _p="$_p;useradd's mail spool can still stand in for an installed package" +# ...and every pkg.lst ALREADY on disk contains it, so the reader has to skip +# it too -- excluding it only from new records left pathspec skipped again on +# the very next run +python3 - "$_pmF" <<'PYMAIL' || _p="$_p;an old manifest of nothing but a mail spool still reads as installed" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = tempfile.mkdtemp(); ml = os.path.join(d, "pkg.lst") +u = pm.User.__new__(pm.User); u.pkg_list = ml +open(ml, "w").write("/var/mail/p_pathspec\n") +assert pm.User._manifest_has_real_paths(u) is False, "mail spool alone counted" +open(ml, "w").write("/var/mail/p_x\n/usr\n/etc\n") +assert pm.User._manifest_has_real_paths(u) is True, "real paths were discarded" +PYMAIL +sed -n '/def _declares_content_that_is_missing/,/def _manifest_has_real_paths/p' "$_pmF" \ + | grep -q 'installed_directories' \ + || _p="$_p;the page's own installed paths are never checked" +python3 - "$_pmF" <<'PYST' || _p="$_p;a package whose declared paths are all missing still counts as installed" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +u = pm.User.__new__(pm.User) +u.meta = {"installed_directories": "/nonexistent/site-packages/pathspec"} +assert pm.User._declares_content_that_is_missing(u) is True +u.meta = {"installed_directories": "/usr /nonexistent/x"} +assert pm.User._declares_content_that_is_missing(u) is False +u.meta = {} +assert pm.User._declares_content_that_is_missing(u) is False +PYST +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a python module installs itself, and an empty install cannot look installed" +fi - # exit 0 is not proof. init-pkgusr must read the group back. - _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" - _p="" - # owner AND group: the rule is root:install, and reading back only the group - # is how /usr/share sat at lfs:install without anything noticing - case "$_blk" in - *'stat -c %U:%G'*) ;; - *) _p="$_p;init-pkgusr trusts the exit code instead of reading the result back" ;; - esac - case "$_blk" in - *'NO install directories'*) ;; - *) _p="$_p;init-pkgusr reports zero install directories as success" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "init-pkgusr confirms the install group took, and zero is an error" - fi +# ---- ask the book once, not once per package (1.14.18) --------------------- # +# "why is the lookup for the script taking that long? arent we somehow indexing +# all the package names?" There IS an index -- book_versions_all() reads the +# whole book in one call -- and only `update` used it. book_version() ran +# `blfs debug <anchor>`: a subprocess, opening and reading the book cache, to +# answer ONE lookup. A sway stack preview asks 56 of them and the planner asks +# one per package considered. Measured on 26 names: 7.07s of subprocesses +# against 0.79s for the map, and every lookup after the first free. +_p="" +_pmG="$(dirname "$LFS_TOOL")/packagemanager" +_bkG="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkG="$_c" && break +done +if [ -n "$_bkG" ]; then + BLFS_BOOK_FILE="$_bkG" python3 - "$_pmG" <<'PYIDX' || _p="$_p;the book is still asked one process per package" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +calls = [] +real = pm.run_blfs +def counting(argv): + calls.append(list(argv)) + return real(argv) +pm.run_blfs = counting +names = ["dbus", "curl", "mesa", "pango", "gtk4", "cmake", "xmlto", "alsa-lib"] +got = [pm.book_version(n) for n in names] +assert all(got), dict(zip(names, got)) +# ONE process for all of them -- the whole-book map +assert len(calls) == 1, calls +assert calls[0][0] == "versions", calls +# and nothing at all the second time +before = len(calls) +[pm.book_version(n) for n in names] +assert len(calls) == before, calls[before:] +# a name the map does not have still gets its own answer, and is remembered +pm.book_version("this-is-not-a-package-at-all") +assert len(calls) == before + 1, calls[before:] +PYIDX +fi +grep -q 'def book_versions_all' "$_pmG" \ + || _p="$_p;the whole-book version map is gone" +sed -n '/^def book_version(/,/^def /p' "$_pmG" | grep -q 'book_versions_all()' \ + || _p="$_p;book_version does not use the map" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the book is read once and answers every lookup from memory" fi -# ---- an empty override must not disable the whole scheme -------------------- # -# install_dirs_list is overridable through $INSTALLDIRS and returned early when -# the file existed. An override yielding nothing meant NO directory was given -# to the install group, NO directory was protected from adoption, and `verify` -# reported a clean tree because it walked the same empty list. Nine callers, -# one silent no-op. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( STATE=/tmp/_idl.$$; INSTALLDIRS="$STATE/installdirs.lst" - mkdir -p "$STATE" - eval "$(_slice_fn "$helper_src" install_dirs_list)" - warn() { :; } - # no override at all -> the built-in list - [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ - || echo "the built-in list is empty" - # an override with only comments and blanks -> fall back, never nothing - printf '# just a comment\n\n \n' > "$INSTALLDIRS" - [ "$(install_dirs_list 2>/dev/null | grep -c .)" -gt 10 ] \ - || echo "an empty override silently disables every install directory" - # a real override is still honoured - printf '/usr/bin\n/usr/lib\n' > "$INSTALLDIRS" - [ "$(install_dirs_list 2>/dev/null | grep -c .)" = 2 ] \ - || echo "a real override is no longer honoured" - rm -rf "$STATE" )" - if [ -z "$_o" ]; then - ok "an override that lists nothing falls back to the built-in list" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- a typo in a group name is permanent (1.14.19) ------------------------- # +# Reported from a real build: the collector-group prompt offered nimgnu_gcc or +# nimgnu_lib, the user typed their own name and wrote `nimnu_gdb` -- one letter +# short of `nimgnu_`. It is a valid group name, so it was created and the +# directory handed to it, and nothing that looks for ${COLLECTOR_PREFIX}_* +# would ever see it again. +_p="" +_hlp="$helper_src" +# a near miss for the prefix is offered as a correction; a deliberate name is not +_nm="$T/nearmiss.sh"; _slice_fn "$_hlp" _near_miss > "$_nm" +_near_check() { # <a> <b> <expected: yes|no> + if ( . "$_nm"; _near_miss "$1" "$2" ); then _r=yes; else _r=no; fi + [ "$_r" = "$3" ] || _p="$_p;_near_miss $1 $2 said $_r, expected $3" +} +_near_check nimnu nimgnu yes # the reported typo: one letter missing +_near_check nmgnu nimgnu yes # one letter missing, elsewhere +_near_check nimgnux nimgnu yes # one letter too many +_near_check nimgno nimgnu yes # one letter wrong +_near_check nimgnu nimgnu no # already right: nothing to offer +_near_check gcc nimgnu no # a deliberate name, left alone +_near_check shared nimgnu no +# the prompt actually consults it, and only for names that have a prefix part +_pick="$(sed -n '/A TYPO IN A GROUP NAME IS PERMANENT/,/using .\$picked./p' "$_hlp")" +printf '%s' "$_pick" | grep -q '_near_miss "$_head" "$_pfx"' \ + || _p="$_p;the prompt does not check a typed name against the collector prefix" +# (the fallback is "sysgroup" since 1.14.35 -- the same default lfs and +# packagemanager use; it was the one place that said nimgnu) +printf '%s' "$_pick" | grep -q 'COLLECTOR_PREFIX:-sysgroup' \ + || _p="$_p;the prompt hardcodes a prefix instead of using the configured one" +printf '%s' "$_pick" | grep -q '\[Y/n\]' \ + || _p="$_p;the correction is applied without asking" +# and the ones already made can be repaired: renaming keeps the gid, so every +# directory already granted stays granted +grep -q 'rename-group) shift; cmd_rename_group' "$_hlp" \ + || _p="$_p;there is no way to fix a group that was already misnamed" +_rg="$(_slice_fn "$_hlp" cmd_rename_group)" +printf '%s' "$_rg" | grep -q 'groupmod' \ + || _p="$_p;rename-group does not rename the group" +printf '%s' "$_rg" | grep -q 'group_exists "$new" && die' \ + || _p="$_p;rename-group would collide with an existing group" +printf '%s' "$_rg" | grep -q 'dry run' \ + || _p="$_p;rename-group has no dry run" +if command -v groupadd >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then + _g1="lfstest_nimnu_$$"; _g2="lfstest_nimgnu_$$" + if groupadd "$_g1" 2>/dev/null; then + _gd="$T/granted"; mkdir -p "$_gd"; chgrp "$_g1" "$_gd" 2>/dev/null + _gid_before="$(getent group "$_g1" | cut -d: -f3)" + SNAP_ROOT="$T" bash "$_hlp" rename-group "$_g1" "$_g2" --run >/dev/null 2>&1 + _gid_after="$(getent group "$_g2" | cut -d: -f3)" + [ -n "$_gid_after" ] || _p="$_p;rename-group did not create the new name" + [ "$_gid_before" = "$_gid_after" ] \ + || _p="$_p;the gid changed, so every directory already granted lost its group" + [ "$(stat -c %G "$_gd" 2>/dev/null)" = "$_g2" ] \ + || _p="$_p;a directory the group owned did not follow the rename" + groupdel "$_g2" 2>/dev/null || groupdel "$_g1" 2>/dev/null + fi +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a mistyped collector prefix is caught, and an old one can be renamed" +fi + +# ---- not every name is an LFS step (1.14.20) ------------------------------- # +# Reported from a real chroot. The failure box's own hint said +# lfs-helper build p_glib2 --phase all --force +# and a session note said `lfs-helper build glib2 --phase install --force`. +# glib2 is a BLFS package: there is no $SCRIPTS/glib2.sh and never will be. +# cmd_build looked only at the step scripts and answered +# !! no script for 'glib2' ... +# From outside the chroot: lfs build-system gen-chroot-scripts --run +# -- advice for a different problem, naming a command that REFUSES to run +# inside the chroot the person is standing in. ("that hint was dumb.") +_p="" +_hlpB="$helper_src" +# a name that is not a step is recognised as such +_so="$T/steporder"; mkdir -p "$_so" +printf 'binutils\ngcc\n' > "$_so/steporder" +_cls="$(SCRIPTS="$_so" STATE_PROGRESS="$_so" bash -c ' + . <(sed -n "/^steps() {/,/^}/p" "'"$_hlpB"'") + . <(sed -n "/^in_step_order() {/,/^}/p" "'"$_hlpB"'") + for n in gcc glib2; do + in_step_order "$n" && echo "$n=step" || echo "$n=not" + done')" +case "$_cls" in + *"gcc=step"*) ;; + *) _p="$_p;a real LFS step is no longer recognised as one" ;; +esac +case "$_cls" in + *"glib2=not"*) ;; + *) _p="$_p;a BLFS package is mistaken for an LFS step" ;; +esac +# ...and gets the answer that fits: build it through packagemanager +_die="$(sed -n '/ONE OF THREE THINGS/,/die "\$_msg"/p' "$_hlpB")" +printf '%s' "$_die" | grep -q 'packagemanager install $name --run --recursive' \ + || _p="$_p;an unknown name is not pointed at packagemanager" +printf '%s' "$_die" | grep -q 'packagemanager script install $name' \ + || _p="$_p;there is no way offered to re-run one phase" +printf '%s' "$_die" | grep -q 'in_step_order "$name"' \ + || _p="$_p;the message does not depend on what the name actually is" +# the gen-chroot-scripts advice survives ONLY for a real step +_stepbranch="$(printf '%s' "$_die" | sed -n '/if in_step_order/,/else/p')" +printf '%s' "$_stepbranch" | grep -q 'gen-chroot-scripts' \ + || _p="$_p;a genuinely missing step script no longer says how to generate it" +_elsebranch="$(printf '%s' "$_die" | sed -n '/^ else$/,/^ fi$/p')" +printf '%s' "$_elsebranch" | grep -q 'gen-chroot-scripts' \ + && _p="$_p;a BLFS package is still told to regenerate the chroot scripts" +# before any of that: a package user's OWN script is used when there is one +_fb="$(sed -n '/NOT EVERY NAME IS AN LFS STEP/,/^ fi$/p' "$_hlpB")" +printf '%s' "$_fb" | grep -q 'pkg_owner_name "$name"' \ + || _p="$_p;the package user's own script is never looked for" +printf '%s' "$_fb" | grep -q 'install_last' \ + || _p="$_p;there is no fallback to the record of what last ran" +printf '%s' "$_fb" | grep -q "grep -v '\\\\.edited" \ + || _p="$_p;an .edited backup could be run instead of the script" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a BLFS package builds through its own script, not an LFS step message" fi -# ---- an option is not a package name ---------------------------------------- # -# /usr/src/pkgusr/p_--version turned up in a sanity run: a home for an account -# that could never exist, left by something that reached add-user with -# "--version" in the name position. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _slice_fn "$helper_src" _refuse_option_as_name | grep -q 'looks like an option' \ - && case "$(_slice_fn "$helper_src" cmd_add_user)" in - *_refuse_option_as_name*) ok "a name starting with a dash is refused, not made into a user" ;; - *) bad "add-user does not refuse an option-shaped name" ;; - esac \ - || bad "there is no guard against an option being used as a package name" +# ---- one door for every build (1.14.21) ------------------------------------ # +# Reported from a real chroot. glib2's install stopped on +# PermissionError: [Errno 13] '/usr/share/gettext/its' +# Through lfs-helper that failure is REPAIRED: the grant-and-retry offers a +# collector group for the directory and runs the phase again. Through +# `packagemanager script install glib2` -- the command the failure box itself +# tells people to run -- the identical traceback repeated forever, because that +# path built its own `su` and had none of the wrappers, tracking, failure box +# or repair around it. Second bug from the same split (the first: the build +# environment, 1.14.7). +_p="" +_pmH="$(dirname "$LFS_TOOL")/packagemanager" +_sd="$(sed -n '/THE SAME DOOR AS EVERY OTHER BUILD/,/printInfo(f" \$ su/p' "$_pmH")" +printf '%s' "$_sd" | grep -q 'shutil.which("lfs-helper")' \ + || _p="$_p;the script path does not look for lfs-helper" +printf '%s' "$_sd" | grep -q '"build", name, "--script", home_script' \ + || _p="$_p;a phase is still run without the wrappers and the repair" +printf '%s' "$_sd" | grep -q 'phase != "fetch"' \ + || _p="$_p;the fetch phase, which runs nothing, is sent through the builder" +printf '%s' "$_sd" | grep -q 'else:' \ + || _p="$_p;there is no fallback when lfs-helper is absent" +# and the file it runs is the one the user is told to edit +grep -q '_versioned_script_in(home)' "$_pmH" \ + || _p="$_p;the script path still runs install_<user> instead of the canonical file" +python3 - "$_pmH" "$T/canon" <<'PYCANON' || _p="$_p;the canonical script is not found or not preferred" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +h = sys.argv[2]; os.makedirs(h, exist_ok=True) +for f in ("install_GLib-2.86.4", "install_p_glib2"): + open(os.path.join(h, f), "w").write("x") +got = pm._versioned_script_in(h) +assert got and got.endswith("install_GLib-2.86.4"), got +PYCANON +# the message about a regenerated script has to describe what happened +grep -q 'is never regenerated)' "$_pmH" \ + && _p="$_p;it still claims the file it just rewrote is never regenerated" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "packagemanager script runs a phase through the same door as every build" fi -# ---- the sanity report checks the group it prints --------------------------- # -# Section 1 printed owner:group from the start and asserted only the mode. A -# tree ran 34 packages with every install directory at root:root 775 -- -# group-writable, but group `root`, while package users are in `install`. The -# column that would have shown it was decoration. -san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" -if [ -f "$san" ]; then - _p="" - grep -q "want 'install'" "$san" \ - || _p="$_p;the sanity report does not check the install group" - grep -q "a package has taken a shared directory" "$san" \ - || _p="$_p;the sanity report does not check who owns a shared directory" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the sanity report asserts the install-dir owner and group" - fi +# ---- an undeclared symbol usually belongs to someone else (1.14.22) -------- # +# cairo stopped with twelve of these: +# error: 'FC_HINT_NONE' undeclared (first use in this function) +# error: 'FC_RGBA_RGB' undeclared ... +# all of them fontconfig's -- a DEPENDENCY whose headers are missing or too +# old, not a bug in cairo. The box said "this is a COMPILE error ... the tools +# cannot repair it" and stopped there, leaving the person to work out whose +# symbols those were. The tools can look: either some installed header +# declares it (an include path or .pc problem) or none does (the package that +# provides it is not properly installed). +_p="" +_hlpC="$helper_src" +_cr="$T/compile"; mkdir -p "$_cr" +# build the whole failure report, not the tail of it -- the earlier harness +# sliced from a marker halfway down and never covered this branch at all +python3 - "$_hlpC" > "$_cr/rep.sh" <<'PYSLICE' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/"; s="/x"; staged="/x"; auto_repair=1') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE +bash -n "$_cr/rep.sh" || _p="$_p;the failure report does not parse when sliced whole" +# a symbol that IS declared somewhere: name the header and the package +_hdr="" +for _c in /usr/include/stdio.h /usr/include/stdlib.h; do + [ -f "$_c" ] && _hdr="$_c" && break +done +printf "../src/x.c:56:14: error: 'SEEK_SET' undeclared (first use in this function)\n" \ + > "$_cr/known.log" +if [ -n "$_hdr" ]; then + _o="$(bash -c '. "'"$_cr"'/rep.sh"; report demo all "'"$_cr"'/known.log"' 2>&1)" + case "$_o" in + *"IS declared, in:"*) ;; + *) _p="$_p;a symbol that exists in an installed header is not located" ;; + esac fi - -# ---- one wrapper directory, not two ----------------------------------------- # -# make_wrappers wrote to $STATE/wrappers while /etc/pkgusr/bash_profile -- and -# packagemanager's copy of it -- put /usr/lib/pkgusr first on a package user's -# PATH. Nothing created that, so a package user's login shell led with a -# directory that did not exist, and the wrappers only ever applied because -# cmd_build overrides PATH on the command line. Two names for one decision. -# -# /usr/lib/pkgusr is the survivor: $STATE is build state, and packagemanager on -# the finished system hands its users the same profile. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - grep -q '^WRAPPERS="\${LFS_WRAPPERS:-\${SNAP_ROOT%/}/usr/lib/pkgusr}"' "$helper_src" \ - || _p="$_p;the wrapper directory is not /usr/lib/pkgusr" - grep -q 'WRAPPERS="\$STATE/wrappers"' "$helper_src" \ - && _p="$_p;the old \$STATE/wrappers definition is still there" - # the profile must not name the directory a second time - case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in - *'PATH=/usr/lib/pkgusr:'*) - _p="$_p;the profile still hardcodes the wrapper path instead of using \$WRAPPERS" ;; - esac - case "$(_slice_fn "$helper_src" ensure_pkgusr_etc)" in - *'s|@WRAPPERS@|$WRAPPERS|'*) ;; - *) _p="$_p;the profile's PATH is not filled in from \$WRAPPERS" ;; - esac - # and it must be protected: under /usr/lib, which IS an install dir, so - # without this any package user could rewrite the wrappers that constrain it - case "$(_slice_fn "$helper_src" never_adopt_list)" in - *'"$WRAPPERS"'*) ;; - *) _p="$_p;the wrapper directory can be adopted or given to the install group" ;; - esac - case "$(_slice_fn "$helper_src" make_wrappers)" in - *'chown root:root "$WRAPPERS"'*) ;; - *) _p="$_p;the wrappers are not explicitly root-owned" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "there is one wrapper directory, and it outlives the build" - fi +# a symbol nothing declares: say the providing package is missing +printf "../src/x.c:9:5: error: 'FC_MADE_UP_SYMBOL_XYZ' undeclared (first use in this function)\n" \ + > "$_cr/unknown.log" +_o2="$(bash -c '. "'"$_cr"'/rep.sh"; report demo all "'"$_cr"'/unknown.log"' 2>&1)" +case "$_o2" in + *"Nothing under /usr/include declares"*) ;; + *) _p="$_p;an undeclared symbol nothing provides is not diagnosed" ;; +esac +case "$_o2" in + *"which-package FC_MADE_UP_SYMBOL_XYZ"*) ;; + *) _p="$_p;no way offered to find which package should provide it" ;; +esac +# the machine.conf advice, which is right for a genuine compile error, stays +case "$_o2" in + *"machine.conf"*) ;; + *) _p="$_p;the per-package build-flag advice was lost" ;; +esac +# the search must be bounded: a failure report cannot hang the build +_sd2="$(sed -n '/AN UNDECLARED SYMBOL USUALLY BELONGS/,/^ fi$/p' "$_hlpC")" +printf '%s' "$_sd2" | grep -q 'timeout ' \ + || _p="$_p;the header search is unbounded and could hang the report" +printf '%s' "$_sd2" | grep -q 'grep -rlm1' \ + || _p="$_p;the header search reads every match instead of stopping at one" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an undeclared symbol is traced to the header, or to the package that lacks it" fi -# packagemanager builds the same profile for the finished system, so it must -# reach the same path -- a second hardcoded copy is how these drifted -pm_src="$(dirname "$LFS_TOOL")/packagemanager" -if [ -f "$pm_src" ]; then - _p="" - grep -q '^WRAPPERS = os.environ.get("LFS_WRAPPERS", "/usr/lib/pkgusr")' "$pm_src" \ - || _p="$_p;packagemanager has no WRAPPERS setting" - grep -q 'PATH=/usr/lib/pkgusr:' "$pm_src" \ - && _p="$_p;packagemanager still hardcodes the wrapper path in the profile" - grep -q '_PKGUSR_BASH_PROFILE.replace("@WRAPPERS@", WRAPPERS)' "$pm_src" \ - || _p="$_p;packagemanager does not fill the wrapper path into the profile" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "both tools write a profile pointing at the same wrapper directory" - fi +# ---- a commented line is not a command (1.14.23) --------------------------- # +# A plan refused to start: +# JT_JAVA needed by 1 package(s), first: openjdk +# packagemanager env --set JT_JAVA=... +# JT_JAVA appears on OpenJDK's page ONLY inside the test block -- which the +# generated script comments out -- and the page EXPORTS it there itself: +# # export JT_JAVA=$(echo $PWD/build/*/jdk) && +# # jtreg/bin/jtreg -jdk:$JT_JAVA ... +# So the guard demanded a value for a variable nothing live would read, and +# the person was asked to invent one. +_p="" +_bkH="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkH="$_c" && break +done +if [ -n "$_bkH" ]; then + _gd="$T/guards"; mkdir -p "$_gd" + python3 "$(dirname "$LFS_TOOL")/blfs" --book-file "$_bkH" \ + script openjdk xorg-server texlive -o "$_gd" >/dev/null 2>&1 + _oj="$(ls "$_gd"/install_OpenJDK* 2>/dev/null | head -1)" + if [ -n "$_oj" ]; then + grep -q 'needs JT_JAVA to be set' "$_oj" \ + && _p="$_p;a variable used only in a commented-out test block is still demanded" + # the commented line itself must survive: it is the book's own text + grep -q 'JT_JAVA' "$_oj" \ + || _p="$_p;the page's commented test commands were dropped entirely" + else + _p="$_p;could not generate the openjdk script" + fi + # a variable the user really must provide is still guarded + _xs="$(ls "$_gd"/install_Xorg-Server* 2>/dev/null | head -1)" + if [ -n "$_xs" ]; then + grep -q 'needs XORG_PREFIX to be set' "$_xs" \ + || _p="$_p;XORG_PREFIX, which the user must set, is no longer guarded" + fi + # ...and one the page sets ITSELF is not asked for + _tl="$(ls "$_gd"/install_texlive* 2>/dev/null | head -1)" + if [ -n "$_tl" ]; then + grep -q 'needs TEXLIVE_PREFIX to be set' "$_tl" \ + && _p="$_p;a variable the book assigns itself is demanded from the user" + grep -q 'TEXLIVE_PREFIX=' "$_tl" \ + || _p="$_p;the book's own assignment was lost" + fi +fi +# both rules, in the generator +_gf="$(sed -n '/^def _required_env_guard/,/^def /p' "$(dirname "$LFS_TOOL")/blfs")" +printf '%s' "$_gf" | grep -q 'A COMMENTED LINE IS NOT A COMMAND' \ + || _p="$_p;the guard still scans commented-out commands" +printf '%s' "$_gf" | grep -q "startswith(\"#\")" \ + || _p="$_p;comments are not filtered before scanning" +printf '%s' "$_gf" | grep -q 'the page is telling us its value' \ + || _p="$_p;a variable the commands assign is still asked for" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "only variables live commands read, and the book does not set, are demanded" fi -# ---- an install directory is root:install, owner AND group ------------------- # -# init-pkgusr set only the GROUP, while _vfy_install_dirs demands root:install -# and `verify --fix` applies both. Two rules for one directory. +# ---- an avoided package is not walked into (1.14.24) ----------------------- # +# From a real plan: cups, Java, OpenJDK and apache-ant were queued, each +# labelled "required by libreoffice" -- with libreoffice in --ignore. Ignoring +# dropped the NAMED package and kept everything it alone wanted, so a sway +# desktop was about to build a JDK, and stopped on a JT_JAVA guard for it. +# sway.stack's 80-name avoid list was the workaround for this. # -# They disagreed on a real build: chapters 5-6 build as the `lfs` user, and book -# 7.2's handover only reaches one level below $LFS, so /usr/share stayed -# lfs-owned and came out -# lfs:install drwxrwxr-x /usr/share -# The group looked right and nothing said the owner was not. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _blk="$(_slice_fn "$helper_src" cmd_init_pkgusr)" - _p="" - # owner AND group -- through the shared rule, which applies both by number. - # This used to look for `real_chown root:install`, which is the by-name form - # that cannot work before book 7.6 writes /etc/passwd. - case "$_blk" in - *'set_install_dir_owner "$d"'*) ;; - *) _p="$_p;init-pkgusr sets the group but not the owner of an install directory" ;; - esac - case "$_blk" in - *'stat -c %U:%G'*) ;; - *) _p="$_p;init-pkgusr reads back only the group, so a wrong owner goes unnoticed" ;; - esac - # the two rules must want the same thing - case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in - *root:install*) ;; - *) _p="$_p;verify no longer wants root:install" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "init-pkgusr and verify agree that an install dir is root:install" - fi +# 1.12.82 pruned on the finished list and was reverted: `blfs order` reports +# ONE parent per package, so graphite2 -- recorded under an ignored parent, +# genuinely needed by harfbuzz -- was dropped and the build failed with +# "Dependency 'graphite2' is required but not found". The prune belongs in +# the WALK, where every edge is still known. +_p="" +_b24="$(dirname "$LFS_TOOL")/blfs" +_bk24="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk24="$_c" && break +done +if [ -n "$_bk24" ]; then + # poppler/qt6: 185 -> 61. (This used harfbuzz/libreoffice until 1.14.31 + # removed that edge -- it came from a parenthetical aside and was never + # real. A test needs a relationship the book actually states.) + _all="$(python3 "$_b24" --book-file "$_bk24" order poppler --anchors 2>/dev/null | wc -l)" + _cut="$(python3 "$_b24" --book-file "$_bk24" order poppler --anchors \ + --ignore qt6 2>/dev/null)" + _ncut="$(printf '%s\n' "$_cut" | grep -c .)" + [ "${_all:-0}" -gt 100 ] || _p="$_p;the unpruned order looks wrong ($_all)" + [ "${_ncut:-0}" -lt "${_all:-0}" ] \ + || _p="$_p;ignoring a package pruned nothing ($_ncut of $_all)" + # what libreoffice ALONE wanted is gone + for _gone in qt6 vlc phonon phonon-backend-vlc; do + printf '%s\n' "$_cut" | cut -f1 | grep -qix "$_gone" \ + && _p="$_p;$_gone survived although only the ignored package wanted it" + done + # ...and what harfbuzz itself needs is NOT: this is the 1.12.82 regression + for _keep in cairo fontconfig glib2 libpng boost; do + printf '%s\n' "$_cut" | cut -f1 | grep -qix "$_keep" \ + || _p="$_p;$_keep was pruned although another package needs it" + done + # the ignored package itself is not built + printf '%s\n' "$_cut" | cut -f1 | grep -qix qt6 \ + && _p="$_p;the ignored package is in the order" + # asking for an ignored package BY NAME still works: it is the target + python3 "$_b24" --book-file "$_bk24" order qt6 --anchors \ + --ignore qt6 2>/dev/null | cut -f1 | grep -qix qt6 \ + || _p="$_p;a package cannot be built even when asked for by name" +fi +# the walker takes the list, and packagemanager hands it over +sed -n '/^def walk_deps/,/^def /p' "$_b24" | grep -q 'ignore=()' \ + || _p="$_p;the dependency walk does not take an ignore list" +sed -n '/^def walk_deps/,/^def /p' "$_b24" | grep -q 'a.lower() in ignore and a != anchor' \ + || _p="$_p;the walk still descends into an ignored package" +sed -n '/^def blfs_order_anchors/,/^def /p' "$(dirname "$LFS_TOOL")/packagemanager" \ + | grep -q '"--ignore"' \ + || _p="$_p;packagemanager never passes the ignore list to the walk" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an avoided package takes what only it wanted, and nothing else" fi -# ---- every tool we install gets an owner ------------------------------------- # -# install_helper copies lfs/lfs-helper/packagemanager/blfs and calls -# _chown_tools. _install_pkgusr_helpers copied the hint's scripts beside them -# and never did -- so /usr/bin/list_package, uninstall_package, -# forall_direntries_from and the rest came out root:root while the four tools -# next to them belonged to p_pkgusr. Two copy routines, one claiming ownership. -# -# The WRAPPER directory is the deliberate exception: it stays root:root, because -# a package user that could rewrite chown or install could rewrite the rule that -# constrains it. -python3 - "$LFS_TOOL" <<'PYHELPEROWN' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -m = re.search(r"^def _install_pkgusr_helpers\(.*?(?=\n\ndef )", src, re.S | re.M) -if not m: - problems.append("_install_pkgusr_helpers is gone") -else: - body = m.group(0) - if "_chown_tools" not in body: - problems.append("the hint's helpers are copied but never given an owner") - if "wrapper_sub" not in body: - problems.append("the wrapper directory is not excluded from the chown") -if not re.search(r'^WRAPPERS = os\.environ\.get\("LFS_WRAPPERS"', src, re.M): - problems.append("lfs does not read the shared WRAPPERS setting") -# and the helper table must not spell the wrapper path out a fourth time -tbl = re.search(r"PKGUSR_HELPERS = \[(.*?)\n\]", src, re.S) -if tbl and '"usr/lib/pkgusr"' in tbl.group(1): - problems.append("PKGUSR_HELPERS still hardcodes the wrapper path") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS every tool installed into the tree is given an owner") -PYHELPEROWN -_count_rc $? +# ---- where the page's own commands left off (1.14.25) ---------------------- # +# Xorg Libraries downloaded all thirty-two tarballs, verified every checksum, +# and then: +# the list this build loops over is missing or empty: ../lib-7.md5 +# Its prepare step writes lib-7.md5 in the build root, then `mkdir lib && cd +# lib` and downloads into it -- and the build loop reads `../lib-7.md5`, which +# only resolves from INSIDE lib. _pp_unpack ran prepare_pkg and then `cd +# "$BUILD_ROOT"`, throwing that away, so the build began one level too high. +_p="" +_mt="$T/meta"; mkdir -p "$_mt/src" +{ + echo '#!/bin/bash' + echo 'name_version="meta-1.0"' + echo 'prepare_pkg() {' + echo 'cat > lib-7.md5 << "XEOF"' + echo 'd41d8cd98f00b204e9800998ecf8427e a.txt' + echo 'XEOF' + echo 'mkdir -p lib &&' + echo 'cd lib &&' + echo 'echo hello > a.txt' + echo '}' + echo 'build_pkg() {' + echo '[ -s "../lib-7.md5" ] || { echo "MISSING LIST" >&2; exit 1; }' + echo 'echo "BUILT-IN:$PWD"' + echo '}' + echo 'install_pkg() { echo "INSTALLED-IN:$PWD"; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_mt/install_meta" +_o="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_mt/src" \ + bash "$_mt/install_meta" all 2>&1)" +case "$_o" in + *"MISSING LIST"*) _p="$_p;the build no longer resumes where the page's commands ended" ;; +esac +case "$_o" in + *"BUILT-IN:$_mt/src/lib"*) ;; + *) _p="$_p;the build phase did not resume inside the page's own directory: $_o" ;; +esac +# ...and so does every phase after it, not just the build +case "$_o" in + *"INSTALLED-IN:$_mt/src/lib"*) ;; + *) _p="$_p;the install phase started somewhere else again" ;; +esac +# a package with a real tarball is unaffected: it enters its unpacked source +_mt2="$T/meta2"; mkdir -p "$_mt2/src" "$_mt2/sources" "$_mt2/stage/demo-5.0" +printf 'x\n' > "$_mt2/stage/demo-5.0/file" +( cd "$_mt2/stage" && tar czf ../sources/demo-5.0.tar.gz demo-5.0 ) 2>/dev/null +{ + echo 'name_version="demo-5.0"' + echo 'pkg_glob="demo-5.0.tar.*"' + echo 'prepare_pkg() { mkdir -p elsewhere && cd elsewhere; }' + echo 'build_pkg() { echo "BUILT-IN:$PWD"; }' + echo 'install_pkg() { :; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_mt2/install_demo" +_o2="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_mt2/src" \ + LFS_SOURCES_DIR="$_mt2/sources" bash "$_mt2/install_demo" all 2>&1)" +case "$_o2" in + *"BUILT-IN:$_mt2/src/demo-5.0"*) ;; + *) _p="$_p;a package with a tarball no longer builds in its unpacked source: $_o2" ;; +esac +sed -n '/WHERE THE PAGE.S OWN COMMANDS LEFT OFF/,/cd "\$BUILD_ROOT"/p' "$PHASES_LIB" \ + | grep -q '_pp_prepared_in="$PWD"' \ + || _p="$_p;the directory prepare_pkg ended in is not remembered" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a meta page's build resumes where its own download commands ended" +fi -# ---- scratch never enters a manifest ---------------------------------------- # -# $BUILD_ROOT is the busiest directory on the system during a build: unpacked -# source trees, object files, the tarballs themselves. Every one is newer than -# the build stamp, so the touched-file scan swallowed the whole scratch tree -# into every package's manifest and then chowned it -- and the next package -# chowned it right back. A real `verify` came back with 3363 wrong-owner, of -# which 3348 were /build: -# /build/bash-5.3.tar.gz is p_shadow want p_acl -# The two snapshot scans already excluded it; the build's own scan did not. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _cb="$(_slice_fn "$helper_src" cmd_build)" - _p="" - case "$_cb" in - *'-not -path "$_bld/*"'*) ;; - *) _p="$_p;the build scratch is still scanned into every manifest" ;; - esac - case "$_cb" in - *'BUILD_ROOT:-$r/build'*) ;; - *) _p="$_p;the scratch exclusion does not follow \$BUILD_ROOT" ;; - esac - # and verify must ignore scratch left in manifests written before this - case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in - *'BUILD_ROOT'*) ;; - *) _p="$_p;verify still reports scratch recorded by an older build" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the build scratch is never recorded as a package's files" - fi +# ---- three reasons a dependency is "not found" (1.14.26) ------------------- # +# mesa stopped with meson's whole account of the problem: +# ERROR: Dependency "libdrm_intel" not found, tried pkgconfig and cmake +# libdrm WAS installed -- version 2.4.131, found two lines earlier. It had +# been built before libpciaccess existed (the book only RECOMMENDS +# Xorg-Libraries for libdrm, so nothing ordered them), so it has no +# libdrm_intel.pc and never will until it is rebuilt. The .pc files on disk +# tell the three cases apart, and the box now does. +_p="" +_hlpD="$helper_src" +_dr="$T/depreport"; mkdir -p "$_dr" +python3 - "$_hlpD" > "$_dr/rep.sh" <<'PYSLICE2' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/"; s="/x"; staged="/x"; auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE2 +bash -n "$_dr/rep.sh" || _p="$_p;the failure report does not parse" +printf '../meson.build:1719:6: ERROR: Dependency "libdrm_intel" not found, tried pkgconfig and cmake\n' \ + > "$_dr/mesa.log" +# 1. the provider is installed but lacks the component -- the reported case +mkdir -p "$_dr/root1/usr/lib/pkgconfig"; : > "$_dr/root1/usr/lib/pkgconfig/libdrm.pc" +_o="$(bash -c '. "'"$_dr"'/rep.sh"; SNAP_ROOT="'"$_dr"'/root1"; report mesa all "'"$_dr"'/mesa.log"' 2>&1)" +case "$_o" in + *"'libdrm_intel' is missing, but 'libdrm' IS installed"*) ;; + *) _p="$_p;a half-built provider is not recognised" ;; +esac +case "$_o" in + *"install libdrm --run --recursive --reinstall"*) ;; + *) _p="$_p;no command offered to rebuild the provider" ;; +esac +# 2. the .pc exists and was not found -- a path problem, not a missing package +mkdir -p "$_dr/root2/usr/lib/pkgconfig"; : > "$_dr/root2/usr/lib/pkgconfig/libdrm_intel.pc" +_o2="$(bash -c '. "'"$_dr"'/rep.sh"; SNAP_ROOT="'"$_dr"'/root2"; report mesa all "'"$_dr"'/mesa.log"' 2>&1)" +case "$_o2" in + *"EXISTS, at"*"PKG_CONFIG_PATH"*) ;; + *) _p="$_p;an installed .pc that was not found is reported as missing" ;; +esac +case "$_o2" in + *"is missing, but"*) _p="$_p;it claims the provider is half-built when the .pc is right there" ;; +esac +# 3. nothing provides it at all +mkdir -p "$_dr/root3/usr/lib" +_o3="$(bash -c '. "'"$_dr"'/rep.sh"; SNAP_ROOT="'"$_dr"'/root3"; report mesa all "'"$_dr"'/mesa.log"' 2>&1)" +case "$_o3" in + *"Nothing on this system provides 'libdrm_intel'"*) ;; + *) _p="$_p;a genuinely absent dependency is not reported as absent" ;; +esac +# pkg-config's own wording is read too, not only meson's +printf "Package 'foo-bar' not found\nNo package 'foo-bar' found\n" > "$_dr/pc.log" +_o4="$(bash -c '. "'"$_dr"'/rep.sh"; SNAP_ROOT="'"$_dr"'/root3"; report demo all "'"$_dr"'/pc.log"' 2>&1)" +case "$_o4" in + *"foo-bar"*) ;; + *) _p="$_p;pkg-config's own 'No package ... found' is not recognised" ;; +esac +# the searches stay bounded +sed -n '/A DEPENDENCY THE BUILD SYSTEM COULD NOT FIND/,/^ fi$/p' "$_hlpD" \ + | grep -q 'timeout 10 find' \ + || _p="$_p;the .pc search is unbounded" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a missing dependency says which of the three things went wrong" fi -# ---- some files belong to no package ---------------------------------------- # -# Every package that installs an info page rewrites /usr/share/info/dir, and -# ldconfig rewrites /etc/ld.so.cache after every install. Recording those in a -# manifest makes verify report a conflict no repair can settle -- the next -# package takes them straight back: -# /usr/share/info/dir is p_gcc want p_binutils -# /etc/ld.so.cache is p_libcap want p_glibc -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( SNAP_ROOT=/ - eval "$(_slice_fn "$helper_src" never_claim_list)" - eval "$(_slice_fn "$helper_src" _never_claim_load)" - eval "$(_slice_fn "$helper_src" is_never_claimed)" - for f in /usr/share/info/dir /etc/ld.so.cache /var/cache/ldconfig/aux-cache; do - is_never_claimed "$f" || echo "$f can still be claimed by a package" - done - # the wrapper directory too: it is root:root on purpose - is_never_claimed /usr/lib/pkgusr/chown \ - || echo "a wrapper can be claimed by a package" - # but a real package file must NOT be swept up by this - is_never_claimed /usr/bin/bash \ - && echo "an ordinary package file is treated as unclaimable" )" - if [ -z "$_o" ]; then - ok "shared indexes and generated caches are owned by no package" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- an old account can point at the wrong home (1.14.27) ------------------ # +# elogind ended up with TWO half-populated homes: +# /usr/src/p_elogind/ install_elogind, install_last, log +# /usr/src/pkgusr/p_elogind/ build, build.conf, src +# Its .bash_profile was dated Nov 2023: an account carried over from an +# earlier system, whose /etc/passwd home was never re-examined. Every tool +# uses pkgusr_home_for (the canonical path), but `su -` starts in the PASSWD +# home -- so the build ran in one directory while its sources were staged in +# the other. The repair existed and ran only when an account was CREATED. +_p="" +_hlpE="$helper_src" +_pmE2="$(dirname "$LFS_TOOL")/packagemanager" +sed -n '/^cmd_pm_install() {/,/pkgusr_home_for "\$user_name"/p' "$_hlpE" \ + | grep -q '_repair_misplaced_home "$user_name" 1' \ + || _p="$_p;an existing account's home is never re-checked before a build" +# it must be the LOUD form: this is not the expected just-created case +sed -n '/^cmd_pm_install() {/,/pkgusr_home_for "\$user_name"/p' "$_hlpE" \ + | grep -q '_repair_misplaced_home "$user_name" 1 1' \ + && _p="$_p;a home wrong since 2023 is reported as routine" +# ...and only for an account that exists, so creation still owns that path +sed -n '/^cmd_pm_install() {/,/pkgusr_home_for "\$user_name"/p' "$_hlpE" \ + | grep -q 'user_exists "$user_name" && _repair_misplaced_home' \ + || _p="$_p;the repair runs for accounts that do not exist yet" +# a git stack entry needs git, which appears in no command: the runner clones +python3 - "$_pmE2" <<'PYGIT' || _p="$_p;a git entry does not know it needs git" +import sys, io, tempfile, contextlib, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +e = {"target": "elogind", "kind": "git", + "opts": {"url": "https://example.invalid/elogind", "ref": "v255.17", + "build": "meson setup build; ninja -C build"}} +p, _ = pm._stack_git_script(e, tempfile.mkdtemp()) +assert "is_git" in open(p).read() +pm._tool_available = lambda t: False +pm._MISSING_TOOLS.clear() +with contextlib.redirect_stderr(io.StringIO()), contextlib.redirect_stdout(io.StringIO()): + pm._warn_missing_build_tools("elogind", p) +assert "git" in pm._MISSING_TOOLS, sorted(pm._MISSING_TOOLS) +# and git IS a book package, so the command offered has to work +assert pm._BUILD_TOOLS.get("git") == "git" +# nothing is reported when everything is installed +pm._tool_available = lambda t: True +pm._MISSING_TOOLS.clear() +with contextlib.redirect_stderr(io.StringIO()), contextlib.redirect_stdout(io.StringIO()): + pm._warn_missing_build_tools("elogind", p) +assert not pm._MISSING_TOOLS, sorted(pm._MISSING_TOOLS) +PYGIT +# the stack checks a git entry before it creates anything +sed -n '/THE SAME CHECKS AS A BOOK PACKAGE/,/install_local(target, spath/p' "$_pmE2" \ + | grep -q '_check_script_prereqs(target, spath)' \ + || _p="$_p;a git entry is built without checking what it needs" +sed -n '/THE SAME CHECKS AS A BOOK PACKAGE/,/install_local(target, spath/p' "$_pmE2" \ + | grep -q 'ignore_prereqs' \ + || _p="$_p;there is no way to push a git entry past the check" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an inherited account's home is repaired, and a clone knows it needs git" fi -# ---- the 7.2 handover is decided by the tree, not one inode ------------------ # -# `_owner_is_root` looks at $LFS/usr. The pass it guarded is -# `chown -h -R root:root` over EIGHT trees. A tree whose /usr was already -# root-owned got no handover at all, and everything chapters 5-6 created below -# it kept the host uid: -# drwxr-xr-x 1 lfs lfs /usr/share/gcc-15.2.0/python/libstdcxx -# lfs:install drwxrwxr-x /usr/share -# It must also refuse to run once package users own files -- chown -R root:root -# over /usr is right before chapter 8 and catastrophic after it. -python3 - "$LFS_TOOL" <<'PYHANDOVER' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -for fn in ("_handover_needed", "_handover_is_safe"): - if not re.search(r"^def %s\(" % fn, src, re.M): - problems.append("there is no %s" % fn) -m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) -if m and "_find_uid" not in m.group(0): - problems.append("_handover_needed still decides from a single directory") -m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) -# It must ask the TREE's passwd which accounts are package users, not infer it -# from a uid range: book 4.3 pins no uid for `lfs`, so on one host the build -# account came out 10753 -- inside the package range -- and a range filter -# excluded the very user it was looking for. -if m and "_pkg_users_in_tree" not in m.group(0): - problems.append("_handover_is_safe infers package users from a uid range") -if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): - problems.append("there is no _pkg_users_in_tree") -m2 = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) -if m2 and "PKG_UID_MIN" in m2.group(0): - problems.append("_handover_needed still filters the build user out by uid range") -m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) -if m: - body = m.group(0) - if "_handover_needed" not in body: - problems.append("chroot prepare still guards 7.2 on _owner_is_root alone") - if "_handover_is_safe" not in body: - problems.append("chroot prepare would run chown -R over a chapter-8 tree") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the 7.2 handover is decided by the tree and refuses a built one") -PYHANDOVER -_count_rc $? +# ---- build it, do not just name it (1.14.28) ------------------------------- # +# The stack stopped correctly, and then asked the person to type the thing the +# stack exists to do: +# stack stopped at git 'elogind': it needs a tool that is not installed +# packagemanager install git --run --recursive +# A PLAN handles this itself (1.14.14: a missing provider joins the plan and +# the plan is rebuilt). A git entry is not a plan, so it stopped. git is a +# book package; an unattended run should install it and carry on. +_p="" +_pmF2="$(dirname "$LFS_TOOL")/packagemanager" +_gb="$(sed -n '/BUILD IT, DO NOT JUST NAME IT/,/To start regardless/p' "$_pmF2")" +printf '%s' "$_gb" | grep -q '_install_via_self(_prov, args.yes, recursive=True)' \ + || _p="$_p;a missing tool the book provides is still only named" +printf '%s' "$_gb" | grep -q '_provider_of(t) for t in _MISSING_TOOLS' \ + || _p="$_p;the tool is not resolved to a book package before installing" +# a tool with NO book page cannot be installed, and must still stop the run +printf '%s' "$_gb" | grep -q 'no BLFS page provides' \ + || _p="$_p;a tool the book cannot provide is treated as installable" +printf '%s' "$_gb" | grep -q 'come from LFS' \ + || _p="$_p;it does not say where an unprovidable tool comes from" +# a failed install of the tool stops the run rather than building on +printf '%s' "$_gb" | grep -q 'could not"' \ + || _p="$_p;a failed tool install does not stop the stack" +# and the split itself is right: git resolves, meson does not +_bk28="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk28="$_c" && break +done +if [ -n "$_bk28" ]; then + BLFS_BOOK_FILE="$_bk28" python3 - "$_pmF2" <<'PYSPLIT' || _p="$_p;the installable/not-installable split is wrong" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm._provider_of("git") == "git", pm._provider_of("git") +assert pm._provider_of("meson") == "", pm._provider_of("meson") +assert pm._provider_of("ninja") == "", pm._provider_of("ninja") +PYSPLIT +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a stack installs a tool the book has instead of asking for it" +fi -# and a tree that already has the damage must be repairable -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q '\-user lfs' \ - || _p="$_p;verify cannot find files the build user still owns" - _slice_fn "$helper_src" _vfy_build_user_leftovers | grep -q 'real_chown -h root:root' \ - || _p="$_p;verify --fix cannot hand the build user's leftovers back to root" - case "$(_slice_fn "$helper_src" cmd_verify)" in - *_vfy_build_user_leftovers*) ;; - *) _p="$_p;verify does not run the build-user pass" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a tree the handover skipped can be repaired from inside the chroot" - fi +# ---- the setup for a test is part of the test (1.14.29) -------------------- # +# Xwayland's build cloned piglit, cloned the X Test Suite, started an Xvfb and +# tried to build xts -- during an ORDINARY build -- and died: +# install_Xwayland-24.1.9: line 54: cd: xts: No such file or directory +# Fatal server error +# The page puts all of that in its BUILD block and only the last line, +# `ninja test`, looks like a test run, so 1.13's filter commented that one +# line and left the setup running. Commenting a test but not what it needs is +# worse than commenting neither: the cost stays and the point is gone. +_p="" +_b29="$(dirname "$LFS_TOOL")/blfs" +_bk29="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk29="$_c" && break +done +if [ -n "$_bk29" ]; then + _xw="$T/xwayland"; mkdir -p "$_xw" + python3 "$_b29" --book-file "$_bk29" script xwayland -o "$_xw" >/dev/null 2>&1 + _f="$(ls "$_xw"/install_Xwayland* 2>/dev/null | head -1)" + if [ -n "$_f" ]; then + _live="$(sed -n '/^build_pkg() {/,/^}/p' "$_f" | grep -vE '^\s*#|^\s*$')" + # the real build survives + printf '%s' "$_live" | grep -q 'meson setup' \ + || _p="$_p;the real build commands were commented out too" + printf '%s' "$_live" | grep -qw 'ninja' \ + || _p="$_p;the compile step was lost" + # the test scaffolding does not + for _t in piglit xts Xvfb autogen.sh; do + printf '%s' "$_live" | grep -q "$_t" \ + && _p="$_p;$_t still runs during an ordinary build" + done + printf '%s' "$_live" | grep -q 'mkdir tools' \ + && _p="$_p;the directory made only for the test suite is still created" + # the book's text is kept, commented, so it can be read and re-enabled + grep -q '# git clone https://gitlab.freedesktop.org/mesa/piglit' "$_f" \ + || _p="$_p;the page's own test setup was deleted rather than commented" + grep -q 'builds the test suite that the commented run below uses' "$_f" \ + || _p="$_p;nothing says why that block is commented" + bash -n "$_f" || _p="$_p;the generated script does not parse" + else + _p="$_p;could not generate the xwayland script" + fi +fi +# the rule is narrow: a pushd group is only taken when it IS a test suite +_ts="$(sed -n '/_TEST_SETUP_RE = re.compile/,/re.I)/p' "$_b29")" +printf '%s' "$_ts" | grep -q 'piglit' \ + || _p="$_p;the test-suite names the books fetch are not recognised" +sed -n '/AND SO DOES THE BLOCK THAT BUILT THE TEST SUITE/,/out.insert/p' "$_b29" \ + | grep -q '_TEST_SETUP_RE.search(_span)' \ + || _p="$_p;any pushd group before a test would be commented, not just a test suite" +sed -n '/AND SO DOES THE BLOCK THAT BUILT THE TEST SUITE/,/out.insert/p' "$_b29" \ + | grep -q 'popd' \ + || _p="$_p;the group is not bounded by pushd/popd" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a test suite is not built during an ordinary build" fi -# ---- every function called is a function that exists ------------------------ # -# `verify --fix` called cmd_fix_orphans, which does not exist -- `fix-orphans` -# is dispatched as an ALIAS for `verify --fix`, so the call was both undefined -# and circular. It died mid-repair: -# line 4793: cmd_fix_orphans: command not found -# after chowning 3473 paths, leaving the pass half-done and the status wrong. -# -# A missing function in the REPAIR path is the worst place for one: it only -# fires when something is already broken, which is when it is least welcome. -for _t in lfs-helper; do - _src="$(dirname "$LFS_TOOL")/$_t" - [ -f "$_src" ] || continue - _missing="" - # comments stripped first: the fix for this bug NAMES the function it - # removed, and matching that text would make the test permanently red - for _fn in $(grep -vE '^[[:space:]]*#' "$_src" \ - | grep -oE '(^|[^_[:alnum:]])cmd_[a-z0-9_]+' \ - | grep -oE 'cmd_[a-z0-9_]+' | sort -u); do - grep -qE "^${_fn}\(\) \{" "$_src" || _missing="$_missing $_fn" - done - if [ -n "$_missing" ]; then - for _m in $_missing; do bad "$_t calls $_m, which is not defined"; done - else - ok "$_t defines every cmd_* function it calls" - fi +# ---- ask the way the build will ask (1.14.30) ------------------------------ # +# A plan printed itself twice and refused twice: +# cargo needed by 2 package(s), first: cargo-c +# adding to the plan: rust -- its commands call `cargo` +# ... same plan, same refusal ... +# Rust WAS installed. BLFS puts it in /opt/rustc and adds /opt/rustc/bin in +# /etc/profile.d/rustc.sh; a build runs through `su -`, a LOGIN shell, which +# reads that. `shutil.which` reads OUR path, which does not. So a tool the +# build could see was reported missing -- and the 1.14.14 repair could not +# help, because adding an already-installed package changes nothing, so the +# second pass produced the identical stop. +_p="" +_pmG="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmG" <<'PYTOOL' || _p="$_p;a tool is not looked for the way the build will find it" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm._tool_available("bash") is True +assert pm._tool_available("zzz-definitely-not-a-tool") is False +# what a login shell adds is found: that is exactly /opt/rustc/bin's case +os.makedirs("/tmp/lfstest-opt/bin", exist_ok=True) +p = "/tmp/lfstest-opt/bin/lfstest-tool" +open(p, "w").write("#!/bin/sh\nexit 0\n"); os.chmod(p, 0o755) +prof = "/etc/profile.d/lfstest-tool.sh" +wrote = False +try: + open(prof, "w").write('export PATH=/tmp/lfstest-opt/bin:$PATH\n') + wrote = True +except OSError: + pass +if wrote: + pm._TOOL_SEEN.clear() + try: + assert pm._tool_available("lfstest-tool") is True, \ + "a tool added by /etc/profile.d is reported missing" + finally: + os.remove(prof) +PYTOOL +sed -n '/def _tool_available/,/^def /p' "$_pmG" | grep -q 'bash", "-lc"' \ + || _p="$_p;the login shell is never consulted" +sed -n '/def _tool_available/,/^def /p' "$_pmG" | grep -q 'timeout=' \ + || _p="$_p;the login-shell probe is unbounded" +sed -n '/def _tool_available/,/^def /p' "$_pmG" | grep -q '_TOOL_SEEN' \ + || _p="$_p;the probe is repeated for every package that wants the tool" +# nothing may use the bare PATH check for a build tool any more +grep -q 'if shutil.which(word) or word in missing' "$_pmG" \ + && _p="$_p;the tool scan still reads only our own PATH" +# and a provider that is already installed is never "added to the plan" again +grep -q 'gather_user(a).state != "installed"' "$_pmG" \ + || _p="$_p;an installed provider can still be added, so the plan repeats" +sed -n '/SAY WHEN THE PACKAGE IS THERE AND ONLY THE TOOL/,/command -v/p' "$_pmG" \ + | grep -q 'profile.d' \ + || _p="$_p;a PATH problem is not distinguished from a missing package" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a build tool is looked for on the build's PATH, not only on ours" +fi + +# ---- a link in an aside is not a dependency (1.14.31) ---------------------- # +# "should we block some things so its not getting that heavy?" Planning +# Xwayland -- an X server for Wayland -- came to 235 packages including +# LibreOffice, OpenJDK, PostgreSQL, TeX Live and a print stack. The reason +# was one sentence on the harfbuzz page: +# Graphite2-1.3.14 (required for building texlive-20250308 or +# LibreOffice-26.2.1.2 with system harfbuzz) +# Those name what NEEDS graphite2. The parser harvested every <a class="xref"> +# in the paragraph, so harfbuzz "recommended" LibreOffice and TeX Live, and +# poppler "recommended" okular from "Qt-6.10.2 (required for PDF support in +# okular)". The qualified package is always the link BEFORE the parenthesis, +# so what is inside can go. +_p="" +_b31="$(dirname "$LFS_TOOL")/blfs" +_bk31="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk31="$_c" && break done +if [ -n "$_bk31" ]; then + python3 - "$_b31" "$_bk31" <<'PYASIDE' || _p="$_p;links inside a parenthetical aside are still read as dependencies" +import sys, types, importlib.machinery as m +b = m.SourceFileLoader('blfs', sys.argv[1]).load_module() +book = b.get_book(types.SimpleNamespace(book_file=sys.argv[2], book=None)) +def deps(name): + d = book.get(b.resolve_anchor(book, b.anchor_from_input(name))) + return d["required"] + d["recommended"] +h = deps("harfbuzz") +# the aside named these; they are not harfbuzz's dependencies +assert not [x for x in h if x.lower().startswith("libreoffice")], h +assert not [x for x in h if x.lower().startswith("texlive")], h +# ...and what the sentence really recommends is still there +for want in ("Graphite2", "icu", "FreeType", "GLib"): + assert [x for x in h if x.lower().startswith(want.lower())], (want, h) +p = deps("poppler") +assert not [x for x in p if x.lower().startswith("okular")], p +assert [x for x in p if x.lower().startswith("qt-6")], p # Qt6 IS required +# the page that started it: an X server should not pull an office suite +o = "\n".join(x for x in []) +PYASIDE + _n="$(python3 "$_b31" --book-file "$_bk31" order xwayland --anchors 2>/dev/null | wc -l)" + [ "${_n:-999}" -lt 100 ] \ + || _p="$_p;xwayland still plans $_n packages -- the aside edges are back" + python3 "$_b31" --book-file "$_bk31" order xwayland --anchors 2>/dev/null \ + | cut -f1 | grep -qixE "libreoffice|openjdk|postgresql|texlive" \ + && _p="$_p;an office suite or a JDK is still in the xwayland tree" + # nothing was lost at the other end: the required-only tree is unchanged + _r="$(python3 "$_b31" --book-file "$_bk31" order xwayland --anchors \ + --no-recommended 2>/dev/null | wc -l)" + [ "${_r:-0}" -ge 20 ] \ + || _p="$_p;the required tree shrank too -- real dependencies were dropped" +fi +sed -n '/def _in_parenthetical/,/^def /p' "$_b31" | grep -q 'previous_siblings' \ + || _p="$_p;the aside test does not look at what precedes the link" +# (the number moves on; what matters is that a PARSE change bumped it -- the +# aside fix was v48, and any later parse change must bump it again) +grep -qE 'CACHE_VERSION = (4[89]|[5-9][0-9])' "$_b31" \ + || _p="$_p;the cache version was not bumped, so old parses are still served" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package named in an aside is not treated as a dependency" +fi -# ---- a package builds in its own home, not a shared scratch ----------------- # -# /build was a permanent ownership fight. Every build's touched-file scan swept -# the whole tree in, so each manifest claimed every other package's sources and -# chowned them; the next package chowned them back. Repairing it made it worse, -# because the question had no answer: -# /build/bash-5.3.tar.gz is p_shadow want p_acl -# /build/bash-5.3.tar.gz is p_zstd want p_acl (after the repair) -# -# Under the package user's home it is settled by construction: the user creates -# the tree so the user owns it, and $SRCROOT is already excluded from every tree -# scan, so it can never reach a manifest at all. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( SNAP_ROOT=/; SRCROOT=/usr/src - PKGUSR_ROOT=/usr/src/pkgusr; CFGUSR_ROOT=/usr/src/cfg - PKGUSR_PREFIX="p_"; CFGUSR_PREFIX="cfg_" - STATE=/.lfs-pkgusr; SCRIPTS="$STATE/scripts" - PKGUSR_BUILD_SUBDIR=src - for fn in pkgusr_kind pkgusr_prefix_for pkgusr_root_for unprefix_pkg_user \ - pkg_owner_name pkgusr_home_for _is_not_a_package build_root_for; do - eval "$(_slice_fn "$helper_src" "$fn")" - done - if ! command -v build_root_for >/dev/null 2>&1; then - echo "there is no build_root_for -- the scratch is still shared" - exit 0 # the rest would be noise, not findings - fi - # a package builds under its own account's home - [ "$(build_root_for gcc)" = "/usr/src/pkgusr/p_gcc/$PKGUSR_BUILD_SUBDIR" ] \ - || echo "gcc does not build in its own home: $(build_root_for gcc)" - # an earlier stage folds onto the same package, so it reuses that tree - [ "$(build_root_for util-linux-tmp)" = "$(build_root_for util-linux)" ] \ - || echo "an earlier stage builds somewhere else than its package" - # and it must be under $SRCROOT, which every tree scan already excludes -- - # that is what keeps it out of manifests without another find exclusion - case "$(build_root_for gcc)" in - "$SRCROOT"/*) ;; - *) echo "the build tree is not under SRCROOT, so scans will sweep it in" ;; - esac - # a step with no account keeps the shared scratch: it runs as root, which - # owns everything there anyway - case "$(build_root_for init-dirs)" in - /usr/src/*) echo "a step with no account was given a home to build in" ;; - esac )" - if [ -z "$_o" ]; then - ok "each package unpacks and builds inside its own account's home" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- say what the cycle cost (1.14.32) ------------------------------------- # +# vala stopped at +# configure: error: Package requirements (libgvc >= 2.16) were not met +# libgvc is Graphviz, which vala RECOMMENDS -- and the plan had graphviz in +# it, 20 packages later. The cause is a real cycle: +# vala -> graphviz -> gegl -> babl -> librsvg -> vala +# all recommended, so one edge has to give and vala loses. That ordering is +# the best available; doing it silently is not. +_p="" +_b32="$(dirname "$LFS_TOOL")/blfs" +_pm32="$(dirname "$LFS_TOOL")/packagemanager" +_bk32="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk32="$_c" && break +done +if [ -n "$_bk32" ]; then + _co="$(python3 "$_b32" --book-file "$_bk32" order vala --anchors 2>/dev/null)" + printf '%s\n' "$_co" | grep -q '^#cycle' \ + || _p="$_p;a broken cycle edge is not reported by the walker" + printf '%s\n' "$_co" | grep -q '^#cycle vala graphviz' \ + || _p="$_p;the vala/graphviz cycle is not among them" + # the order itself must still be a valid one: every package appears once + _dups="$(printf '%s\n' "$_co" | grep -v '^#' | cut -f1 | sort | uniq -d)" + [ -z "$_dups" ] || _p="$_p;the order repeats packages: $_dups" +fi +# packagemanager reports only the direction that actually happened +sed -n '/def _report_cycles/,/^def /p' "$_pm32" | grep -q '_pos\[a.lower()\] < _pos\[d.lower()\]' \ + || _p="$_p;both directions of a two-package cycle are reported, so one line is false" +sed -n '/def _report_cycles/,/^def /p' "$_pm32" | grep -q 'reinstall' \ + || _p="$_p;no rebuild is offered for what the cycle cost" +grep -q '_report_cycles(steps)' "$_pm32" \ + || _p="$_p;the plan never reports its cycles" +# and autoconf's wording reaches the .pc diagnosis (1.14.26 knew only meson's) +_dr2="$T/depwording"; mkdir -p "$_dr2/root/usr/lib" +python3 - "$helper_src" > "$_dr2/rep.sh" <<'PYSLICE3' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/"; s="/x"; staged="/x"; auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE3 +printf "configure: error: Package requirements (libgvc >= 2.16) were not met:\nPackage 'libgvc' not found\n" \ + > "$_dr2/vala.log" +_o="$(bash -c '. "'"$_dr2"'/rep.sh"; SNAP_ROOT="'"$_dr2"'/root"; report vala all "'"$_dr2"'/vala.log"' 2>&1)" +case "$_o" in + *"provides 'libgvc'"*) ;; + *) _p="$_p;autoconf's 'Package requirements ... were not met' is not diagnosed" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a cycle says which package it shortchanged, and how to make it good" +fi + +# ---- the manifests know who claimed it (1.14.33) --------------------------- # +# librsvg stopped on `Program 'cargo-cbuild' not found`. The box knew the +# shape of the problem -- a package recorded as installed whose build did not +# finish -- and then said "find the package, purge its record, rebuild it", +# leaving the person to find it. Every installed package has a pkg.lst naming +# the files it installed; a grep answers in a second, and the answer decides +# WHICH of two situations it is. +_p="" +_hlpF="$helper_src" +_mf="$T/manifests"; mkdir -p "$_mf/p_cargo-c" +python3 - "$_hlpF" > "$_mf/rep.sh" <<'PYSLICE4' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/"; s="/x"; staged="/x"; auto_repair=1; auto_fix=0; _perm_ish=0') +print('SRCROOT="' + sys.argv[2] + '"' if len(sys.argv) > 2 else 'SRCROOT="/tmp"') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE4 +sed -i "s|^SRCROOT=.*|SRCROOT=\"$_mf\"|" "$_mf/rep.sh" +bash -n "$_mf/rep.sh" || _p="$_p;the report does not parse" +printf "../meson.build:28:10: ERROR: Program 'cargo-cbuild' not found or not executable\n" \ + > "$_mf/log" +# 1. a package's record CLAIMS the file: name it and rebuild it +printf '/usr/bin/cargo-cbuild\n/usr/bin/cargo-capi\n' > "$_mf/p_cargo-c/pkg.lst" +_o="$(bash -c '. "'"$_mf"'/rep.sh"; report librsvg all "'"$_mf"'/log"' 2>&1)" +case "$_o" in + *"'p_cargo-c' RECORDS having installed it"*) ;; + *) _p="$_p;the package whose manifest claims the file is not named" ;; +esac +case "$_o" in + *"install cargo-c --run --reinstall"*) ;; + *) _p="$_p;the rebuild command does not name the package (or keeps the p_ prefix)" ;; +esac +case "$_o" in + *"rm /usr/src/pkgusr/p_<pkg>"*) _p="$_p;it still tells the user to delete files by hand" ;; +esac +# 2. nobody claims it: a different situation, a different answer +rm -f "$_mf/p_cargo-c/pkg.lst" +_o2="$(bash -c '. "'"$_mf"'/rep.sh"; report librsvg all "'"$_mf"'/log"' 2>&1)" +case "$_o2" in + *"No installed package records that file"*) ;; + *) _p="$_p;an unprovided program is reported as a bad record" ;; +esac +case "$_o2" in + *"packagemanager search cargo-cbuild"*) ;; + *) _p="$_p;no way offered to find what provides it" ;; +esac +# the search stays bounded -- a failure report cannot hang +sed -n '/THE MANIFESTS KNOW WHO CLAIMED IT/,/^ fi$/p' "$_hlpF" | grep -q 'timeout ' \ + || _p="$_p;the manifest search is unbounded" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a missing program names the package that claims to have installed it" +fi + +# ---- a health check must lead with what is wrong (1.14.34) ----------------- # +# A real `packagemanager verify` on a working tree: +# broken: 4 not installed: 12 unvalidated: 226 +# The 226 are normal -- "unvalidated" means the page named no file to check, +# not that anything is broken -- and the twelve that need action were inside +# them. Worse, three of the four "broken" were nimgnu_* COLLECTOR GROUPS, +# which are groups and not packages at all, reported as "has a user but no +# home dir". A false alarm in a health check teaches people to skip it. +_p="" +_pmI="$(dirname "$LFS_TOOL")/packagemanager" +sed -n '/A COLLECTOR GROUP IS NOT A PACKAGE/,/startswith(_cp/p' "$_pmI" \ + | grep -q 'collector_prefix()' \ + || _p="$_p;collector groups are still listed as package-users" +python3 - "$_pmI" <<'PYCOLL' || _p="$_p;the collector-group filter does not work" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +root = tempfile.mkdtemp() +for d in ("p_zlib", "p_bash", "nimgnu_perl-modules", "nimgnu_python-modules"): + os.makedirs(os.path.join(root, d)) + # a package user's home carries the skeleton; without it these are just + # directories, and 1.14.34 drops those (a stray /usr/src/root was being + # reported as a broken package). Give them the evidence, so the only + # thing that can exclude the collector groups is the PREFIX. + open(os.path.join(root, d, "build.conf"), "w").write("") +pm.pkgusr_roots = lambda: [root] +pm.collector_prefix = lambda: "nimgnu" +got = set(pm.list_all_users()) +assert "zlib" in got and "bash" in got, got +assert not [g for g in got if g.startswith("nimgnu")], got +# a different configured prefix is honoured, not a hardcoded one +# nimgnu and sysgroup are BOTH recognised now (1.14.35: the two tools had +# different fallbacks, so real systems carry groups under either name), so a +# third prefix is the way to show the configured value is honoured +for d in ("acme_shared-thing",): + os.makedirs(os.path.join(root, d)) + open(os.path.join(root, d, "build.conf"), "w").write("") +pm.collector_prefix = lambda: "acme" +got2 = set(pm.list_all_users()) +assert not [g for g in got2 if g.startswith("acme")], got2 +pm.collector_prefix = lambda: "somethingelse" +assert [g for g in pm.list_all_users() if g.startswith("acme")], \ + "the configured prefix is ignored" +PYCOLL +# and the summary repeats the actionable ones with a command +sed -n '/LEAD WITH WHAT NEEDS DOING/,/nothing needs attention/p' "$_pmI" \ + > "$T/verify_tail.py" 2>/dev/null +grep -q 'in ("not installed", "broken")' "$_pmI" \ + || _p="$_p;the summary does not pick out the states that need action" +grep -q 'nothing needs attention' "$_pmI" \ + || _p="$_p;a clean tree is not told so" +sed -n '/LEAD WITH WHAT NEEDS DOING/,/nothing needs attention/p' "$_pmI" \ + | grep -q 'packagemanager install %s --run --recursive' \ + || _p="$_p;no command is offered to rebuild what verify found" +sed -n '/LEAD WITH WHAT NEEDS DOING/,/nothing needs attention/p' "$_pmI" \ + | grep -q "unvalidated" \ + || _p="$_p;nothing explains that 'unvalidated' is normal" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "verify ends with the packages that need action, not with 226 that do not" +fi - # and the build must actually be told about it - case "$(_slice_fn "$helper_src" cmd_build)" in - *"BUILD_ROOT='\$_bldroot'"*) - ok "the per-package build root is passed into the build" ;; - *) bad "the build still inherits whatever BUILD_ROOT it was started with" ;; - esac +# ---- a stray directory is not a package (1.14.34) -------------------------- # +# The account list walked every directory under BASE_DIR (/usr/src), which +# holds more than package users, so `root` came out as +# root broken has a home dir but no user +# An account either exists in the user database, carries one of the prefixes +# accounts are created with (p_, u_, cfg_), or has the skeleton every package +# user gets. A bare directory is none of those. +_p="" +_pmJ="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmJ" <<'PYSTRAY' || _p="$_p;a stray directory is still listed as a package" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +root = tempfile.mkdtemp() +for d in ("p_zlib", "cfg_bootscripts", "lfs-pkgusr", "root"): + os.makedirs(os.path.join(root, d)) +pm.pkgusr_roots = lambda: [root] +got = set(pm.list_all_users()) +# the prefix is evidence enough for a real account +assert "zlib" in got, got +assert "cfg_bootscripts" in got, got +# ...and a directory that is none of the three is not an account +assert "lfs-pkgusr" not in got, got +# a directory with the skeleton IS one, whatever it is called +os.mkdir(os.path.join(root, "handmade")) +open(os.path.join(root, "handmade", "install_last"), "w").write("") +assert "handmade" in pm.list_all_users(), pm.list_all_users() +PYSTRAY +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the account list holds accounts, not every directory in /usr/src" fi -# ---- the root symlinks are chowned, not followed ---------------------------- # -# Book 4.2 creates /bin -> usr/bin, /lib and /sbin as the build user. They came -# out lfs:lfs, and a chown WITHOUT -h follows the link and retargets /usr/bin -# instead -- silently doing the wrong thing to the right-looking path. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _blk="$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" - _p="" - case "$_blk" in - *'real_chown -h'*) ;; - *) _p="$_p;the build-user repair follows symlinks instead of chowning them" ;; - esac - case "$(printf '%s\n' "$_blk" | grep -vE '^[[:space:]]*#')" in - *'-type f'*) _p="$_p;the build-user scan skips symlinks and directories" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a symlink owned by the build user is chowned, not followed" - fi +# ---- an install that leaves no record is invisible (1.14.37) --------------- # +# `packagemanager verify` reported beautifulsoup4, certifi, charset_normalizer, +# idna, requests, soupsieve and urllib3 as +# not installed user + home exist, but nothing installed yet +# while blfs was using bs4 to parse the book on the same machine. Their homes +# held build.conf and .project and nothing else: `packagemanager pip` installed +# the module and wrote NO record -- no pkg.lst, no install_last. Then verify +# offered to rebuild them with a command that could not work either, because +# they are not BLFS packages. +_p="" +_pmK="$(dirname "$LFS_TOOL")/packagemanager" +# the installer records what it installed +sed -n '/RECORD IT LIKE EVERY OTHER INSTALL/,/return True/p' "$_pmK" \ + | grep -q '_record_pip_install(user, mod' \ + || _p="$_p;a pip install still leaves no record" +_rec="$(sed -n '/^def _record_pip_install/,/^def /p' "$_pmK")" +printf '%s' "$_rec" | grep -q 'write_pkg_list(user, background=False)' \ + || _p="$_p;no manifest is written, so the files it installed are unknown" +printf '%s' "$_rec" | grep -q 'install_last' \ + || _p="$_p;no version stamp is written" +printf '%s' "$_rec" | grep -q 'validate_cmd' \ + || _p="$_p;nothing is recorded that can confirm the module later" +# ...and the version comes from the module, not from what was asked for +printf '%s' "$_rec" | grep -q 'importlib.metadata' \ + || _p="$_p;the recorded version is not the one pip actually installed" +# a module installed BEFORE this can be given its record without reinstalling +grep -q 'def cmd_pip_record' "$_pmK" \ + || _p="$_p;there is no way to record an already-installed module" +python3 ./packagemanager pip --help 2>&1 | grep -q 'record' \ + || _p="$_p;pip record is not reachable from the command line" +python3 - "$_pmK" <<'PYREC' || _p="$_p;the module probe is wrong, so record would refuse or lie" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +# these tools need bs4 themselves, so it is installed wherever they run +assert pm._python_module_here("beautifulsoup4") is True +assert pm._python_module_here("definitely-not-a-module-xyz") is False +# the underscore/dash spelling difference must not decide it +assert pm._python_module_here("charset-normalizer") == \ + pm._python_module_here("charset_normalizer") +PYREC +# refusing to record something that is NOT installed is the point of the probe +sed -n '/^def cmd_pip_record/,/^def /p' "$_pmK" | grep -q 'record refused' \ + || _p="$_p;record would stamp a module that is not installed" +# and verify says which state this is, instead of "nothing installed yet" +sed -n '/A PIP MODULE INSTALLED BEFORE 1.14.37/,/continue/p' "$_pmK" \ + | grep -q 'packagemanager pip record' \ + || _p="$_p;verify does not tell you how to fix an unrecorded module" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a pip install records itself, and an old one can be recorded in place" fi -# ---- the tree can be set up before book 7.6 exists -------------------------- # -# A chapter 5-6 snapshot restored before 7.6 has NO /etc/passwd -- the chroot -# prompt says "I have no name!". `chown root:install` needs `root` to resolve, -# so every call failed and init-pkgusr reported: -# # 0 install directories are now root:install (group-writable) -# # could not set the group on: /usr /usr/bin /usr/lib /usr/share ... -# while ensure_install_dirs_writable used `chgrp` and worked fine. Two rules -# for one directory, and only one of them survived an empty passwd file. -# -# Ownership is stored as a NUMBER, so use the number. +# ---- a report names the thing it is about (1.14.38) ------------------------ # +# root broken has a home dir but no user +# sent the user looking for /usr/src/root, which is not where it was -- the +# path was in hand when the line was printed and was thrown away. And the +# advice under it read "remove with: packagemanager remove <name>", which for +# a name like `root` is an invitation to an accident: the fix for a directory +# with no account is to delete the DIRECTORY. _p="" -_pre="$T/pre76" -mkdir -p "$_pre/etc" "$_pre/usr/share/man/man3" -: > "$_pre/etc/passwd" # exists, but empty: no root entry -printf 'install:x:9999:\n' > "$_pre/etc/group" -# the shape that broke: a name that cannot resolve -if chown root:install "$_pre/usr/share/man/man3" 2>/dev/null; then - : # this host HAS a root and an install group, so the fixture proves nothing +_pmL="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmL" <<'PYBROKE' || _p="$_p;a broken entry does not name the directory it is about" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +root = tempfile.mkdtemp() +d = os.path.join(root, "root"); os.makedirs(d) +open(os.path.join(d, "build.conf"), "w").write("") +pm.pkgusr_roots = lambda: [root] +pm.pkgusr_home = lambda n: os.path.join(root, n) +st, why = pm.assess(pm.gather_user("root"), None) +assert st == "broken", (st, why) +assert d in why, why # the PATH, not just the fact +PYBROKE +sed -n '/Not BLFS packages, and not rebuilt/,/removed/p' "$_pmL" \ + | grep -q 'packagemanager remove <name>' \ + && _p="$_p;it still suggests removing an account for a stray directory" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done else - # numeric must work where the name did not - if chown -h 0:9999 "$_pre/usr/share/man/man3" 2>/dev/null; then - [ "$(stat -c '%u:%g' "$_pre/usr/share/man/man3")" = "0:9999" ] \ - || _p="$_p;a numeric chown did not take" + ok "a broken entry names its directory, and suggests nothing dangerous" +fi + +# ---- a commented block does not move the stream (1.14.39) ------------------ # +# gegl's generated script had an EMPTY build phase and did the whole build +# inside install_pkg, so when meson failed the following `ninja install` ran +# anyway: ninja: error: loading 'build.ninja': No such file or directory +# +# The rule "everything after a root block belongs after it" (GLib's +# user -> root -> user interleaving) is right for a root step that RUNS. +# gegl's first root block is conditional -- "If you are installing over a +# previous version ... remove it" -- so it is emitted COMMENTED, and nothing +# has happened; but it flipped the latch anyway. +_p="" +_b39="$(dirname "$LFS_TOOL")/blfs" +_bk39="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk39="$_c" && break +done +if [ -n "$_bk39" ]; then + _sd39="$T/streams"; mkdir -p "$_sd39" + python3 "$_b39" --book-file "$_bk39" script gegl glib2 -o "$_sd39" >/dev/null 2>&1 + _g="$(ls "$_sd39"/install_gegl* 2>/dev/null | head -1)" + if [ -n "$_g" ]; then + _gb="$(sed -n '/^build_pkg() {/,/^}/p' "$_g")" + _gi="$(sed -n '/^install_pkg() {/,/^}/p' "$_g")" + printf '%s' "$_gb" | grep -q 'meson setup' \ + || _p="$_p;gegl still builds nothing in its build phase" + printf '%s' "$_gb" | grep -qw 'ninja' \ + || _p="$_p;the compile step is not in the build phase" + printf '%s' "$_gi" | grep -q 'ninja install' \ + || _p="$_p;the install step left the install phase" + printf '%s' "$_gi" | grep -q 'meson setup' \ + && _p="$_p;the build is still being done in the install phase" + # the book's own conditional line stays, commented + grep -q '# rm -f /usr/lib/gegl-0.4/vector-fill.so' "$_g" \ + || _p="$_p;the conditional root command was dropped instead of commented" else - _p="$_p;even a numeric chown fails here -- fixture is not usable" + _p="$_p;could not generate the gegl script" + fi + # ...and a root block that REALLY runs still moves everything after it: + # GLib builds, installs gobject-introspection as root, then reconfigures + _gl="$(ls "$_sd39"/install_GLib* 2>/dev/null | head -1)" + if [ -n "$_gl" ]; then + sed -n '/^install_pkg() {/,/^}/p' "$_gl" | grep -q 'gi-build' \ + || _p="$_p;GLib's interleaved reconfigure moved back into the build phase" fi fi -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - # one helper, used by all three passes that set an install directory - _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ - || _p="$_p;the install-dir owner is not set by number" - _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ - || _p="$_p;the install-dir check reads names, which are UNKNOWN before 7.6" - for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do - _slice_fn "$helper_src" "$_fn" | grep -q 'set_install_dir_owner' \ - || _p="$_p;$_fn does not use the shared install-dir rule" - done - # and no pass may still reach for the names - for _fn in cmd_init_pkgusr ensure_install_dirs_writable _vfy_install_dirs; do - _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ - | grep -qE 'chown root:install|chgrp install ' \ - && _p="$_p;$_fn still sets an install dir by name" - done +# ...and gegl does not need the override at all: a subproject with NO BLFS +# page is allowed to fall back by name, so the protection stays everywhere +# else (1.14.40). --force-fallback-for overrides the wrap mode for exactly +# the named subprojects. +if [ -n "$_bk39" ]; then + _gm="$(grep -h 'meson setup' "$_sd39"/install_gegl* 2>/dev/null | head -1)" + case "$_gm" in + *"--force-fallback-for=libnsgif,poly2tri-c"*) ;; + *) _p="$_p;gegl cannot get the subprojects the book has no page for" ;; + esac + case "$_gm" in + *"--wrap-mode=nofallback"*) ;; + *) _p="$_p;the vendoring protection was dropped instead of narrowed" ;; + esac + # a package whose deps ARE in the book keeps the plain protection + python3 "$_b39" --book-file "$_bk39" script cairo -o "$_sd39" >/dev/null 2>&1 + _cm="$(grep -h 'meson setup' "$_sd39"/install_[Cc]airo* 2>/dev/null | head -1)" + case "$_cm" in + *force-fallback-for*) _p="$_p;cairo may vendor freetype again" ;; + esac + # every listed subproject must genuinely be absent from the book + BLFS_BOOK_FILE="$_bk39" python3 - "$(dirname "$LFS_TOOL")/packagemanager" "$_b39" <<'PYNOPAGE' || _p="$_p;a subproject on the allow-list has a book page and should be built as a package" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +b = m.SourceFileLoader('b', sys.argv[2]).load_module() +for pkg, subs in b._FALLBACK_ALLOWED.items(): + for sub in subs: + assert not pm.book_version(sub), (pkg, sub, "IS in the book") +PYNOPAGE fi +# the wrap-mode we add can also be overridden per package by hand +_pmM="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmM" "$T/wrapmode" <<'PYWRAP' || _p="$_p;wrap_mode cannot be set per package" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_gegl-0.4.66") +open(p, "w").write("build_pkg() {\nmeson setup --wrap-mode=nofallback --prefix=/usr ..\n}\n") +pm._machine_opts_for = lambda a: {"wrap_mode": "default"} +pm.apply_machine_opts("gegl", p) +body = open(p).read() +assert "--wrap-mode=default" in body, body +assert "nofallback" not in body, body +PYWRAP +# and the failure says who disabled it -- meson's message never does +sed -n '/A FALLBACK \*WE\* DISABLED/,/regenerate/p' "$helper_src" \ + | grep -q 'wrap_mode = default' \ + || _p="$_p;a disabled fallback does not say how to allow it" +sed -n '/A FALLBACK \*WE\* DISABLED/,/regenerate/p' "$helper_src" \ + | grep -q 'disabled BY US' \ + || _p="$_p;it does not say the flag is ours, not the book's" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" done else - ok "install directories are set by number, so a tree with no passwd works" + ok "a commented-out root block leaves the build in the build phase" fi -# ---- nothing is an orphan before there is a passwd file --------------------- # -# `find -nouser` asks whether a uid resolves. With no passwd file NOTHING -# resolves, so every file in the tree is an "orphan". On a restored chapter 5-6 -# tree that reported -# noowner / (no passwd entry) -> lfs -# noowner /dev (no passwd entry) -> lfs -# ... and 11797 more with no owner -# and `--fix` would have handed the ENTIRE tree -- root's own directories -# included -- to the build user in one pass. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do - _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ - || _p="$_p;$_fn runs before the user database exists" - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the ownership passes wait for book 7.6 to write /etc/passwd" - fi +# ---- a test runner is a command, not a substring (1.14.41) ----------------- # +# librsvg died on `Program 'cargo-cbuild' not found`, and cargo-c -- which +# provides it -- had installed nothing. Its one install line is +# install -vm755 target/release/cargo-{capi,cbuild,cinstall,ctest} /usr/bin/ +# and `\bctest\b` matched inside the brace list, because a comma is a word +# boundary. So the line that installs four binaries was commented as "runs +# the test suite", verify was right to call cargo-c not installed, and the +# failure surfaced two packages later. +_p="" +_b41="$(dirname "$LFS_TOOL")/blfs" +_bk41="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk41="$_c" && break +done +python3 - "$_b41" <<'PYDRIVER' || _p="$_p;the test-runner pattern still matches inside a word" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +hit = lambda l: bool(b._TEST_DRIVER_RE.search(l)) +# the reported line, and the shapes around it +assert not hit("install -vm755 target/release/cargo-{capi,cbuild,cinstall,ctest} /usr/bin/") +assert not hit("install -v -m755 foo-pytest /usr/bin") +assert not hit("cp build/mytest-helper /usr/bin") +# ...while a real test run is still recognised, however it is reached +assert hit("ctest --output-on-failure") +assert hit("cd build && ctest") +assert hit("pytest -v") +assert hit("make check") +assert hit("ninja test") +PYDRIVER +if [ -n "$_bk41" ]; then + _wd41="$T/drivers"; mkdir -p "$_wd41" + python3 "$_b41" --book-file "$_bk41" script cargo-c whois git -o "$_wd41" >/dev/null 2>&1 + _cg="$(ls "$_wd41"/install_cargo-c* 2>/dev/null | head -1)" + [ -n "$_cg" ] && { sed -n '/^install_pkg() {/,/^}/p' "$_cg" \ + | grep -q '^install -vm755 target/release/cargo-' \ + || _p="$_p;cargo-c still installs nothing"; } + # A BARE `make` BUILDS THE PACKAGE. Whois builds with `make` and installs + # with `make prefix=/usr install-whois`; there is no `make whois` target, + # so all three install lines were commented as building nothing. + _wh="$(ls "$_wd41"/install_Whois* 2>/dev/null | head -1)" + [ -n "$_wh" ] && { sed -n '/^install_pkg() {/,/^}/p' "$_wh" \ + | grep -q '^make prefix=/usr install-whois' \ + || _p="$_p;a package built by a bare make installs nothing"; } + # ...but a bare make does NOT build the manuals: git's install-man still + # waits for the conditional `make man`, which is why this filter exists + _gt="$(ls "$_wd41"/install_[Gg]it-2* 2>/dev/null | head -1)" + [ -n "$_gt" ] && { grep -q "^# make install-man" "$_gt" \ + || _p="$_p;git installs man pages it never built"; } + # and every page in the book installs SOMETHING + _none=0 + for _f in "$_wd41"/install_*; do + sed -n '/^install_pkg() {/,/^}/p' "$_f" | grep -vE '^\s*#|^\s*$|install_pkg|^\}' \ + | grep -qv '^:$' || _none=$((_none + 1)) + done + [ "$_none" = 0 ] || _p="$_p;$_none of the generated scripts install nothing" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a test runner is recognised as a command, and every page installs something" fi -# ---- nothing privileged may need a name to resolve -------------------------- # -# init-pkgusr runs BEFORE book 7.6 creates /etc/passwd. The prompt in there -# literally reads "I have no name!" because not even root resolves. So -# `chown root:install` fails on every directory: -# # 0 install directories are now root:install (group-writable) -# # could not set the group on: /bin /sbin /lib /lib64 /etc /usr ... -# and every package user is then locked out of /usr/bin for the whole build. +# ---- one answer per failure (1.14.42) -------------------------------------- # +# gtk4 stopped on `Dependency 'libtiff-4' is required but not found`, and the +# box told the user to allow a vendored copy: +# [gtk4] +# wrap_mode = default +# libtiff-4 is the pkg-config module of libtiff -- a BLFS package, installed +# on that system. Following the advice would have built a second libtiff +# inside gtk4 and buried a pkg-config problem. # -# Ownership is stored as a NUMBER. uid 0 is root whether anything says so or -# not, and the install gid is ours -- init-pkgusr writes /etc/group by hand, so -# the group can exist long before any passwd entry does. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" set_install_dir_owner | grep -q '0:\$INSTALL_GID' \ - || _p="$_p;an install directory is set by name, which fails before 7.6" - _slice_fn "$helper_src" install_dir_owner_ok | grep -q "%u:%g" \ - || _p="$_p;the readback uses names, so a correct directory reads as wrong" - # No privileged pass may reach for the NAMES. Match the INVOCATION, not the - # string: these functions legitimately print "now root:install" and carry - # trailing comments saying the same, and matching those made the test - # impossible to satisfy while changing nothing about the behaviour. - for _fn in cmd_init_pkgusr _vfy_install_dirs ensure_install_dirs_writable; do - _slice_fn "$helper_src" "$_fn" | grep -vE '^[[:space:]]*#' \ - | grep -qE '(real_)?(chown|chgrp)( +-[a-zA-Z]+)* +(root:install|install) ' \ - && _p="$_p;$_fn still sets the group by name, which fails before 7.6" - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "install directories are set by number, so 7.6 is not a prerequisite" - fi +# Two faults: meson's SECOND wording ("'x' is required but not found", single +# quotes) matched none of the extraction patterns, so the whole .pc diagnosis +# was skipped and only the fallback note remained; and the fallback note ran +# before that diagnosis, so it could not know better. +_p="" +_hlpG="$helper_src" +_dr42="$T/onedep"; mkdir -p "$_dr42/root/usr/lib" +python3 - "$_hlpG" > "$_dr42/rep.sh" <<'PYSLICE5' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { [ "$1" = "p_libtiff" ]; }') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('s="/x"; staged="/x"; auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE5 +bash -n "$_dr42/rep.sh" || _p="$_p;the report does not parse" +printf "Not looking for a fallback subproject for the dependency libtiff-4 because:\nUse of fallback dependencies is disabled.\n../meson.build:9: ERROR: Dependency 'libtiff-4' is required but not found.\n" \ + > "$_dr42/gtk4.log" +_o="$(bash -c '. "'"$_dr42"'/rep.sh"; SNAP_ROOT="'"$_dr42"'/root"; report gtk4 all "'"$_dr42"'/gtk4.log"' 2>&1)" +# the module name is resolved to the package that owns it +case "$_o" in + *"pkg-config name of 'libtiff'"*) ;; + *) _p="$_p;a pkg-config module name is not traced to its package" ;; +esac +case "$_o" in + *"install libtiff --run --reinstall"*) ;; + *) _p="$_p;no way offered to rebuild the package that owns it" ;; +esac +# ...and the WRONG advice is not given: libtiff is ours, not a subproject +case "$_o" in + *"wrap_mode = default"*) _p="$_p;it still offers to vendor a package we have" ;; +esac +case "$_o" in + *"Nothing on this system provides"*) + _p="$_p;it says the provider is installed AND that nothing provides it" ;; +esac +# a dependency nothing can supply still gets the fallback advice +printf "Not looking for a fallback subproject for the dependency libnsgif because:\nUse of fallback dependencies is disabled.\n../meson.build:384: ERROR: Dependency 'libnsgif' is required but not found.\n" \ + > "$_dr42/gegl.log" +_o2="$(bash -c '. "'"$_dr42"'/rep.sh"; SNAP_ROOT="'"$_dr42"'/root"; report gegl all "'"$_dr42"'/gegl.log"' 2>&1)" +case "$_o2" in + *"wrap_mode = default"*) ;; + *) _p="$_p;a subproject nothing provides no longer gets the fallback advice" ;; +esac +case "$_o2" in + *"pkg-config name of"*) _p="$_p;it invents an owner for a name nothing provides" ;; +esac +# all four spellings of "not found" reach the diagnosis +_dep_block="$(sed -n '/A DEPENDENCY THE BUILD SYSTEM COULD NOT FIND/,/local _dep_provider/p' "$_hlpG")" +for _w in 'is required but not found' 'No package' 'Package requirements' '" not found'; do + printf '%s' "$_dep_block" | grep -qF "$_w" \ + || _p="$_p;the wording '$_w' is not recognised" +done +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a missing dependency gets one answer, and it is the right one" +fi + +# ---- not installed is not the same as not a package (1.14.43) -------------- # +# gtk4 wanted `libtiff-4` on a system with NO p_libtiff at all. The account +# check found nothing, so the box said "nothing provides it -- packagemanager +# install libtiff-4", naming a package the book does not have, and then +# offered to vendor a copy. Two wrong answers where the right one was one +# string operation away: libtiff-4 is the pkg-config module of libtiff. +_p="" +_hlpH="$helper_src" +_dr43="$T/guessdep"; mkdir -p "$_dr43" +python3 - "$_hlpH" > "$_dr43/rep.sh" <<'PYSLICE6' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; } # nothing installed') +print('phase_done() { return 1; }; missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE6 +printf "Not looking for a fallback subproject for the dependency libtiff-4 because:\nUse of fallback dependencies is disabled.\n../meson.build:9: ERROR: Dependency 'libtiff-4' is required but not found.\n" \ + > "$_dr43/gtk4.log" +_o="$(bash -c '. "'"$_dr43"'/rep.sh"; report gtk4 all "'"$_dr43"'/gtk4.log"' 2>&1)" +case "$_o" in + *"pkg-config module of"*"'libtiff'"*) ;; + *) _p="$_p;a module name is not traced to its package when nothing is installed" ;; +esac +case "$_o" in + *"install libtiff --run --recursive"*) ;; + *) _p="$_p;it does not offer to install the package that provides it" ;; +esac +case "$_o" in + *"install libtiff-4 --run"*) _p="$_p;it still names a package the book cannot have" ;; +esac +case "$_o" in + *"wrap_mode = default"*) _p="$_p;it still offers to vendor a real package" ;; +esac +# a name with no version suffix has no better guess: the fallback IS the answer +printf "Not looking for a fallback subproject for the dependency libnsgif because:\nUse of fallback dependencies is disabled.\n../meson.build:384: ERROR: Dependency 'libnsgif' is required but not found.\n" \ + > "$_dr43/gegl.log" +_o2="$(bash -c '. "'"$_dr43"'/rep.sh"; report gegl all "'"$_dr43"'/gegl.log"' 2>&1)" +case "$_o2" in + *"wrap_mode = default"*) ;; + *) _p="$_p;a subproject with no package behind it lost the fallback advice" ;; +esac +case "$_o2" in + *"pkg-config module of"*) _p="$_p;it invented a package name for libnsgif" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an absent dependency is named as the package that would provide it" +fi + +# ---- a ref that does not exist is a typo, not a build error (1.14.44) ------ # +# A stack entry pinned usrsctp to ref=1.0.0, which does not exist upstream +# (the tags are 0.9.3.0, 0.9.4.0, 0.9.5.0). All the person saw was git's own +# error: pathspec '1.0.0' did not match any file(s) known to git +# followed by "phase 'unpack' FAILED" -- which reads like a build failure and +# sends them to a log. The repository is right there and knows what it has. +_p="" +_gr="$T/gitref"; mkdir -p "$_gr/src" +if command -v git >/dev/null 2>&1; then + ( cd "$_gr" && git init -q repo && cd repo \ + && git config user.email t@t && git config user.name t \ + && echo x > f && git add f && git commit -qm one \ + && git tag v0.9.5 && git tag v1.0-real ) >/dev/null 2>&1 + { + echo '#!/bin/bash' + echo 'name="demo"' + echo 'name_version="demo-git"' + echo 'is_git="1"' + echo "link=\"$_gr/repo\"" + echo 'git_ref="1.0.0"' + echo 'build_pkg() { :; }' + echo 'install_pkg() { :; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' + } > "$_gr/install_demo" + _o="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_gr/src" \ + bash "$_gr/install_demo" unpack 2>&1)" + case "$_o" in + *"is not a tag, branch or commit"*) ;; + *) _p="$_p;a missing git ref is not explained: $_o" ;; + esac + case "$_o" in + *"v1.0-real"*) ;; + *) _p="$_p;the tags the repository DOES have are not shown" ;; + esac + case "$_o" in + *"Fix ref= in the stack entry"*) ;; + *) _p="$_p;it does not say where the wrong value came from" ;; + esac + # and the phase still fails: explaining is not excusing + env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_gr/src" \ + bash "$_gr/install_demo" unpack >/dev/null 2>&1 \ + && _p="$_p;a bad ref no longer fails the phase" + # a ref that DOES exist still works + sed -i 's/^git_ref="1.0.0"/git_ref="v0.9.5"/' "$_gr/install_demo" + env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_gr/src" \ + bash "$_gr/install_demo" unpack >/dev/null 2>&1 \ + || _p="$_p;a valid ref no longer checks out" +fi +# the stack's own pin was the wrong one and is corrected +_ss="$(dirname "$LFS_TOOL")/stacks/sway.stack" +[ -f "$_ss" ] && { grep -q 'usrsctp.git ref=1.0.0' "$_ss" \ + && _p="$_p;the stack still pins a usrsctp tag that does not exist"; } +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a git ref that does not exist names the tags that do" +fi + +# ---- one flag, and the same one every time (1.14.45) ----------------------- # +# webrtc-audio-processing stopped with +# ../webrtc/rtc_base/trace_event.h:202:18: error: 'uint8_t' does not name a type +# which is not a broken package: GCC 13 stopped pulling <cstdint> in through +# other headers, so code written before that no longer compiles. The box said +# only "if it needs a build flag, put it in machine.conf" -- true, and no help +# at all when the flag is always the same one. +_p="" +_hlpI="$helper_src" +_pmN="$(dirname "$LFS_TOOL")/packagemanager" +_st="$T/stdint"; mkdir -p "$_st" +python3 - "$_hlpI" > "$_st/rep.sh" <<'PYSLICE7' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE7 +printf "../webrtc/rtc_base/trace_event.h:202:18: error: 'uint8_t' does not name a type\n../webrtc/rtc_base/foo.cc:9:1: note: here\n" \ + > "$_st/cpp.log" +_o="$(bash -c '. "'"$_st"'/rep.sh"; report webrtc-audio-processing all "'"$_st"'/cpp.log"' 2>&1)" +case "$_o" in + *"GCC 13+ change"*) ;; + *) _p="$_p;the stdint compile error is not recognised" ;; +esac +case "$_o" in + *"cpp_args = -include cstdint"*) ;; + *) _p="$_p;a C++ failure is not given the C++ flag" ;; +esac +# a C source gets the C spelling, not the C++ one +printf "../src/foo.c:12:5: error: 'uint32_t' does not name a type\n" > "$_st/c.log" +_o2="$(bash -c '. "'"$_st"'/rep.sh"; report demo all "'"$_st"'/c.log"' 2>&1)" +case "$_o2" in + *"c_args = -include stdint.h"*) ;; + *) _p="$_p;a C failure is given the C++ flag" ;; +esac +# an ordinary compile error keeps the generic advice and gains nothing +printf "../src/foo.c:12:5: error: too few arguments to function 'bar'\n" > "$_st/plain.log" +_o3="$(bash -c '. "'"$_st"'/rep.sh"; report demo all "'"$_st"'/plain.log"' 2>&1)" +case "$_o3" in + *"GCC 13+ change"*) _p="$_p;every compile error is blamed on cstdint" ;; +esac +# ...and the flag it recommends must survive being written into the script: +# a value with a space is ONE value +python3 - "$_pmN" "$_st" <<'PYQUOTE2' || _p="$_p;a multi-word option value is split by the shell" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +p = os.path.join(sys.argv[2], "install_w") +open(p, "w").write("build_pkg() {\nmeson setup build --prefix=/usr\n}\n") +pm._machine_opts_for = lambda a: {"cpp_args": "-include cstdint"} +pm.apply_machine_opts("webrtc-audio-processing", p) +body = open(p).read() +assert "-Dcpp_args='-include cstdint'" in body, body +# and applying it twice does not nest the quotes +pm.apply_machine_opts("webrtc-audio-processing", p) +assert body == open(p).read(), open(p).read() +PYQUOTE2 +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the GCC 13 cstdint break is named, with the flag that fixes it" +fi + +# ---- advice that nothing reads (1.14.46) ----------------------------------- # +# The 1.14.45 box told the user to put +# [webrtc-audio-processing] +# cpp_args = -include cstdint +# in machine.conf. webrtc-audio-processing is a GIT stack entry, and +# apply_machine_opts was called only from resolve_script -- the BOOK path. +# Nothing would have read that section. The advice was correct and inert, +# which is worse than wrong: it looks like it worked until the same error +# comes back. +_p="" +_pmO="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmO" <<'PYMOPT' || _p="$_p;machine.conf does not reach a git stack entry" +import sys, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +e = {"target": "webrtc-audio-processing", "kind": "git", + "opts": {"url": "https://example.invalid/w.git", "ref": "v2.1", + "build": "meson setup build --prefix=/usr; ninja -C build", + "install": "ninja -C build install"}} +p, _ = pm._stack_git_script(e, tempfile.mkdtemp()) +pm._machine_opts_for = lambda a: {"cpp_args": "-include cstdint"} +pm.apply_machine_opts("webrtc-audio-processing", p) +body = open(p).read() +assert "-Dcpp_args='-include cstdint'" in body, body +PYMOPT +# the stack's git/tar branch calls it, for the same reason a book package does +sed -n '/elif kind in ("git", "tar")/,/install_local(target, spath/p' "$_pmO" \ + | grep -q 'apply_machine_opts(target, spath)' \ + || _p="$_p;a git or tar entry is built without its machine.conf options" +# every path that RUNS a script applies them: two in resolve_script, one here +[ "$(grep -c 'apply_machine_opts(' "$_pmO")" -ge 4 ] \ + || _p="$_p;some path still runs a script without applying machine.conf" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "machine.conf reaches every script, whatever wrote it" +fi + +# ---- one typo must not disable a whole file (1.14.47) ---------------------- # +# The user followed the advice twice, so machine.conf ended up with +# [webrtc-audio-processing] +# cpp_args = -include cstdint +# [webrtc-audio-processing] +# cpp_args = -include cstdint +# configparser raises on a repeated section, and _read_conf_section discarded +# the ENTIRE file on any error. So the new section did nothing -- and neither +# did [mesa], whose driver selection quietly reverted to "build everything" +# because of a duplicate heading further down the file. +_p="" +_pmP="$(dirname "$LFS_TOOL")/packagemanager" +_mc="$T/machineconf"; mkdir -p "$_mc" +printf '[mesa]\nplatforms = x11,wayland\n\n[webrtc-audio-processing]\ncpp_args = -include cstdint\n[webrtc-audio-processing]\ncpp_args = -include cstdint\n' \ + > "$_mc/dup.conf" +PM_MACHINE_CONF="$_mc/dup.conf" python3 - "$_pmP" <<'PYDUP' || _p="$_p;a repeated section still discards the whole file" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +# the repeated section itself is read +assert pm._machine_opts_for("webrtc-audio-processing") == {"cpp_args": "-include cstdint"}, \ + pm._machine_opts_for("webrtc-audio-processing") +# ...and so is every OTHER section, which is the part that was dangerous +assert pm._machine_opts_for("mesa").get("platforms") == "x11,wayland", \ + pm._machine_opts_for("mesa") +PYDUP +# a file that is genuinely malformed still reports, and says how far the +# damage reaches +printf '[mesa]\nplatforms = x11\n[oops\nbroken = yes\n' > "$_mc/bad.conf" +_o="$(PM_MACHINE_CONF="$_mc/bad.conf" python3 - "$_pmP" 2>&1 <<'PYBAD' +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._machine_opts_for("mesa") +PYBAD +)" +case "$_o" in + *"NO options from that file were applied"*) ;; + *) _p="$_p;a broken machine.conf does not say that everything was skipped" ;; +esac +sed -n '/def _read_conf_section/,/^def /p' "$_pmP" | grep -q 'strict=False' \ + || _p="$_p;the parser is strict again, so a duplicate heading breaks the file" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a repeated section is merged, and a broken file says what it cost" +fi + +# ---- the wrong book, found by the package (1.14.48) ------------------------ # +# BlueZ stopped with +# configure: error: systemd system unit directory is required +# on a SysV system. Nothing was broken: the SYSTEMD book's page produced the +# script, and its configure line omits --disable-systemd because on a systemd +# machine there is nothing to disable. Every run had been printing "no book +# matched 'stable-sysv'; using the cached BLFS-BOOK-...-systemd" -- this is +# what that note costs when it scrolls past. +_p="" +_hlpJ="$helper_src" +_sy="$T/sysvbook"; mkdir -p "$_sy" +python3 - "$_hlpJ" > "$_sy/rep.sh" <<'PYSLICE8' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE8 +printf "checking systemd system unit dir... configure: error: systemd system unit directory is required\n" \ + > "$_sy/bluez.log" +# on a system WITHOUT systemctl, the book mismatch is named +_o="$(bash -c 'command() { if [ "${2:-}" = systemctl ]; then return 1; fi; builtin command "$@"; } +. "'"$_sy"'/rep.sh"; report bluez all "'"$_sy"'/bluez.log"' 2>&1)" +case "$_o" in + *"blfs fetch stable-sysv"*) ;; + *) _p="$_p;a systemd-only configure check does not point at the sysv book" ;; +esac +case "$_o" in + *"configure_args = --disable-systemd"*) ;; + *) _p="$_p;no offline fallback is offered for a machine with no network" ;; +esac +case "$_o" in + *"every later package too"*) ;; + *) _p="$_p;it does not say the book fixes more than this one package" ;; +esac +# ...and on a systemd system it says nothing: there the page is correct +if command -v systemctl >/dev/null 2>&1; then + _o2="$(bash -c '. "'"$_sy"'/rep.sh"; report bluez all "'"$_sy"'/bluez.log"' 2>&1)" + case "$_o2" in + *"This is a SysV system"*) + _p="$_p;it blames the book on a systemd system, where the page is right" ;; + esac +fi +# the check must be OUTSIDE the compile-error branch: a configure error has no +# file:line:col to match, which is why the first attempt at this never fired +_pos_dep="$(grep -n 'A DEPENDENCY THE BUILD SYSTEM COULD NOT FIND' "$_hlpJ" | cut -d: -f1)" +_pos_book="$(grep -n 'THE WRONG BOOK, FOUND BY THE PACKAGE' "$_hlpJ" | cut -d: -f1)" +_pos_compile="$(grep -n 'This is a COMPILE error' "$_hlpJ" | cut -d: -f1)" +[ -n "$_pos_book" ] && [ -n "$_pos_compile" ] && [ "$_pos_book" -gt "$_pos_compile" ] \ + && [ -n "$_pos_dep" ] && [ "$_pos_book" -lt "$_pos_dep" ] \ + || _p="$_p;the book check is not where a configure failure can reach it" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a systemd-only configure check names the book, not the package" +fi - # and the orphan pass must not fire before there is a database to be - # orphaned FROM. With no passwd file `find -nouser` matches everything: a - # real run reported 11802 orphans including / and /proc, and --fix would have - # handed the entire tree to the build user in one pass. - _p="" - case "$(_slice_fn "$helper_src" _vfy_orphans)" in - *_user_db_ready*) ;; - *) _p="$_p;the orphan pass would hand the whole tree to the build user before 7.6" ;; - esac - case "$(_slice_fn "$helper_src" _has_orphaned_files)" in - *_user_db_ready*) ;; - *) _p="$_p;a tree with no passwd file is reported as entirely orphaned" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "an orphan needs a user database to be missing from" - fi +# ---- see the difference, do not guess it (1.14.48) ------------------------- # +# BlueZ stopped with "systemd system unit directory is required" on a SysV +# system: the SYSTEMD book's page was used, and its configure line has no +# --disable-systemd because on a systemd machine there is nothing to disable. +# Every run already warned "no book matched 'stable-sysv'"; this is what that +# note costs. Per-package machine.conf entries paper over one page at a time, +# and only after each has failed. +_p="" +_b48="$(dirname "$LFS_TOOL")/blfs" +_bk48="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk48="$_c" && break +done +python3 "$_b48" diff-init --help >/dev/null 2>&1 \ + || _p="$_p;there is no way to compare two books" +if [ -n "$_bk48" ]; then + # a copy with one page changed the way the sysv book changes it + _alt="$T/BLFS-BOOK-13.0-sysv-nochunks.html" + python3 - "$_bk48" "$_alt" <<'PYDOCTOR' +import sys +s = open(sys.argv[1], encoding="utf-8", errors="replace").read() +i = s.index('id="bluez"') +seg = s[i:i + 40000] +alt = seg.replace('--enable-library', '--enable-library --disable-systemd', 1) +open(sys.argv[2], "w", encoding="utf-8").write(s[:i] + alt + s[i + 40000:]) +PYDOCTOR + # one package: the actual command difference, as a diff + _d="$(python3 "$_b48" --book-file "$_bk48" diff-init "$_alt" bluez 2>&1)" + case "$_d" in + *"--disable-systemd"*) ;; + *) _p="$_p;the per-package diff does not show the changed command: $_d" ;; + esac + case "$_d" in + *"+++"*"---"*|*"---"*"+++"*) ;; + *) _p="$_p;the output is not a diff, so it cannot be read as one" ;; + esac + # ...and the survey: which pages differ at all, before anything is built + _s="$(timeout 300 python3 "$_b48" --book-file "$_bk48" diff-init "$_alt" 2>&1)" + case "$_s" in + *"bluez"*"commands differ"*) ;; + *) _p="$_p;the survey does not name the page that differs" ;; + esac + case "$_s" in + *"1 of "*" shared pages differ"*) ;; + *) _p="$_p;the survey does not count what it compared" ;; + esac + # comparing a book with ITSELF is refused outright (1.14.49): answering + # "0 differ" for it is true and useless, and hid a fallback that had + # silently picked the same book for both sides + _z="$(timeout 300 python3 "$_b48" --book-file "$_bk48" diff-init "$_bk48" 2>&1)" + case "$_z" in + *"same file"*) ;; + *) _p="$_p;comparing a book with itself is not refused" ;; + esac +fi +# and the failure box points at the book, not only at a per-package patch +sed -n '/THE WRONG BOOK, FOUND BY THE PACKAGE INSTEAD OF BY US/,/configure_args = --disable-systemd/p' \ + "$helper_src" | grep -q 'blfs fetch stable-sysv' \ + || _p="$_p;a systemd-only configure failure does not mention the sysv book" +sed -n '/THE WRONG BOOK, FOUND BY THE PACKAGE INSTEAD OF BY US/,/configure_args/p' \ + "$helper_src" | grep -q 'command -v systemctl' \ + || _p="$_p;it would say this on a systemd system too" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "two books can be compared, page by page, before anything is built" fi -# ---- the build user's uid is not a range ------------------------------------ # -# Book 4.3's useradd pins no uid, so the host's `lfs` account lands wherever the -# host had a gap -- 10753 on a Debian host, INSIDE the package-user range. Any -# rule that identifies the build user by "uid outside 10000+" therefore excludes -# the very account it is looking for. That is the same mistake _normalize_sources -# exists to correct, and _handover_needed walked straight into it: it reported -# # 7.2 already done (tree owned by root) -# while /bin -> usr/bin was still lfs:lfs. -python3 - "$LFS_TOOL" <<'PYUIDRANGE' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -m = re.search(r"^def _handover_needed\(.*?(?=\n\ndef )", src, re.S | re.M) -if not m: - problems.append("_handover_needed is gone") -else: - body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) - if "PKG_UID_MIN" in body: - problems.append("_handover_needed still identifies the build user by uid range") -m = re.search(r"^def _handover_is_safe\(.*?(?=\n\ndef )", src, re.S | re.M) -if not m: - problems.append("_handover_is_safe is gone") -else: - body = re.sub(r"^\s*#.*$", "", m.group(0), flags=re.M) - if "_pkg_users_in_tree" not in body: - problems.append("_handover_is_safe does not read the tree's own passwd") -if not re.search(r"^def _pkg_users_in_tree\(", src, re.M): - problems.append("there is no _pkg_users_in_tree") -else: - m = re.search(r"^def _pkg_users_in_tree\(.*?(?=\n\ndef |\Z)", src, re.S | re.M) - if m and "pkgusr_prefix" not in m.group(0): - problems.append("_pkg_users_in_tree does not identify accounts by prefix") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS the build user is identified by the tree, never by a uid range") -PYUIDRANGE -_count_rc $? +# ---- a fetch that fetches nothing is not "cached" (1.14.49) ---------------- # +# $ blfs fetch stable-sysv +# note: no book matched 'stable-sysv'; using the cached ...-systemd +# cached. +# Nothing was fetched, and it said success. THERE IS NO 13.0 SYSV BOOK -- +# BLFS stopped publishing that edition after 12.4 (the user's own `blfs books` +# lists 13.0-systemd and 13.1-systemd, and `stable` = 12.4 [sysv]). Then +# $ blfs diff-init 12.4 +# 0 of 947 shared pages differ +# which was true and useless: the default selector has no book, so it fell +# back to the newest cached one -- 12.4, the very book being compared against. +_p="" +_b49="$(dirname "$LFS_TOOL")/blfs" +_bk49="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk49="$_c" && break +done +# a selector nobody publishes fails, and says so +_o="$(python3 "$_b49" fetch definitely-not-a-published-book 2>&1)" +case "$_o" in + *"no book is published as"*) ;; + *) _p="$_p;fetching a nonexistent book does not report a problem" ;; +esac +case "$_o" in + *cached.*) _p="$_p;a fetch that fetched nothing still claims success" ;; +esac +if [ -n "$_bk49" ]; then + # comparing a book with itself is refused, not answered with "0 differ" + _s="$(python3 "$_b49" --book-file "$_bk49" diff-init "$_bk49" 2>&1)" + case "$_s" in + *"same file"*) ;; + *) _p="$_p;a book compared with itself reports a meaningless zero" ;; + esac + # ...and both sides are always named, so the fallback cannot hide + case "$_s" in + *"comparing:"*) ;; + *) _p="$_p;the two books being compared are not named" ;; + esac + # across versions, only the init-related differences are shown by default + _alt="$T/BLFS-BOOK-13.0-sysv-nochunks.html" + [ -f "$_alt" ] || python3 - "$_bk49" "$_alt" <<'PYDOC2' +import sys +s = open(sys.argv[1], encoding="utf-8", errors="replace").read() +i = s.index('id="bluez"') +seg = s[i:i + 40000] +open(sys.argv[2], "w", encoding="utf-8").write( + s[:i] + seg.replace('--enable-library', '--enable-library --disable-systemd', 1) + + s[i + 40000:]) +PYDOC2 + _d="$(timeout 400 python3 "$_b49" --book-file "$_bk49" diff-init "$_alt" 2>&1)" + case "$_d" in + *"differ in their init handling"*) ;; + *) _p="$_p;the survey does not say it is filtering to init differences" ;; + esac + case "$_d" in + *"--all compares everything"*) ;; + *) _p="$_p;there is no way to see the differences it filtered out" ;; + esac +fi +python3 "$_b49" diff-init --help 2>&1 | grep -q '\-\-all' \ + || _p="$_p;--all is not offered" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a fetch that finds no book says so, and no book is compared with itself" +fi -# ---- one step where ownership becomes real ---------------------------------- # -# Before book 7.6 there is no /etc/passwd, so no name resolves -- not even -# root's; the prompt in the chroot literally reads "I have no name!". Every -# ownership question asked before that point gets a confident wrong answer: -# -# chown root:install fails on every directory -> "0 install directories" -# find -nouser matches every file in the tree -> 11802 "orphans" -# -# Three separate bugs, one cause: work that needs a user database was being done -# before there was one. So there is one gate now. Nothing touches ownership -# before it; everything maintains ownership after it. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - for _fn in ownership_established ownership_possible establish_ownership \ - _ownership_checkpoint cmd_establish_ownership; do - _slice_fn "$helper_src" "$_fn" | grep -q . \ - || _p="$_p;there is no $_fn" - done - # the gate must test the user database and the tools it needs - _op="$(_slice_fn "$helper_src" ownership_possible)" - case "$_op" in - *_user_db_ready*) ;; - *) _p="$_p;the gate does not wait for the user database" ;; - esac - case "$_op" in - *chown*) ;; - *) _p="$_p;the gate does not check that chown is available" ;; - esac - # it must be durable, so a resumed or restored build does not repeat it - grep -q '^OWNERSHIP_MARK=' "$helper_src" \ - || _p="$_p;crossing the epoch is not recorded, so it repeats" - case "$(_slice_fn "$helper_src" establish_ownership)" in - *'> "$OWNERSHIP_MARK"'*) ;; - *) _p="$_p;establish_ownership never writes its marker" ;; - esac - # and everything ownership-related must be inside it, not scattered - _eo="$(_slice_fn "$helper_src" establish_ownership)" - for _need in cmd_init_pkgusr _vfy_build_user_leftovers _vfy_manifest_ownership \ - _adopt_pkgusr_tools _vfy_orphans; do - case "$_eo" in - *"$_need"*) ;; - *) _p="$_p;$_need does not happen at the ownership epoch" ;; - esac - done - # reachable by hand as well as automatically - grep -q 'establish-ownership)' "$helper_src" \ - || _p="$_p;there is no way to run the ownership step by hand" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "ownership is established once, at one step, and recorded" - fi +# ---- what differs AND concerns you (1.14.50) ------------------------------- # +# The survey came back with 47 of 929 pages differing in their init handling. +# True, and most of them are for software this machine does not have and never +# will -- GNOME, samba, qemu, postgresql. The useful question is not "what +# differs" but "what differs AND is on my system", and the answer is on disk: +# a package with an account has been built or is about to be. +_p="" +_b50="$(dirname "$LFS_TOOL")/blfs" +python3 "$_b50" diff-init --help 2>&1 | grep -q '\-\-installed' \ + || _p="$_p;there is no way to narrow the survey to this machine" +_bk50="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk50="$_c" && break +done +if [ -n "$_bk50" ]; then + _alt50="$T/BLFS-BOOK-13.0-sysv-nochunks.html" + [ -f "$_alt50" ] || python3 - "$_bk50" "$_alt50" <<'PYDOC3' +import sys +s = open(sys.argv[1], encoding="utf-8", errors="replace").read() +i = s.index('id="bluez"') +seg = s[i:i + 40000] +open(sys.argv[2], "w", encoding="utf-8").write( + s[:i] + seg.replace('--enable-library', '--enable-library --disable-systemd', 1) + + s[i + 40000:]) +PYDOC3 + _fr="$T/fakeroot"; mkdir -p "$_fr/p_bluez" "$_fr/p_dbus" "$_fr/p_polkit" + _o="$(LFS_PKGUSR_ROOT="$_fr" timeout 400 python3 "$_b50" \ + --book-file "$_bk50" diff-init "$_alt50" --installed 2>&1)" + # only the three accounts are considered, not all 954 pages + case "$_o" in + *"of 3 shared pages"*) ;; + *) _p="$_p;--installed does not narrow to the packages that exist here" ;; + esac + case "$_o" in + *bluez*) ;; + *) _p="$_p;the page that does differ was filtered out" ;; + esac + # and a machine with no package users says so instead of comparing nothing + _empty="$T/emptyroot"; mkdir -p "$_empty" + _o2="$(LFS_PKGUSR_ROOT="$_empty" timeout 400 python3 "$_b50" \ + --book-file "$_bk50" diff-init "$_alt50" --installed 2>&1)" + case "$_o2" in + *"nothing to filter by"*) ;; + *) _p="$_p;an empty package-user root silently compares nothing" ;; + esac +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the book comparison can be narrowed to the packages this machine has" +fi - # the passes that need a user database must refuse to run without one - _p="" - for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do - _slice_fn "$helper_src" "$_fn" | grep -q '_user_db_ready' \ - || _p="$_p;$_fn would run before there is a user database" - done - case "$(_slice_fn "$helper_src" _vfy_install_dirs)" in - *ownership_*) ;; - *) _p="$_p;verify would report 27 wrong install dirs on a chapter 5-6 tree" ;; - esac - # build-all must NOT set the package-user system up before the epoch -- - # that is what printed "0 install directories are now root:install" - case "$(_slice_fn "$helper_src" cmd_build_all)" in - *ownership_established*) ;; - *) _p="$_p;build-all still sets up ownership before 7.6 has run" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "nothing asks who owns what before there is a user database" - fi +# ---- judge the difference on lines that run (1.14.51) ---------------------- # +# The survey named eight pages on a real system; four were nothing. gcc +# differed only in a NOTE whose URL contains "13.0-systemd"; libxml2 in a NOTE +# saying "systemctl stop httpd.service"; rust in a NOTE about the "systemd +# journal". Prose, all of it -- the same mistake as 1.14.23, in a different +# filter: a commented line is not a command. +_p="" +_b51="$(dirname "$LFS_TOOL")/blfs" +python3 - "$_b51" <<'PYLIVE' || _p="$_p;the init filter still judges comment text" +import sys, difflib, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +live = lambda t: [l for l in t.splitlines() + if l.strip() and not l.strip().startswith("#")] +def is_init(old, new): + d = "\n".join(l for l in difflib.ndiff(live(old), live(new)) if l[:1] in "+-") + return bool(b._INIT_WORDS_RE.search(d)) +# the four that were noise, in the shapes they really had +assert not is_init("## NOTE: lfs/view/12.4/chapter08/gcc.html\nmake install", + "## NOTE: lfs/view/13.0-systemd/chapter08/gcc.html\nmake install") +assert not is_init("## NOTE: shut down the server: /etc/init.d/httpd stop\nninja install", + "## NOTE: shut down the server: systemctl stop httpd.service\nninja install") +assert not is_init("## NOTE: messages in the system log\n./x.py install", + "## NOTE: messages in the systemd journal\n./x.py install") +# ...and the three that were real +assert is_init("./configure --enable-library --disable-systemd &&\nmake", + "./configure --enable-library &&\nmake") +assert is_init("cat > /etc/cron.weekly/update-pki.sh << EOF\nEOF", + "systemctl enable update-pki.timer") +assert is_init("meson setup -D systemd=disabled ..", "meson setup ..") +PYLIVE +sed -n '/A COMMENTED LINE IS NOT A COMMAND -- HERE TOO/,/_INIT_WORDS_RE.search/p' "$_b51" \ + | grep -q 'startswith("#")' \ + || _p="$_p;the comparison does not strip comments before judging" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "an init difference is judged on the commands, not on the prose" fi -# ---- the build tree must not collide with the skeleton ---------------------- # -# init_package_user_home symlinks .bash_profile, .bashrc and `build` into every -# package user's home -- `build` being the hint's build helper script. The -# per-package build tree was first called `build` too, so `mkdir -p ~/build` hit -# the symlink and the build died two lines later complaining about the wrong -# thing: -# mkdir: cannot create directory '.../p_gettext/build': File exists -# cd: /usr/src/pkgusr/p_gettext/build: Not a directory -# The recovery code then read that as a permissions problem and retried twice -# before advising collector groups. None of it was true. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" pkgusr_skel_links | grep -q . \ - || _p="$_p;the skeleton links are not named in one place" - # the home skeleton must be built FROM that list, not from a second copy - case "$(_slice_fn "$helper_src" init_package_user_home)" in - *pkgusr_skel_links*) ;; - *) _p="$_p;init_package_user_home keeps its own copy of the link list" ;; - esac - # and the build subdir must not be one of them - _sub="$(grep -m1 '^PKGUSR_BUILD_SUBDIR=' "$helper_src" \ - | sed 's/.*:-\([a-z_]*\)}.*/\1/')" - if [ -z "$_sub" ]; then - _p="$_p;there is no PKGUSR_BUILD_SUBDIR" - else - for _l in $(_slice_fn "$helper_src" pkgusr_skel_links \ - | grep -oE "'[^']+'|[a-z._:]+" | tr ' ' '\n'); do - case "$_l" in - "$_sub":*|"$_sub") _p="$_p;the build tree is called '$_sub', which the home skeleton already links" ;; - esac - done - fi - # and a name that is taken must be reported HERE, not as a cd failure later - case "$(_slice_fn "$helper_src" cmd_build)" in - *'is not a directory'*) ;; - *) _p="$_p;a build tree that is not a directory fails somewhere else" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the build tree cannot collide with the package user's home skeleton" - fi +# ---- a boot script belongs to blfs-bootscripts (1.14.52) ------------------- # +# With the SysV book finally in use, BlueZ installed cleanly and then: +# ## Configuration +# make: *** No rule to make target 'install-bluetooth'. Stop. +# !! phase 'configure' FAILED (exit 2) +# `make install-bluetooth` is the SysV books' idiom for installing a boot +# script, and that target lives in the blfs-bootscripts package -- not in the +# package being built. So a finished, correct install ended as a failure. +# The systemd books have no such sections, which is why this appeared only +# after switching books. +_p="" +_b52="$(dirname "$LFS_TOOL")/blfs" +python3 - "$_b52" <<'PYBOOT' || _p="$_p;a boot-script install is still run in the wrong package" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +f = b._comment_bootscript_installs +skipped = lambda l: f(l).startswith("## SKIPPED") +# the reported case, and another of the same shape +assert skipped("make install-bluetooth") +assert skipped("make install-dbus") +# the package's OWN install is untouched +assert not skipped("make install") +assert not skipped("ninja install") +assert not skipped("make -C build install") +# ...and so are the real automake targets, which are NOT boot scripts +for t in ("strip", "exec", "data", "man", "html", "info"): + assert not skipped("make install-" + t), t +# an already-commented line is left alone +assert not skipped("# make install-bluetooth") +# the note says where the target actually lives and how to get there +out = f("make install-bluetooth") +assert "blfs-bootscripts" in out, out +assert "cd <blfs-bootscripts source> && make install-bluetooth" in out, out +# the original command survives as a comment, not deleted +assert "# make install-bluetooth" in out, out +PYBOOT +sed -n '/CACHE_VERSION = /p' "$_b52" | grep -qE 'CACHE_VERSION = (5[3-9]|[6-9][0-9])' \ + || _p="$_p;the cache version was not bumped for a parse change" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a boot-script install is explained, not run in the wrong package" +fi + +# ---- a fallback must not take you back a release (1.14.52) ----------------- # +# The user was building 13.0. They fetched the 12.4 book as a REFERENCE for +# SysV commands, and the next build produced BlueZ-5.83 -- 12.4's version. +# The configured default is 'stable-sysv', which no longer exists, so the +# fallback scored the cached books by init flavour: the 12.4 sysv book matched +# and won, and the 13.0 book it passed over was never mentioned. +_p="" +_b52="$(dirname "$LFS_TOOL")/blfs" +python3 - "$_b52" "$T/bookcache" <<'PYFALL' || _p="$_p;the fallback ignores versions" +import sys, os, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +names = ["BLFS-BOOK-12.4-nochunks.html", + "BLFS-BOOK-13.0-systemd-nochunks.html", + "BLFS-BOOK-11.0-nochunks.html"] +cached = [] +for n in names: + p = os.path.join(d, n) + open(p, "w").write("") + cached.append(p) +# the asked-for flavour still decides -- that is what the selector means +assert b._best_cached_for("stable-sysv", cached).endswith("12.4-nochunks.html") +assert b._best_cached_for("stable-systemd", cached).endswith("13.0-systemd-nochunks.html") +# ...but among books of the SAME flavour, the newest wins: a fallback must not +# quietly take a system back a release +assert b._best_cached_for("sysv", cached).endswith("12.4-nochunks.html"), \ + b._best_cached_for("sysv", cached) +# and versions sort as versions, not as strings ("13.0" > "12.4" > "9.1") +assert b._version_key("13.0") > b._version_key("12.4") > b._version_key("9.1") +assert b._version_key("unknown") < b._version_key("1.0") +PYFALL +# the note names the newer book it passed over, and how to pin it +sed -n '/NAME WHAT ELSE WAS ON DISK/,/set-default/p' "$_b52" \ + | grep -q 'a NEWER book is also cached' \ + || _p="$_p;the fallback does not mention a newer cached book" +sed -n '/NAME WHAT ELSE WAS ON DISK/,/set-default/p' "$_b52" \ + | grep -q 'blfs set-default' \ + || _p="$_p;it does not say how to pin the book to build from" +# a boot-script install belongs to blfs-bootscripts, not to the package +python3 - "$_b52" <<'PYBOOT' || _p="$_p;a SysV boot-script step is run in the wrong package" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +out = b._comment_bootscript_installs("make install-bluetooth") +assert out.startswith("## SKIPPED"), out +assert "blfs-bootscripts" in out, out +# the package's own install targets are untouched +for keep in ("make install", "make install-strip", "make install-data", + "make -C build install", "ninja install"): + assert not b._comment_bootscript_installs(keep).startswith("## SKIPPED"), keep +PYBOOT +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a book fallback names what it passed over, and never silently downgrades" fi -# ---- no function is defined twice ------------------------------------------ # -# `cmd_verify` was defined TWICE. bash keeps the last, so the tidy definition -# that called the _vfy_* helpers had never run -- and the surviving one carried -# its own inline copies of the manifest and install-dir rules, without any of -# the guards the helpers had grown. -# -# A finished 104/104 build therefore reported three things the real pass knows -# to leave alone, and --fix would have acted on all three: -# /usr/bin/hostname is p_inetutils want p_coreutils (inetutils owns it) -# /usr/share/info/dir is p_e2fsprogs want p_glibc (nobody owns it) -# /etc/group- is root want p_ncurses (useradd wrote it) -# -# A duplicate definition is invisible: both parse, one silently wins. -for _t in lfs-helper; do - _src="$(dirname "$LFS_TOOL")/$_t" - [ -f "$_src" ] || continue - _dups="$(grep -oE '^[a-z_][a-z0-9_]*\(\) \{' "$_src" | sort | uniq -d)" - if [ -n "$_dups" ]; then - printf '%s\n' "$_dups" | while IFS= read -r _d; do - bad "$_t defines ${_d%\(\) \{} more than once -- one of them never runs" - done - else - ok "$_t defines each function exactly once" - fi +# ---- a page that offers a choice must still build (1.14.53) ---------------- # +# WebKitGTK unpacked and then: +# ninja: error: loading 'build.ninja': No such file or directory +# Its page gives TWO complete builds -- "If you want to install the GTK-3 +# version, run the following commands", and the same for GTK-4 -- so each is +# introduced by a conditional and _conditional_guard commented BOTH. The +# build phase came out as `:` and the install phase ran `ninja install` +# against a directory with no build in it. +_p="" +_b53="$(dirname "$LFS_TOOL")/blfs" +python3 - "$_b53" <<'PYALT' || _p="$_p;a page whose build blocks are all conditional still builds nothing" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +# every block skipped -> the first is restored +skipped = ("## OPTIONAL, skipped -- the book says: If you want the GTK-3 version:\n" + "# mkdir build &&\n" + "# cd build &&\n" + "# cmake .. &&\n" + "# ninja\n" + ":") +out = b._enable_one_alternative(skipped) +live = [l for l in out.splitlines() + if l.strip() and not l.strip().startswith("#") and l.strip() != ":"] +assert any("cmake" in l for l in live), out +assert any("ninja" in l for l in live), out +assert "ENABLED HERE" in out, out +# a phase that already has real commands is untouched +keep = "mkdir build &&\ncd build &&\ncmake ..\n## OPTIONAL, skipped -- something else\n# make foo" +assert b._enable_one_alternative(keep) == keep +# and a phase that is empty for another reason (nothing skipped) stays empty +assert b._enable_one_alternative(":") == ":" +PYALT +_bk53="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk53="$_c" && break done - -# and verify must USE the helpers rather than carry its own copy of the rules -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _cv="$(_slice_fn "$helper_src" cmd_verify)" - _p="" - for _need in _vfy_install_dirs _vfy_manifest_ownership _vfy_orphans \ - _vfy_user_homes _vfy_build_user_leftovers _vfy_user_order; do - case "$_cv" in - *"$_need"*) ;; - *) _p="$_p;verify does not run $_need" ;; - esac - done - # no second copy of the ownership rule inline - case "$(printf '%s\n' "$_cv" | grep -vE '^[[:space:]]*#')" in - *'want $owner'*) _p="$_p;verify still carries its own copy of the ownership rule" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "verify runs the one implementation of each check" - fi +if [ -n "$_bk53" ]; then + _wd53="$T/alts"; mkdir -p "$_wd53" + python3 "$_b53" --book-file "$_bk53" script webkitgtk -o "$_wd53" >/dev/null 2>&1 + _w="$(ls "$_wd53"/install_WebKitGTK* 2>/dev/null | head -1)" + if [ -n "$_w" ]; then + _wb="$(sed -n '/^build_pkg() {/,/^}/p' "$_w" | grep -vE '^\s*#|^\s*$')" + printf '%s' "$_wb" | grep -q 'cmake' \ + || _p="$_p;webkitgtk still configures nothing" + printf '%s' "$_wb" | grep -qw 'ninja' \ + || _p="$_p;webkitgtk still compiles nothing" + bash -n "$_w" || _p="$_p;the restored block does not parse" + # the alternative it did NOT pick stays commented, and is findable + grep -q '# *-D USE_GTK4=ON' "$_w" \ + || _p="$_p;the other variant was lost rather than left commented" + else + _p="$_p;could not generate the webkitgtk script" + fi +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a page offering only conditional builds still builds one of them" fi -# ---- a collector group is named for the package ----------------------------- # -# It took the ACCOUNT name, so owner `p_perl` produced `nimgnu_p_perl` -- two -# prefixes, and the wrong two: `nimgnu_` says collector group, `p_` says package -# user, which it is not. The prompt offered it as a choice: -# 1) nimgnu_p_perl (everything owned by 'p_perl' -- fewer groups) -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( COLLECTOR_PREFIX=nimgnu; PKGUSR_PREFIX=p_ - eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" - eval "$(_slice_fn "$helper_src" collector_group_for)" - [ "$(collector_group_for p_perl)" = "nimgnu_perl" ] \ - || echo "a collector group still carries the package-user prefix: $(collector_group_for p_perl)" - # a bare package name is unchanged - [ "$(collector_group_for perl)" = "nimgnu_perl" ] \ - || echo "a bare package name does not reach the same group" - # a config step keeps its own prefix -- cfg_ is part of the step's name - [ "$(collector_group_for cfg_bootscripts)" = "nimgnu_cfg_bootscripts" ] \ - || echo "a config step's group name lost its kind: $(collector_group_for cfg_bootscripts)" )" - if [ -z "$_o" ]; then - ok "a collector group carries the collector prefix and no other" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- an ignored package that something REQUIRES (1.14.54) ------------------ # +# WebKitGTK configured for an hour and then: +# CMake Error: Ruby 2.5 or higher is required. +# webkitgtk REQUIRES Ruby-4.0.1, and the stack's avoid list contains `ruby`. +# Not walking into an avoided branch is right (1.14.24); leaving out a hard +# requirement of a package the user asked for, without a word, is not. +_p="" +_b54="$(dirname "$LFS_TOOL")/blfs" +_pmQ="$(dirname "$LFS_TOOL")/packagemanager" +_bk54="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk54="$_c" && break +done +if [ -n "$_bk54" ]; then + _o="$(python3 "$_b54" --book-file "$_bk54" order webkitgtk --anchors \ + --ignore ruby 2>/dev/null)" + printf '%s\n' "$_o" | grep -q '^#ignored-required webkitgtk ruby' \ + || _p="$_p;an ignored REQUIRED dependency is not reported by the walk" + # the ignore still works: ruby is not built + printf '%s\n' "$_o" | grep -v '^#' | cut -f1 | grep -qix ruby \ + && _p="$_p;the ignored package is in the order after all" + # a dependency that is only ever RECOMMENDED is not reported -- that is + # the normal, intended use of an avoid list. (bubblewrap is the wrong + # example: webkitgtk merely recommends it, but glycin in the same tree + # REQUIRES it, so reporting it is correct.) + _o2="$(python3 "$_b54" --book-file "$_bk54" order webkitgtk --anchors \ + --ignore enchant 2>/dev/null)" + printf '%s\n' "$_o2" | grep -q '^#ignored-required' \ + && _p="$_p;ignoring a merely recommended package is reported as a problem" +fi +# packagemanager turns that into a warning with a way out +sed -n '/def _report_ignored_required/,/^def /p' "$_pmQ" \ + | grep -q 'REQUIRE something the ignore list' \ + || _p="$_p;the plan does not warn about it" +sed -n '/def _report_ignored_required/,/^def /p' "$_pmQ" \ + | grep -q 'packagemanager install %s --run --recursive' \ + || _p="$_p;no command offered to resolve it" +grep -q '_report_ignored_required(steps)' "$_pmQ" \ + || _p="$_p;the report is never called" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "ignoring a package that something requires is said out loud" fi -# ---- root's own files are root's ------------------------------------------- # -# Every cmd_add_user during a build writes /etc/passwd, /etc/group and Shadow's -# backups of them. Whichever package happened to be building at the time swept -# them into its manifest: -# /etc/.pwd.lock is root want p_ncurses -# /etc/group- is root want p_ncurses -# /etc/passwd- is root want p_psmisc -# ncurses did not install /etc/group-; useradd did, while ncurses was building. -# A package owning the file that says which packages exist is the one ownership -# mistake with no way back. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( ETC=/etc; SNAP_ROOT=/ - eval "$(_slice_fn "$helper_src" never_claim_list)" - eval "$(_slice_fn "$helper_src" _never_claim_load)" - eval "$(_slice_fn "$helper_src" is_never_claimed)" - for f in /etc/passwd /etc/passwd- /etc/group /etc/group- \ - /etc/shadow /etc/gshadow /etc/.pwd.lock; do - is_never_claimed "$f" || echo "$f can be claimed by a package" - done - # ordinary configuration is still a package's to own - is_never_claimed /etc/fstab \ - && echo "an ordinary config file was swept into the never-claim list" )" - if [ -z "$_o" ]; then - ok "the user database belongs to root and to no package" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- cmake takes -D options too (1.14.55) ---------------------------------- # +# WebKitGTK stopped on "Enchant is needed for ENABLE_SPELLCHECK". enchant is +# in the user's avoid list and only RECOMMENDED, so 1.14.54's check correctly +# stayed quiet -- but the natural fix, turning the feature off, had nowhere to +# live: +# machine.conf has [webkitgtk] but the script has no `meson setup` line +# -- options NOT applied +# webkitgtk is a cmake package. Both build systems take -Dkey=value; only the +# command word differs. +_p="" +_pmR="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmR" "$T/cmakeopts" <<'PYCMAKE' || _p="$_p;machine.conf does not reach a cmake package" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_w") +open(p, "w").write("build_pkg() {\nmkdir -vp build &&\ncd build &&\n" + "cmake -D CMAKE_BUILD_TYPE=Release \\\n -D PORT=GTK ..\n" + "ninja\n}\n") +pm._machine_opts_for = lambda a: {"ENABLE_SPELLCHECK": "OFF"} +pm.apply_machine_opts("webkitgtk", p) +body = open(p).read() +assert "-DENABLE_SPELLCHECK=OFF" in body, body +assert "cmake" in body and "ninja" in body, body +# an option the BOOK already sets is replaced, not duplicated +pm._machine_opts_for = lambda a: {"PORT": "GTK4"} +pm.apply_machine_opts("webkitgtk", p) +body = open(p).read() +assert body.count("PORT=") == 1, body +assert "-DPORT=GTK4" in body, body +# applying twice changes nothing further +before = body +pm.apply_machine_opts("webkitgtk", p) +assert open(p).read() == before, open(p).read() +PYCMAKE +# a script with neither build system says so, and names the alternative +python3 - "$_pmR" "$T/cmakeopts" <<'PYNONE' || _p="$_p;a script with no -D build system is not explained" +import sys, os, io, contextlib, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +p = os.path.join(sys.argv[2], "install_plain") +open(p, "w").write("build_pkg() {\nmake\n}\n") +pm._machine_opts_for = lambda a: {"FOO": "bar"} +buf = io.StringIO() +with contextlib.redirect_stderr(buf), contextlib.redirect_stdout(buf): + pm.apply_machine_opts("plain", p) +out = buf.getvalue() +assert "no `meson setup` or `cmake` line" in out, out +assert "configure_args" in out, out +assert open(p).read() == "build_pkg() {\nmake\n}\n" # untouched +PYNONE +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "machine.conf options reach cmake packages, not only meson ones" +fi - # the tools we install are ours, in bin AND sbin - _p="" - grep -q 'add_package_user install_package' "$helper_src" \ - || _p="$_p;the hint's helper scripts are not claimed by p_pkgusr" - case "$(_slice_fn "$helper_src" _adopt_pkgusr_tools)" in - *'/usr/bin /usr/sbin'*) ;; - *) _p="$_p;only /usr/bin is searched, so the sbin helpers stay unclaimed" ;; - esac - # and the wrappers must not be reported as needing an owner - case "$(_slice_fn "$helper_src" _vfy_unclaimed)" in - *is_never_claimed*) ;; - *) _p="$_p;verify asks who should own the wrappers, which are root's by design" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "our own tools get an owner; the wrappers stay root's" - fi +# ---- a demonstration is not an installation step (1.14.56) ----------------- # +# enchant installed perfectly -- 17 files tracked, enchant-2.pc in place -- and +# then the configure phase failed: +# No dictionary available for 'en_GB' +# Its page ends with "You can test your installation and configuration by +# creating a test file and running the following commands", and the generator +# turned that demonstration into a step. It needs an aspell dictionary the +# system may not have, and running it is nobody's idea of installing enchant. +_p="" +_b56="$(dirname "$LFS_TOOL")/blfs" +_bk56="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk56="$_c" && break +done +if [ -n "$_bk56" ]; then + _ed="$T/enchant"; mkdir -p "$_ed" + python3 "$_b56" --book-file "$_bk56" script enchant -o "$_ed" >/dev/null 2>&1 + _e="$(ls "$_ed"/install_enchant* 2>/dev/null | head -1)" + if [ -n "$_e" ]; then + _live="$(sed -n '/^configure_pkg() {/,/^}/p' "$_e" \ + | grep -vE '^\s*#|^\s*$|^configure_pkg|^\}')" + printf '%s' "$_live" | grep -q 'enchant-2 -d en_GB' \ + && _p="$_p;the book's spell-check demo still runs as a config step" + printf '%s' "$_live" | grep -q 'test-enchant.txt' \ + && _p="$_p;the demo's heredoc still runs" + # the text is kept, commented, with a reason + grep -q 'the book offers this as a way to TEST' "$_e" \ + || _p="$_p;nothing says why those lines are commented" + grep -q '# enchant-2 -d en_GB' "$_e" \ + || _p="$_p;the demo was deleted rather than commented" + bash -n "$_e" || _p="$_p;the enchant script does not parse" + else + _p="$_p;could not generate the enchant script" + fi +fi +# ...and a real configuration step is NOT commented by that rule +python3 - "$_b56" <<'PYCFG' || _p="$_p;a genuine configuration step was commented away" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +real = ("# Create the configuration file:\n" + "cat > /etc/foo.conf << \"EOF\"\nx=1\nEOF") +assert b._comment_config_tests(real) == real, b._comment_config_tests(real) +PYCFG +# a shell word is not a program: `not: command not found` is noise +_nf="$T/notfound"; mkdir -p "$_nf" +python3 - "$helper_src" > "$_nf/rep.sh" <<'PYSLICE8' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }; _package_providing() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE8 +printf './configure: line 26491: not: command not found\ninstall_x: line 12: cmake: command not found\n' \ + > "$_nf/both.log" +_o="$(bash -c '. "'"$_nf"'/rep.sh"; report enchant all "'"$_nf"'/both.log"' 2>&1)" +case "$_o" in + *"'cmake' is not installed"*) ;; + *) _p="$_p;the real missing command is not reported" ;; +esac +case "$_o" in + *"'not' is not installed"*) _p="$_p;a shell word is still reported as a package" ;; +esac +printf './configure: line 1: not: command not found\n' > "$_nf/noise.log" +_o2="$(bash -c '. "'"$_nf"'/rep.sh"; report enchant all "'"$_nf"'/noise.log"' 2>&1)" +case "$_o2" in + *"is not installed -- the build needs it"*) + _p="$_p;a log with only shell noise still names a missing program" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a book's demo is not run, and a shell word is not called a package" fi -# ---- book 4.3 must not run on a built tree ---------------------------------- # -# `_chown_tree_to_lfs` is `chown -R lfs` over /usr, /etc, /var and the rest. -# Before chapter 5 that is book 4.3 and exactly right. On a BUILT tree it hands -# the finished system to the host's build account in one pass, and every -# package's ownership is gone -- recoverable only from the manifests. -# -# It happened: `lfs build-system run` on a 104/104 tree replayed step 4.2 and -# drwxrwxr-t 1 lfs install /mnt/lfs/usr -# came back with the host uid 10753 written through the tree. Inside the chroot -# that resolves to nothing -- the chroot's own `lfs` is 9998 -- so `ls -al /` -# showed a raw number where every owner should have been. -python3 - "$LFS_TOOL" <<'PY43' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -m = re.search(r"^def _chown_tree_to_lfs\(.*?(?=\n\ndef )", src, re.S | re.M) -if not m: - problems.append("_chown_tree_to_lfs is gone") -else: - body = m.group(0) - if "_handover_is_safe" not in body: - problems.append("book 4.3 would chown a built tree to the build user") - # the guard has to come BEFORE the chown, or it reports and then does it - # the INVOCATION, not the comment above the guard that names the command - g = body.find("if not _handover_is_safe") - c = body.find("_run_bash('chown -R lfs") - if c < 0: - problems.append("the 4.3 chown is no longer recognisable") - elif g < 0 or g > c: - problems.append("the 4.3 guard runs after the chown it is guarding") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS book 4.3 refuses to hand a built tree back to the build user") -PY43 -_count_rc $? +# ---- one place to set the job count (1.14.57) ------------------------------ # +# Asked where -j is set. Four places, and they did not agree: +# lfs config makeflags -> written into <tree>/usr/src/lfs-pkgusr/config/env +# and exported by every chroot shell +# /etc/pkgusr/build.env -> the per-machine file, read by build_env_pairs +# lfs-helper build --jobs N -> one build +# lfs-phases -> -j$(nproc) only when nothing else set it +# build.env is added to the command FIRST and the caller's MAKEFLAGS after it, +# and the last assignment on an `env` line wins -- so an ambient -j16 from the +# chroot's config silently beat the -j13 someone had put in the file meant for +# exactly this. +_p="" +_hlpJ="$helper_src" +_pp="$(_slice_fn "$_hlpJ" cmd_build)" +printf '%s' "$_pp" | grep -q 'grep -q "MAKEFLAGS="' \ + || _p="$_p;an ambient MAKEFLAGS still overrides the per-machine file" +# the precedence itself: --jobs beats build.env beats the shell +_jt() { # <jobs> <build.env value> <shell value> + ( jobs="$1"; MAKEFLAGS="$3" + envpass="" + [ -n "$2" ] && envpass="MAKEFLAGS='$2' " + if [ -n "$jobs" ]; then + envpass="$envpass MAKEFLAGS='-j$jobs'" + elif [ -n "${MAKEFLAGS:-}" ] && ! printf '%s' "$envpass" | grep -q "MAKEFLAGS="; then + envpass="$envpass MAKEFLAGS='$MAKEFLAGS'" + fi + printf '%s' "$envpass" | sed 's/.*MAKEFLAGS=//' | tr -d "' " ) +} +[ "$(_jt 4 -j13 -j16)" = "-j4" ] || _p="$_p;--jobs does not win" +[ "$(_jt '' -j13 -j16)" = "-j13" ] || _p="$_p;build.env does not beat the ambient value" +[ "$(_jt '' '' -j16)" = "-j16" ] || _p="$_p;the ambient value is ignored when nothing else is set" +# lfs-phases only fills in a default when nobody said anything +sed -n '/if \[ -z "${MAKEFLAGS:-}" \]/,/fi/p' "$PHASES_LIB" | grep -q 'nproc' \ + || _p="$_p;the runner no longer defaults the job count" +grep -q 'export MAKEFLAGS="-j$(nproc)"' "$PHASES_LIB" \ + || _p="$_p;the runner's default is not a job count" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the job count comes from one file, and --jobs still overrides it" +fi -# ---- the ownership step is visible ------------------------------------------ # -# It ran as an invisible checkpoint inside build-all: right place, right work, -# but it never appeared in `lfs-helper list`. The one moment the whole scheme -# turns on was the one moment you could not point at, resume from, or re-run. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - # lfs puts it in the step order, straight after book 7.6 - grep -q 'order.insert(order.index("init-files") + 1, "init-ownership")' "$LFS_TOOL" \ - || _p="$_p;init-ownership is not in the step order" - # lfs-helper performs it itself -- there is no script for it in the book - case "$(_slice_fn "$helper_src" cmd_build)" in - *'"$name" = "init-ownership"'*) ;; - *) _p="$_p;lfs-helper would look for a book script for init-ownership" ;; - esac - # it owns no files, so it gets no account and no home - case "$(_slice_fn "$helper_src" _is_not_a_package)" in - *init-ownership*) ;; - *) _p="$_p;init-ownership would be given a package user" ;; - esac - # and running it must record itself, so list and next agree with reality - case "$(_slice_fn "$helper_src" cmd_build)" in - *'mark_done "$name"'*) ;; - *) _p="$_p;running init-ownership does not mark it built" ;; - esac - case "$(_slice_fn "$helper_src" _ownership_checkpoint)" in - *mark_done*) ;; - *) _p="$_p;a tree that crossed the epoch still shows the step as pending" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the ownership step appears in the list, and can be run on its own" - fi +# ---- a stack file is the user's (1.14.58) ---------------------------------- # +# `make install` copied stacks/* over /etc/pkgusr/stacks/* unconditionally. +# The user had just removed `avoid ruby` from sway.stack to fix a build; the +# next `make install` -- which every fix in this session begins with -- would +# have put it straight back, and the build would have failed the same way with +# no explanation. machine.conf lives in the same directory and was equally at +# risk, which is where the per-package build options go. +_p="" +_mkf="$(dirname "$LFS_TOOL")/Makefile" +if [ -f "$_mkf" ]; then + _blk="$(sed -n '/A STACK FILE IS THE USER/,/done; true/p' "$_mkf")" + printf '%s' "$_blk" | grep -q 'cmp -s' \ + || _p="$_p;an edited stack file is still overwritten" + printf '%s' "$_blk" | grep -q '\.new' \ + || _p="$_p;the new version is not offered alongside" + printf '%s' "$_blk" | grep -q 'KEPT YOURS' \ + || _p="$_p;it does not say that the user's file was kept" + # and it really behaves that way + _d="$T/mkinstall"; rm -rf "$_d" + ( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_d" ) >/dev/null 2>&1 + if [ -f "$_d/etc/pkgusr/stacks/sway.stack" ]; then + # (a leading newline: the shipped file has no trailing one, and an + # appended marker would otherwise join its last line) + printf '\n# LOCAL EDIT\n' >> "$_d/etc/pkgusr/stacks/sway.stack" + _o="$( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_d" 2>&1 )" + grep -q '^# LOCAL EDIT$' "$_d/etc/pkgusr/stacks/sway.stack" \ + || _p="$_p;a second make install destroyed the edit" + [ -f "$_d/etc/pkgusr/stacks/sway.stack.new" ] \ + || _p="$_p;the shipped version was not left for comparison" + case "$_o" in + *"KEPT YOURS"*) ;; + *) _p="$_p;the install is silent about keeping the user's file" ;; + esac + # an untouched file is still refreshed, not left stale + case "$_o" in + *"(unchanged)"*) ;; + *) _p="$_p;identical files are not reported as such" ;; + esac + else + _p="$_p;make install did not place the stack files at all" + fi +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "make install keeps an edited stack file and offers the new one beside it" fi -# ---- a chapter-8 tree can still be taken back from the build user ----------- # -# Refusing the recursive 7.2 chown on a built tree is right -- it would strip -# every package user in one pass. But refusing and stopping there left the tree -# with NO way forward: -# -# 4.3 had taken /usr, /etc, /var and the root symlinks, -# 7.2 said "not touching it, repair from the manifests instead", -# chroot enter said "the tree is not owned by root yet: run chroot prepare" -# -# -- pointing at the command that had just declined. The precise repair is to -# reclaim what the BUILD USER holds, by uid: that leaves every package user's -# files alone, so it is safe at any stage. -python3 - "$LFS_TOOL" <<'PYRECLAIM' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -for fn in ("_build_user_uid", "_handover_build_user_only"): - if not re.search(r"^def %s\(" % fn, src, re.M): - problems.append("there is no %s" % fn) -m = re.search(r"^def _handover_build_user_only\(.*?(?=\n\ndef )", src, re.S | re.M) -if m: - body = m.group(0) - # selects by uid, so package users are untouched - if "-uid" not in body: - problems.append("the reclaim does not select on the build user's uid") - # user only: 4.3 chowned the user, so the inverse must not take the group - # as well -- that would strip `install` off every shared directory - # the INVOCATION -- the docstring above it explains why the group is left - # alone, and matching that text made this impossible to satisfy - code = "\n".join(l for l in body.splitlines() - if not l.lstrip().startswith("#")).split('"""') - code = "".join(code[::2]) - if "chown -h root:root" in code or "chown -R" in code: - problems.append("the reclaim takes the group too, undoing the install group") - if "chown -h root" not in body: - problems.append("the reclaim follows symlinks instead of chowning them") -m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) -if m: - body = m.group(0) - if "_handover_build_user_only" not in body: - problems.append("a chapter-8 tree is still left with no way forward") - # and entering must repair rather than point back at prepare - enter = body.split('if action == "enter"') - if len(enter) > 1 and "_handover_build_user_only" not in enter[1]: - problems.append("chroot enter still sends you to a command that declined") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS a built tree can be taken back from the build user without loss") -PYRECLAIM -_count_rc $? - -# end to end: the build user's paths go back to root, the package user's do not -if [ "$(id -u)" = 0 ] && command -v python3 >/dev/null 2>&1; then - _fx="$T/reclaim/mnt/lfs" - mkdir -p "$_fx/usr/bin" "$_fx/etc" "$_fx/var" "$_fx/tools" - ( cd "$_fx" && ln -sf usr/bin bin ) - echo pkg > "$_fx/usr/bin/owned-by-a-package" - echo cfg > "$_fx/etc/fstab" - # 4242 stands in for a package user, 4243 for the host's build account - chown 4242:4242 "$_fx/usr/bin/owned-by-a-package" - for _d in usr usr/bin etc var tools etc/fstab; do chown 4243 "$_fx/$_d"; done - chown -h 4243 "$_fx/bin" - _out="$(python3 - "$LFS_TOOL" "$_fx" <<'PYFX' -import sys, importlib.util -src = open(sys.argv[1]).read().split('if __name__ ==')[0] -spec = importlib.util.spec_from_loader("t", None) -m = importlib.util.module_from_spec(spec) -exec(compile(src, "lfs", "exec"), m.__dict__) -m._build_user_uid = lambda: 4243 -print(m._handover_build_user_only(sys.argv[2], run=True)) -PYFX -)" - _bad="" - [ "$(stat -c %u "$_fx/usr")" = 0 ] || _bad="$_bad;/usr was not taken back" - [ "$(stat -c %u "$_fx/bin")" = 0 ] || _bad="$_bad;the root symlink was not taken back" - [ "$(stat -c %u "$_fx/usr/bin/owned-by-a-package")" = 4242 ] \ - || _bad="$_bad;a package user's file was taken from it" - [ -L "$_fx/bin" ] || _bad="$_bad;the symlink was followed and replaced" - if [ -n "$_bad" ]; then - printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "reclaiming takes the build user's paths and leaves the packages' alone" - fi - rm -rf "$T/reclaim" +# ---- ninja does not read MAKEFLAGS (1.14.59) ------------------------------- # +# WebKitGTK's own page: "some source files of this package require more than +# 4 GiB of RAM to be built. As the result, you should pass -j<N> to ninja +# (replacing <N> with the quotient of the amount of available RAM and 4 GiB) +# to limit the number of parallel jobs and avoid the job from being killed by +# the kernel OOM killer." +# MAKEFLAGS cannot express that: ninja ignores it and defaults to nproc+2. So +# there was no way to follow the book short of editing the script by hand, and +# the failure it prevents arrives hours in, as a killed compiler. +_p="" +_pmS="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmS" "$T/ninjajobs" <<'PYNINJA' || _p="$_p;a per-package ninja job cap cannot be set" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_w") +open(p, "w").write("build_pkg() {\ncmake .. &&\nninja\n}\n" + "install_pkg() {\nninja install\nninja -C build install\n}\n") +pm._machine_opts_for = lambda a: {"jobs": "8"} +pm.apply_machine_opts("webkitgtk", p) +body = open(p).read() +assert "ninja -j8\n" in body, body +assert "ninja -j8 install" in body, body +assert "ninja -j8 -C build install" in body, body +# applying it again does not stack more -j flags +pm.apply_machine_opts("webkitgtk", p) +assert open(p).read().count("-j8") == 3, open(p).read() +# a ninja line that ALREADY has -j is left alone +p2 = os.path.join(d, "install_x") +open(p2, "w").write("build_pkg() {\nninja -j4\n}\n") +pm.apply_machine_opts("webkitgtk", p2) +assert open(p2).read().count("-j") == 1, open(p2).read() +assert "-j4" in open(p2).read() +PYNINJA +# and `jobs` is not passed on to cmake as -Djobs= +sed -n '/NINJA DOES NOT READ MAKEFLAGS/,/_wrap = opts.pop/p' "$_pmS" \ + | grep -q 'opts.pop("jobs"' \ + || _p="$_p;jobs would be handed to cmake as an option" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package can cap its own ninja jobs, as its book page asks" fi -# ---- state lives under /usr/src, and it is sorted --------------------------- # -# It was /.lfs-pkgusr: a hidden directory at the root of the filesystem, sitting -# next to /boot and /etc as if it were part of the system being built. It is -# not -- it is this toolchain's working notes ABOUT that system -- and it holds -# twenty-two loose files at the top level. -# -# Under /usr/src it sits beside the accounts it describes, and the tree scans -# exclude ONE path instead of two, because everything under $SRCROOT already is. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - grep -q '^STATE="\${LFS_PKGUSR_DIR:-\${SRCROOT%/}/lfs-pkgusr}"' "$helper_src" \ - || _p="$_p;lfs-helper state is not under \$SRCROOT" - grep -q '^PKGUSR_DIR = "usr/src/lfs-pkgusr"' "$LFS_TOOL" \ - || _p="$_p;lfs state is not under /usr/src" - # both tools must name the SAME directory -- they read each other's manifests - case "$(grep -m1 '^STATE=' "$helper_src")" in - *lfs-pkgusr*) ;; - *) _p="$_p;the two tools no longer agree on where state lives" ;; - esac - # SRCROOT has to be defined before what derives from it: `set -u` turns a - # forward reference into "SRCROOT: unbound variable" and the tool will not - # even print its version - _isrc="$(grep -n '^SRCROOT=' "$helper_src" | head -1 | cut -d: -f1)" - _ista="$(grep -n '^STATE=' "$helper_src" | head -1 | cut -d: -f1)" - if [ -n "$_isrc" ] && [ -n "$_ista" ] && [ "$_isrc" -gt "$_ista" ]; then - _p="$_p;SRCROOT is defined after the paths derived from it" - fi - # nothing loose at the top: every state file belongs to a named group - for _v in PROGRESS SNAPSHOT SNAPDIRS INSTALLDIRS; do - case "$(grep -m1 "^$_v=" "$helper_src")" in - *'$STATE_PROGRESS/'*|*'$STATE_CONF/'*|*'$STATE_GROUPS/'*) ;; - *) _p="$_p;$_v is still a loose file at the top of the state dir" ;; - esac - done - # and the directories are created together, so no path can find one missing - case "$(_slice_fn "$helper_src" mkstate)" in - *STATE_PROGRESS*STATE_GROUPS*STATE_CONF*) ;; - *) _p="$_p;mkstate does not create every state directory" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "state lives under /usr/src, sorted into named directories" - fi +# ---- put the option on the command, not in the prose (1.14.60) ------------- # +# WebKitGTK died on +# fatal error: systemd/sd-journal.h: No such file or directory +# with ENABLE_JOURNALD_LOG = OFF sitting in machine.conf. The generated +# script showed why: +# ## NOTE: ... add the -D CMAKE_CXX_FLAGS_RELEASE=... option to the cmake +# -Denable_journald_log=OFF -Duse_gstreamer_webrtc=ON to disable some ... +# Two bugs in one line. The options went into the first line CONTAINING the +# word "cmake" -- a book NOTE -- and never reached the build. And they were +# lower-cased, because configparser lower-cases keys; cmake variables are +# case-sensitive, so even correctly placed, -Denable_journald_log=OFF would +# have set a different variable and been ignored. (meson's options are +# lower-case anyway, which is why [mesa] worked and hid this.) +_p="" +_pmT="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmT" "$T/prose" <<'PYPROSE' || _p="$_p;options still land in a comment, or lose their case" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_w") +note = ("## NOTE: add the -D CMAKE_CXX_FLAGS_RELEASE option to the cmake " + "command to disable optimizations") +open(p, "w").write("build_pkg() {\n%s\nmkdir -vp build &&\ncd build &&\n" + "cmake -D PORT=GTK -D USE_GTK4=OFF ..\nninja\n}\n" % note) +pm._machine_opts_for = lambda a: {"ENABLE_JOURNALD_LOG": "OFF", "USE_GTK4": "ON"} +pm.apply_machine_opts("webkitgtk", p) +body = open(p).read() +# the NOTE is untouched +assert note in body, body +# the real command carries the options, with their case intact +cmd = [l for l in body.splitlines() if l.startswith("cmake ")][0] +assert "-DENABLE_JOURNALD_LOG=OFF" in cmd, cmd +assert "-DUSE_GTK4=ON" in cmd, cmd +assert "enable_journald_log" not in body, body +# an option the book set is still replaced, not duplicated +assert cmd.count("USE_GTK4") == 1, cmd +PYPROSE +# the parser keeps key case +python3 - "$_pmT" "$T/prose" <<'PYCASE' || _p="$_p;machine.conf keys are lower-cased" +import sys, os, importlib.machinery as m +os.environ["LFS_MACHINE_CONF"] = os.path.join(sys.argv[2], "m.conf") +open(os.environ["LFS_MACHINE_CONF"], "w").write( + "[webkitgtk]\nENABLE_JOURNALD_LOG = OFF\nUSE_GTK4 = ON\n") +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +got = pm._read_conf_section(os.environ["LFS_MACHINE_CONF"], "webkitgtk") +assert "ENABLE_JOURNALD_LOG" in got, got +assert "enable_journald_log" not in got, got +PYCASE +sed -n '/CMAKE VARIABLES ARE CASE-SENSITIVE/,/optionxform/p' "$_pmT" \ + | grep -q 'cp.optionxform = str' \ + || _p="$_p;the parser still folds key case" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a build option reaches the command with its case intact" fi -# ---- colour means one thing --------------------------------------------------# -# Four colours, one meaning each, the same in both tools -- they print into the -# same terminal one after the other, and a log where the same colour means two -# different things is worse than no colour at all. Prose gets none: colour on -# every other line stops carrying information. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - for _fn in say ok warn note detail hint die fail; do - grep -qE "^$_fn\(\) +\{" "$helper_src" \ - || _p="$_p;lfs-helper has no $_fn" - done - # say() is prose and must not paint anything - case "$(grep -m1 '^say()' "$helper_src")" in - *C_*) _p="$_p;plain prose is coloured" ;; - esac - # the vocabulary is the interface: no ad-hoc colour outside those helpers - # the helper DEFINITIONS are where the colour lives; everything else must go - # through them - _adhoc="$(grep -nE '(say|echo) "\$\{C_(DIM|WARN|ERR|OK)\}' "$helper_src" \ - | grep -vE '^[0-9]+:(ok|warn|note|detail|hint|die|fail)\(\)' | wc -l)" - [ "${_adhoc:-0}" = 0 ] \ - || _p="$_p;$_adhoc line(s) still paint colour by hand instead of using a helper" - # lfs must offer the same set, or the two halves of a build read differently - for _fn in warn note ok fail hint; do - grep -qE "^def $_fn\(" "$LFS_TOOL" || _p="$_p;lfs has no $_fn" - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "both tools share one colour vocabulary, and prose has none" - fi +# ---- a skipped variant does not keep its install step (1.14.61) ------------ # +# "is it possible that the install of webkitgtk is repeating itself at the +# end? i think im building it the second time!" Its install phase held +# ninja -j8 install +# ## OPTIONAL, skipped -- ... the GTK-4 version ... +# # <the whole GTK-4 build, commented> +# ninja -j8 install <- the GTK-4 variant's install, still live +# The second is a no-op rather than a second build, but on a package that +# takes hours nobody watching can tell. +_p="" +_b61="$(dirname "$LFS_TOOL")/blfs" +python3 - "$_b61" <<'PYDUP' || _p="$_p;a skipped variant still keeps its install step" +import sys, importlib.machinery as m +b = m.SourceFileLoader('b', sys.argv[1]).load_module() +# X ; <skipped> ; X -> the repeat belongs to the skipped block +out = b._drop_orphaned_variant_install( + "ninja install\n## OPTIONAL, skipped -- the GTK-4 version:\n# cmake ..\nninja install") +live = [l for l in out.splitlines() + if l.strip() and not l.strip().startswith("#")] +assert live == ["ninja install"], live +# ...but a repeat with real work in between is NOT redundant: GLib installs, +# rebuilds with introspection enabled, and installs again +glib = ("ninja install\nmeson configure -D introspection=enabled &&\nninja\n" + "## OPTIONAL, skipped -- the documentation:\n# ninja -C docs\nninja install") +out2 = b._drop_orphaned_variant_install(glib) +live2 = [l for l in out2.splitlines() + if l.strip() and not l.strip().startswith("#")] +assert live2.count("ninja install") == 2, live2 +# and a phase with no skipped block at all is untouched +plain = "make install\nmake install-data" +assert b._drop_orphaned_variant_install(plain) == plain +PYDUP +_bk61="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bk61="$_c" && break +done +if [ -n "$_bk61" ]; then + _wd61="$T/variants"; mkdir -p "$_wd61" + python3 "$_b61" --book-file "$_bk61" script webkitgtk glib2 -o "$_wd61" >/dev/null 2>&1 + _w="$(ls "$_wd61"/install_WebKitGTK* 2>/dev/null | head -1)" + [ -n "$_w" ] && { [ "$(sed -n '/^install_pkg() {/,/^}/p' "$_w" \ + | grep -c '^ninja.*install')" = 1 ] \ + || _p="$_p;webkitgtk still installs twice"; } + # GLib's second install is real and must remain + _g="$(ls "$_wd61"/install_GLib* 2>/dev/null | head -1)" + [ -n "$_g" ] && { [ "$(sed -n '/^install_pkg() {/,/^}/p' "$_g" \ + | grep -c '^ninja install')" -ge 2 ] \ + || _p="$_p;GLib lost the install that follows its introspection rebuild"; } +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a commented-out build variant does not leave its install behind" fi -# ---- the collector export must land where the reader looks ------------------ # -# `lfs` copies the export INTO the tree; `lfs-helper` reads it back out. Those -# two paths are the whole mechanism -- nothing else connects them -- and when -# the state directory was sorted, the writer kept using the top level while the -# reader moved into groups/. A rebuild with a perfectly good export then asked -# -# 'p_gcc' needs to install into: /usr/lib/bfd-plugins -# that directory belongs to package 'p_binutils' -# Which group should share it? -# -# for a directory the file already answered. Silent, because a question that -# gets asked looks the same as a question that was never answered. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - # what the writer writes - _w="$(python3 - "$LFS_TOOL" <<'PYW' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r"^def _install_collector_groups\(.*?(?=\n\ndef )", src, re.S | re.M) -if m: - d = re.search(r'dst_dir = pkgusr_state\(lfs, "([a-z]+)"\)', m.group(0)) - print(d.group(1) if d else "TOPLEVEL") -else: - print("MISSING") -PYW -)" - # what the reader reads - _r="$(grep -o '\$STATE_[A-Z]*/collector-groups\.import' "$helper_src" \ - | head -1 | sed 's|\$STATE_\([A-Z]*\)/.*|\1|' | tr 'A-Z' 'a-z')" - _p="" - [ "$_w" = groups ] || _p="$_p;lfs writes the collector export to '$_w', not groups/" - [ "$_r" = groups ] || _p="$_p;lfs-helper reads the collector export from '$_r', not groups/" - [ "$_w" = "$_r" ] || _p="$_p;the two tools disagree about where the export lives" - # an edit on the host has to reach the build, so entry re-copies it - case "$(python3 - "$LFS_TOOL" <<'PYE' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r"^def cmd_bs_chroot\(.*?(?=\n\ndef )", src, re.S | re.M) -body = m.group(0) if m else "" -enter = body.split('if action == "enter"') -print("yes" if len(enter) > 1 and "_install_collector_groups" in enter[1] else "no") -PYE -)" in - yes) ;; - *) _p="$_p;editing the export on the host never reaches the build" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the collector export is written where lfs-helper reads it" - fi +# ---- a library that will not link (1.14.62) -------------------------------- # +# alsa-utils found alsa-lib's headers and then: +# checking for snd_ctl_open in -lasound... no +# configure: error: No linkable libasound was found. +# Headers present but the library unlinkable is a different fault from a +# missing package, and none of the existing diagnoses covered it -- the box +# said nothing at all. Three ways it happens, told apart by what is on disk. +_p="" +_hlpK="$helper_src" +_ld="$T/linklib"; mkdir -p "$_ld" +python3 - "$_hlpK" > "$_ld/rep.sh" <<'PYSLICE9' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('s="/x"; staged="/x"; auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICE9 +printf 'checking for snd_ctl_open in -lasound... no\nconfigure: error: No linkable libasound was found.\n' \ + > "$_ld/alsa.log" +# 1. dev symlink AND runtime library present -> a cache or a second library +mkdir -p "$_ld/both/usr/lib" +: > "$_ld/both/usr/lib/libasound.so.2.0.0" +ln -sf libasound.so.2.0.0 "$_ld/both/usr/lib/libasound.so" +_o="$(bash -c '. "'"$_ld"'/rep.sh"; SNAP_ROOT="'"$_ld"'/both"; report alsa-utils all "'"$_ld"'/alsa.log"' 2>&1)" +case "$_o" in + *"IS installed, and links should work"*) ;; + *) _p="$_p;a fully installed library is reported as missing" ;; +esac +case "$_o" in *ldconfig*) ;; *) _p="$_p;the stale linker cache is not mentioned" ;; esac +case "$_o" in *config.log*) ;; *) _p="$_p;it does not point at the real link error" ;; esac +# 2. runtime library only -> the development symlink is missing +mkdir -p "$_ld/runtime/usr/lib" +: > "$_ld/runtime/usr/lib/libasound.so.2.0.0" +_o2="$(bash -c '. "'"$_ld"'/rep.sh"; SNAP_ROOT="'"$_ld"'/runtime"; report alsa-utils all "'"$_ld"'/alsa.log"' 2>&1)" +case "$_o2" in + *"libasound.so.N is installed but libasound.so is NOT"*) ;; + *) _p="$_p;a missing development symlink is not diagnosed" ;; +esac +case "$_o2" in + *"--run --reinstall"*) ;; + *) _p="$_p;no way offered to repair the package that owns it" ;; +esac +# 3. nothing at all +mkdir -p "$_ld/none/usr/lib" +_o3="$(bash -c '. "'"$_ld"'/rep.sh"; SRCROOT="'"$_ld"'/nosrc"; SNAP_ROOT="'"$_ld"'/none"; report alsa-utils all "'"$_ld"'/alsa.log"' 2>&1)" +case "$_o3" in + *"Nothing here provides libasound at all"*) ;; + *) _p="$_p;a genuinely absent library is not reported" ;; +esac +# 3b. absent from disk but RECORDED by a package -- the reported case: a +# 1711-byte p_alsa-lib/pkg.lst and no libasound anywhere. "Nothing provides +# it" is true and one step short of the answer. +mkdir -p "$_ld/claimed/p_alsa-lib" +printf '/usr/lib/libasound.so.2.0.0 +/usr/lib/libasound.so +' > "$_ld/claimed/p_alsa-lib/pkg.lst" +_o3b="$(bash -c '. "'"$_ld"'/rep.sh"; SRCROOT="'"$_ld"'/claimed"; SNAP_ROOT="'"$_ld"'/none"; report alsa-utils all "'"$_ld"'/alsa.log"' 2>&1)" +case "$_o3b" in + *"'p_alsa-lib'"*"RECORDS having installed it"*) ;; + *) _p="$_p;the package whose record claims the library is not named" ;; +esac +case "$_o3b" in + *"install alsa-lib --run --reinstall"*) ;; + *) _p="$_p;no rebuild offered for the package that owns the library" ;; +esac +# the linker's own wording is read too +printf '/usr/bin/ld: cannot find -lfoo\n' > "$_ld/ld.log" +_o4="$(bash -c '. "'"$_ld"'/rep.sh"; SNAP_ROOT="'"$_ld"'/none"; report demo all "'"$_ld"'/ld.log"' 2>&1)" +case "$_o4" in + *"libfoo"*) ;; + *) _p="$_p;the linker's own 'cannot find -lX' is not recognised" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a library that will not link says which of the three faults it is" +fi - # end to end: a decision in the file must not be asked about again - _cg="$T/collgroups"; mkdir -p "$_cg/groups" - printf '# prefix|nimgnu\ndir|/usr/lib/bfd-plugins|nimgnu_bfd-plugins\n' \ - > "$_cg/groups/collector-groups.import" - _o="$( STATE="$_cg"; STATE_GROUPS="$_cg/groups"; SNAP_ROOT=/ - COLLECTOR_MAP="$_cg/groups/collector-map"; COLLECTOR_PREFIX=nimgnu - PKGUSR_PREFIX="p_" - eval "$(_slice_fn "$helper_src" unprefix_pkg_user)" - eval "$(_slice_fn "$helper_src" collector_group_for)" - # the lookup runs before any of these matter, but they are called on the - # way to it - group_exists() { return 1; } - user_in_group() { return 1; } - eval "$(_slice_fn "$helper_src" sanitise_group_name)" - eval "$(_slice_fn "$helper_src" choose_collector_group)" - # a missing helper must FAIL, not skip: an empty result reading as a - # pass is the trap this suite has fallen into twice - command -v choose_collector_group >/dev/null 2>&1 \ - || { echo "there is no choose_collector_group"; exit 0; } - # no tty, so it can only answer from what it already knows - # <dir> <owner> <user> -- the user is the package asking for access - _g="$(choose_collector_group /usr/lib/bfd-plugins p_binutils p_gcc \ - </dev/null 2>/dev/null)" - [ "$_g" = nimgnu_bfd-plugins ] \ - || echo "a directory the export already answered is asked about again: got '$_g'" )" - if [ -z "$_o" ]; then - ok "a directory named in the export is never asked about again" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- a zero exit is not proof that nothing failed (1.14.64) ---------------- # +# alsa-lib's install ended with +# install: cannot create regular file '/usr/include/sys/asoundlib.h': +# Permission denied +# make[4]: *** [Makefile:786: install-data-hook] Error 1 +# make: *** [Makefile:408: install-recursive] Error 1 +# and the run reported "# alsa-lib: done" and "1 package(s) installed". +# Whatever swallowed the status, the LOG said plainly that make had failed -- +# and every "installed but not really" in this session began exactly here. +_p="" +_hlpL="$helper_src" +_sw="$T/swallowed"; mkdir -p "$_sw" +_check() { # <logtext> <expect: caught|clean> + printf '%s' "$1" > "$_sw/log" + _r="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^install: cannot create|: Permission denied$" \ + "$_sw/log" 2>/dev/null)" + if [ -n "$_r" ]; then _got=caught; else _got=clean; fi + [ "$_got" = "$2" ] || _p="$_p;log check said $_got for: $(printf '%s' "$1" | head -1)" +} +# the reported failure, in the three shapes it took +_check "install: cannot create regular file '/usr/include/sys/x.h': Permission denied +" caught +_check "make[4]: *** [Makefile:786: install-data-hook] Error 1 +" caught +_check "make: *** [Makefile:408: install-recursive] Error 1 +" caught +_check "ninja: *** build stopped: subcommand failed. +" caught +# ...and things that must NOT trip it +_check "make[1]: Leaving directory '/x' +installing files +Done. +" clean +_check "checking for Permission denied handling... yes +" clean +_check " -- Error handling tests: 12 passed +" clean +# the check is wired into run_phase_as, and only when the status was 0 +_rp="$(_slice_fn "$_hlpL" run_phase_as)" +printf '%s' "$_rp" | grep -q 'A ZERO EXIT IS NOT PROOF' \ + || _p="$_p;the log is not checked when a phase exits 0" +printf '%s' "$_rp" | grep -q 'rc=1' \ + || _p="$_p;a swallowed failure does not become a failure" +printf '%s' "$_rp" | grep -q '\[ "$rc" = 0 \] && \[ -n "$log" \]' \ + || _p="$_p;the log check runs even when the phase already failed" +# and the runner itself still stops a phase at the first failing command +_ph="$T/phasefail"; mkdir -p "$_ph/src" +{ + echo '#!/bin/bash' + echo 'name_version="demo-1.0"' + echo 'build_pkg() { :; }' + echo 'install_pkg() {' + echo 'echo "step one"' + echo 'false' + echo 'echo "step two ran anyway"' + echo '}' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_ph/install_demo" +_o="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_ph/src" \ + bash "$_ph/install_demo" install 2>&1)"; _rc=$? +[ "$_rc" != 0 ] || _p="$_p;a failing command mid-phase still exits 0" +case "$_o" in + *"step two ran anyway"*) _p="$_p;the phase continued past a failed command" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a phase whose log records a failure is not reported as done" fi -# ---- verify must not report its own state directory ------------------------- # -# The state directory moved under /usr/src in 1.9.0, and /usr/src IS an install -# directory -- so `verify` began walking its own manifests, scripts and logs and -# reporting each one as a root-owned file no package claims. A clean 104/104 -# build came back with -# 448 root-owned file(s) that NO manifest claims -# and all 448 were this directory, including the two temp files the scan had -# just written for itself: -# unclaimed /usr/src/lfs-pkgusr/tmp/found.13431 -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr - WRAPPERS=/usr/lib/pkgusr - eval "$(_slice_fn "$helper_src" never_claim_list)" - eval "$(_slice_fn "$helper_src" _never_claim_load)" - eval "$(_slice_fn "$helper_src" is_never_claimed)" - command -v is_never_claimed >/dev/null 2>&1 \ - || { echo "there is no is_never_claimed"; exit 0; } - for f in "$STATE/manifests/gcc.files" "$STATE/scripts/gcc.sh" \ - "$STATE/logs/gcc-all.log" "$STATE/progress/steps-built" \ - "$STATE/tmp/found.13431"; do - is_never_claimed "$f" || echo "verify still reports $f" - done - # an account home under the SAME parent is a package's, and must not be - # swept up by this - is_never_claimed /usr/src/pkgusr/p_gcc/VERSION \ - && echo "a package user's own file was excluded too" )" - if [ -z "$_o" ]; then - ok "verify does not report the build's own notes as unclaimed files" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- a warning that is only fatal because of -Werror (1.14.65) ------------- # +# wlroots 0.19 against libinput 1.31: +# error: enumeration value 'LIBINPUT_SWITCH_KEYPAD_SLIDE' not handled in +# switch [-Werror=switch] +# cc1: all warnings being treated as errors +# Nothing is wrong with wlroots: a dependency grew a new enum value since that +# version was written. The box called it "a COMPILE error in the package's +# own source ... the tools cannot repair it" and offered `c_args = +# -DSOMETHING`, which is no help at all. Version skew like this is the normal +# state of a rolling book against pinned git sources, and the cure is always +# the same flag. +_p="" +_hlpM="$helper_src" +_we="$T/werror"; mkdir -p "$_we" +python3 - "$_hlpM" > "$_we/rep.sh" <<'PYSLICEA' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICEA +printf "../backend/libinput/switch.c:32:9: error: enumeration value 'X' not handled in switch [-Werror=switch]\ncc1: all warnings being treated as errors\n" \ + > "$_we/werror.log" +_o="$(bash -c '. "'"$_we"'/rep.sh"; report wlroots all "'"$_we"'/werror.log"' 2>&1)" +case "$_o" in + *"-Werror turned into an error"*) ;; + *) _p="$_p;a -Werror failure is not recognised" ;; +esac +case "$_o" in + *"werror = false"*) ;; + *) _p="$_p;the meson switch is not offered" ;; +esac +# the specific warning class is named, so the narrower flag is possible +case "$_o" in + *"-Wno-switch"*) ;; + *) _p="$_p;the warning class is not carried into the suggested flag" ;; +esac +case "$_o" in + *"(-Werror=switch)"*) ;; + *) _p="$_p;the box does not say which warning became the error" ;; +esac +# a compile error that is NOT a promoted warning keeps the old advice only +printf "../src/foo.c:12:5: error: too few arguments to function 'bar'\n" \ + > "$_we/plain.log" +_o2="$(bash -c '. "'"$_we"'/rep.sh"; report demo all "'"$_we"'/plain.log"' 2>&1)" +case "$_o2" in + *"-Werror turned into an error"*) _p="$_p;every compile error is blamed on -Werror" ;; +esac +case "$_o2" in + *"machine.conf"*) ;; + *) _p="$_p;an ordinary compile error lost its build-flag advice" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a warning promoted by -Werror is named, with the flag that turns it off" fi -# ---- the two tools share one tree snapshot ---------------------------------- # -# `lfs` takes the last snapshot outside the chroot at the end of chapter 6; -# `lfs-helper` diffs against it for the first package inside. It is a handover, -# and it works only while both name the same file. 1.9.0 moved the reader to -# progress/tree-snapshot and left the writer on .snapshot at the top level, so -# the handover snapshot was silently lost and the first chroot package diffed -# against nothing. -python3 - "$LFS_TOOL" "$(dirname "$LFS_TOOL")/lfs-helper" <<'PYSNAP' -import re, sys -lfs = open(sys.argv[1]).read() -problems = [] -if '".snapshot"' in lfs or '".snapshot-dirs"' in lfs: - problems.append("lfs still writes the snapshot to .snapshot at the top level") -if 'pkgusr_state(lfs, "progress")' not in lfs: - problems.append("lfs does not write the snapshot into progress/") -for want in ('"tree-snapshot"', '"tree-snapshot-dirs"'): - if want not in lfs: - problems.append("lfs does not name %s" % want) -try: - helper = open(sys.argv[2]).read() -except OSError: - helper = "" -if helper: - m = re.search(r'^SNAPSHOT="([^"]+)"', helper, re.M) - if not m or "tree-snapshot" not in m.group(1): - problems.append("lfs-helper reads a different snapshot file") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS both tools read and write the same tree snapshot") -PYSNAP -_count_rc $? +# ---- a git entry can name its dependencies (1.14.66) ----------------------- # +# sway is built from git, so nothing in the book orders anything before it: +# ERROR: Dependency "json-c" not found, tried pkgconfig and cmake +# json-c IS a BLFS package (JSON-C-0.18). It had simply never been built, +# because a git stack entry had no way to say it needed one -- book entries +# get their dependencies from the book, git entries got nothing. +_p="" +_pmU="$(dirname "$LFS_TOOL")/packagemanager" +_ssU="$(dirname "$LFS_TOOL")/stacks/sway.stack" +# the stack parser keeps needs=, and the sway entry declares its own +if [ -f "$_ssU" ]; then + python3 - "$_pmU" "$_ssU" <<'PYNEEDS' || _p="$_p;needs= is not parsed from a stack entry" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +rows = pm._parse_stack_file(sys.argv[2]) +sway = [r for r in rows if r.get("target") == "sway"] +assert sway, "no sway entry in the stack" +needs = (sway[0]["opts"].get("needs") or "") +assert "json-c" in needs, needs +PYNEEDS + # every name declared must be a package the book actually has, or the + # install it triggers cannot work (pcre2 was in the first draft: LFS, not + # BLFS, and the lookup returns nothing) + _bkU="" + for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkU="$_c" && break + done + if [ -n "$_bkU" ]; then + BLFS_BOOK_FILE="$_bkU" python3 - "$_pmU" "$_ssU" <<'PYREAL' || _p="$_p;a needs= entry names something the book does not have" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +for r in pm._parse_stack_file(sys.argv[2]): + for n in (r["opts"].get("needs") or "").replace(",", " ").split(): + assert pm.book_version(n), (r["target"], n, "is not a BLFS package") +PYREAL + fi +fi +# the stack installs them before the entry, through the same door +_gb="$(sed -n '/A GIT ENTRY CAN NAME ITS DEPENDENCIES/,/MACHINE.CONF REACHES A GIT ENTRY/p' "$_pmU")" +printf '%s' "$_gb" | grep -q '_install_via_self(_need, args.yes, recursive=True)' \ + || _p="$_p;a declared dependency is not installed before the entry" +printf '%s' "$_gb" | grep -q 'gather_user(_need).state == "installed"' \ + || _p="$_p;an already-installed dependency is rebuilt every run" +printf '%s' "$_gb" | grep -q 'stack stopped at' \ + || _p="$_p;a dependency that cannot be installed does not stop the stack" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a git stack entry can declare the book packages it needs" +fi -# ---- add-user names the account through the chokepoint ---------------------- # -# It used its argument raw for the ACCOUNT and pkg_owner_name only for the HOME. -# Fine while every caller passes an already-prefixed owner -- cmd_build does. -# last_build_step.sh did not: `lfs-helper add-user wget` produced an account -# called `wget` living in /usr/src/pkgusr/p_wget, and the sanity report found -# both halves without being able to connect them: -# !! accounts without a known prefix: wget urllib3 requests ... -# ? /usr/src/pkgusr/p_wget/ has no matching account (stray directory) -# !! group wget (gid 10082) is outside every convention -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _au="$(_slice_fn "$helper_src" cmd_add_user)" - _p="" - case "$(printf '%s\n' "$_au" | grep -vE '^[[:space:]]*#')" in - *'name="$(pkg_owner_name "$name")"'*) ;; - *) _p="$_p;add-user creates the account from the raw name" ;; - esac - # the normalisation must come BEFORE the account is created, or it renames - # nothing - _i1="$(printf '%s\n' "$_au" | grep -n 'name="$(pkg_owner_name' | head -1 | cut -d: -f1)" - _i2="$(printf '%s\n' "$_au" | grep -n 'useradd ' | head -1 | cut -d: -f1)" - if [ -n "$_i1" ] && [ -n "$_i2" ] && [ "$_i1" -gt "$_i2" ]; then - _p="$_p;add-user normalises the name after creating the account" - fi - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "add-user gives the account and its home the same name" - fi +# ---- check every pinned ref at once (1.14.67) ------------------------------ # +# Second rotted pin in one stack: usrsctp ref=1.0.0 (1.14.44) and now +# error: pathspec '1.24.1' did not match any file(s) known to git +# for foot, whose tags are 1.24.0 and 1.28.0 -- 1.24.1 never existed. Each +# cost a clone, a build attempt and a round trip. 1.14.44's HANDOFF note said +# a --check-refs would turn this class into a five-second report; here it is. +_p="" +_pmV="$(dirname "$LFS_TOOL")/packagemanager" +python3 "$_pmV" stack --help 2>&1 | grep -q '\-\-check-refs' \ + || _p="$_p;there is no way to check a stack's pinned refs" +_cr="$(sed -n '/^def _check_stack_refs/,/^def /p' "$_pmV")" +printf '%s' "$_cr" | grep -q 'ls-remote' \ + || _p="$_p;the check does not ask the remote" +printf '%s' "$_cr" | grep -q 'timeout=' \ + || _p="$_p;a hanging remote would hang the check" +printf '%s' "$_cr" | grep -q '0-9a-f.\{0,8\}7,40' \ + || _p="$_p;a commit id is treated as a tag and reported missing" +# (1.14.69: an entry with no ref is still CONTACTED -- the wording changed +# from "tracks the default branch" to a reachability result) +printf '%s' "$_cr" | grep -q 'no ref=' \ + || _p="$_p;an entry with no ref is reported as broken" +# an unreachable remote is NOT an all-clear -- those are the ones that might +# be wrong, and saying "every pinned ref exists" over them is a false pass +printf '%s' "$_cr" | grep -q 'every ref that COULD be checked exists' \ + || _p="$_p;unreachable remotes are counted as verified" +printf '%s' "$_cr" | grep -q 'those refs are unverified' \ + || _p="$_p;it does not say how many could not be checked" +# SORT TAGS AS VERSIONS. foot's real tags include 1.28.0, and a reverse +# STRING sort offered "1.9.2, 1.9.1, 1.9.0, 1.9, 1.8.2" as the newest -- +# a suggestion that invites a downgrade is worse than none. +python3 - "$_pmV" <<'PYVSORT' || _p="$_p;candidate tags are sorted as strings, not versions" +import sys, re +src = open(sys.argv[1]).read() +i = src.index(" def _vkey(n):") +j = src.index(" near = sorted", i) +ns = {"re": re} +NL = chr(10) +exec(NL.join(l[8:] for l in src[i:j].split(NL)), ns) +tags = ["1.8.2", "1.9", "1.9.0", "1.9.1", "1.9.2", "1.10.0", "1.24.0", + "1.26.1", "1.27.0", "1.28.0"] +top = sorted(tags, key=ns["_vkey"], reverse=True)[:3] +assert top == ["1.28.0", "1.27.0", "1.26.1"], top +# a v-prefix and a mixed tag must not crash it +assert ns["_vkey"]("v2.6.0") < ns["_vkey"]("v2.10.0") +ns["_vkey"]("release-3.2.4") +PYVSORT +# and the summary line is a sentence +printf '%s' "$_cr" | grep -q 'y pins" if bad == 1 else "ies pin' \ + || _p="$_p;the summary says '1 entry pin a ref'" +# it builds nothing: the command exits after reporting +sed -n '/if getattr(args, "check_refs", False)/,+2p' "$_pmV" | grep -q 'sys.exit' \ + || _p="$_p;--check-refs falls through into building the stack" +# and the shipped stack no longer pins a ref that never existed +_ssV="$(dirname "$LFS_TOOL")/stacks/sway.stack" +[ -f "$_ssV" ] && { grep -q 'ref=1.24.1' "$_ssV" \ + && _p="$_p;the stack still pins foot at a tag that does not exist"; } +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a stack's pinned refs can be checked before anything is built" +fi - # last_build_step.sh was the caller that exposed it -- it passed bare names. - # It is gone (1.10.0) and this checked a file that no longer exists, so it - # never ran again. The requirement outlives it: `packagemanager setup` is - # the next caller to create accounts from outside the tool, and it must go - # through add-user rather than useradd. Assert that when setup lands. +# ---- an entry with no ref still has a url (1.14.69) ------------------------ # +# --check-refs reported "every pinned ref exists" while four entries -- the +# ones tracking a default branch -- had not been contacted at all. A +# repository that has been renamed, made private or moved fails exactly like a +# bad ref: at clone time, after everything before it has built. "No ref to +# check" is not the same as "nothing to check". +_p="" +_pmW="$(dirname "$LFS_TOOL")/packagemanager" +_cr2="$(sed -n '/^def _check_stack_refs/,/^def /p' "$_pmW")" +_nb="$(printf '%s' "$_cr2" | sed -n '/AN ENTRY WITH NO REF STILL HAS A URL/,/continue/p')" +printf '%s' "$_nb" | grep -q 'ls-remote' \ + || _p="$_p;an entry with no ref= is never contacted" +printf '%s' "$_nb" | grep -q 'default branch, reachable' \ + || _p="$_p;a reachable default-branch entry is not confirmed" +printf '%s' "$_nb" | grep -q 'the remote has NO branches' \ + || _p="$_p;an empty or moved repository is not reported" +printf '%s' "$_nb" | grep -q 'unreachable += 1' \ + || _p="$_p;an unreachable no-ref remote is counted as verified" +printf '%s' "$_nb" | grep -q 'timeout=' \ + || _p="$_p;the no-ref probe can hang" +# the four local entries point at real repositories, not a placeholder +_ssW="$(dirname "$LFS_TOOL")/stacks/sway.stack" +if [ -f "$_ssW" ]; then + grep -qE '^git +(minibrowser|swbr|swov|swas).*github\.com/nimbin2' "$_ssW" \ + && _p="$_p;the local sway tools still point at the old placeholder urls" + for _e in minibrowser swbr swov swas; do + grep -qE "^git +$_e .*url=https://" "$_ssW" \ + || _p="$_p;$_e has no url" + done +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every git entry is contacted, whether or not it pins a ref" fi -# ---- the sanity report reads the sorted state, and knows when a build is done # -# Two findings on a perfect 105/105 tree, both from the report and neither from -# the build: -# -# adopted.list: MISSING (the adoption gate will never close) -# -- it read $STATE/adopted.list; the file is progress/adopted.list -# -# !! /usr/bin is already STICKY -- sealed before the build finished -# -- the build HAD finished, and seal-install-dirs is supposed to do that -# -# The sticky bit means opposite things on either side of that line: during the -# build it is a bug, afterwards its ABSENCE is. -san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" -if [ -f "$san" ]; then - _p="" - grep -q 'PROG="\$STATE/progress"' "$san" \ - || _p="$_p;the sanity report still reads the flat state layout" - grep -q 'BUILD_FINISHED' "$san" \ - || _p="$_p;the sanity report does not know whether the build finished" - grep -q 'the build finished but the seal never fired' "$san" \ - || _p="$_p;a finished build with no seal is not reported" - # an empty manifest is a config step that installed nothing, not an orphan - grep -q 'grep -q . "\$m" 2>/dev/null || continue' "$san" \ - || _p="$_p;empty manifests are still reported as missing an account" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the sanity report reads the sorted state and judges the seal in context" - fi +# ---- retry the phase that failed, not the whole build (1.14.70) ------------ # +# "is it possible that if we fail the install part we recompile everything?" +# Yes. foot needed seven rounds to collect its directory grants (/etc/xdg, +# then zsh site-functions, then fish vendor_completions.d -- meson reports one +# at a time), and EVERY round re-cloned the repository, re-ran meson and +# recompiled all 150 targets, to redo an install that had failed on one +# completion file. Granting a directory cannot invalidate a compile. +_p="" +_hlpN="$helper_src" +_rt="$(_slice_fn "$_hlpN" cmd_build)" +printf '%s' "$_rt" | grep -q 'RETRY THE PHASE THAT FAILED' \ + || _p="$_p;the retry still re-runs the whole build" +printf '%s' "$_rt" | grep -q 'install|configure) _retry_phase=' \ + || _p="$_p;a failure in install or configure is not retried on its own" +# it must learn WHICH phase failed from run_phase_as, not by parsing the log a +# second time -- the bookkeeping has one reader (see the 1.12.x rule above) +printf '%s' "$_rt" | grep -q 'LAST_FAILED_PHASE' \ + || _p="$_p;the retry does not use the phase run_phase_as recorded" +_slice_fn "$_hlpN" run_phase_as | grep -q 'LAST_FAILED_PHASE="$_failed"' \ + || _p="$_p;run_phase_as does not publish which phase failed" +printf '%s' "$_rt" | grep -q 'run_phase_as "$owner" "$staged" "$_retry_phase"' \ + || _p="$_p;the retry does not use the narrowed phase" +# ...and `all` still finishes: configure runs after a retried install +printf '%s' "$_rt" | grep -q '\[ "$_retry_phase" = install \]' \ + || _p="$_p;a retried install skips the configure step that would have followed" +# a failure in unpack or build is still retried as the whole thing +printf '%s' "$_rt" | grep -q '_retry_phase="$phase"' \ + || _p="$_p;an early-phase failure no longer retries everything" +# the runner really can redo one phase against an existing build tree +_rd="$T/retryphase"; mkdir -p "$_rd/src" +{ + echo '#!/bin/bash' + echo 'name_version="demo-2.0"' + echo 'build_pkg() { echo "COMPILING"; touch built.marker; }' + echo 'install_pkg() {' + echo '[ -f built.marker ] || { echo "NO BUILD ARTIFACTS"; exit 9; }' + echo "if [ -f $_rd/allow ]; then echo \"INSTALL OK\"; else echo \"install: cannot create x: Permission denied\" >&2; exit 13; fi" + echo '}' + echo 'configure_pkg() { echo "CONFIGURED"; }' + echo '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"' +} > "$_rd/install_demo" +_o="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_rd/src" \ + bash "$_rd/install_demo" all 2>&1)" +case "$_o" in + *"phase 'install' FAILED"*) ;; + *) _p="$_p;the fixture did not fail where it was meant to" ;; +esac +touch "$_rd/allow" +_o2="$(env PKGUSR_LIB="$PHASES_LIB" LFS_BUILD_ROOT="$_rd/src" \ + bash "$_rd/install_demo" install 2>&1)" +case "$_o2" in + *"INSTALL OK"*) ;; + *) _p="$_p;an install-only retry does not work against an existing build tree" ;; +esac +case "$_o2" in + *COMPILING*) _p="$_p;an install-only retry recompiles" ;; +esac +case "$_o2" in + *"NO BUILD ARTIFACTS"*) _p="$_p;the build tree was thrown away before the retry" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a permission grant retries the install, not the compile" +fi + +# ---- the log belongs beside the package (1.14.71) -------------------------- # +# "shouldnt we log everything within the pkgusers home log dir?" Yes, and +# every failure box already CLAIMS we do: it ends with +# logs: /usr/src/pkgusr/p_json-c/log +# which contained one empty .diff. The phase log went to the central +# $LOGS/<name>-<phase>.log and nowhere else -- and that copy is overwritten by +# the next run of the same package. The history someone wants three days +# later was never kept where they were told to look. +_p="" +_hlpO="$helper_src" +_rp2="$(_slice_fn "$_hlpO" run_phase_as)" +printf '%s' "$_rp2" | grep -q 'THE LOG BELONGS BESIDE THE PACKAGE' \ + || _p="$_p;the phase log is still not copied into the package's home" +printf '%s' "$_rp2" | grep -q 'pkgusr_home_for "$owner"' \ + || _p="$_p;the copy does not go to the owner's home" +printf '%s' "$_rp2" | grep -q 'chown "$owner:$owner"' \ + || _p="$_p;the copied log is not owned by the package user" +# it really copies, with a timestamp so history accumulates +_lg="$T/pkglog"; mkdir -p "$_lg/home/log" "$_lg/central" +python3 - "$_hlpO" "$_lg" <<'PYLOG' || _p="$_p;the log is not placed in the package home" +import sys, os, subprocess +src = open(sys.argv[1]).read() +d = sys.argv[2] +i = src.index(" # THE LOG BELONGS BESIDE THE PACKAGE.") +j = src.index(' if [ "$rc" = 0 ] && [ -n "$log" ] && [ -f "$log" ]; then', i) +body = "\n".join(l[4:] if l.startswith(" ") else l for l in src[i:j].split("\n")) +open(os.path.join(d, "central", "foo-all.log"), "w").write("build output\n") +sh = os.path.join(d, "t.sh") +open(sh, "w").write("set -u\npkgusr_home_for() { echo %s/home; }\n" + "owner=p_foo\nlog=%s/central/foo-all.log\nrc=0\n" + % (d, d) + body) +subprocess.run(["bash", sh], capture_output=True) +got = os.listdir(os.path.join(d, "home", "log")) +assert got, "nothing was copied into the package's log dir" +assert got[0].startswith("foo-all-") and got[0].endswith(".log"), got +assert open(os.path.join(d, "home", "log", got[0])).read() == "build output\n" +# ...and old logs are pruned, so a package that fails twenty times does not +# fill the disk with megabyte webkitgtk logs +for n in range(14): + open(os.path.join(d, "home", "log", "old-%02d.log" % n), "w").write("x") +subprocess.run(["bash", sh], capture_output=True) +kept = [f for f in os.listdir(os.path.join(d, "home", "log")) if f.endswith(".log")] +assert len(kept) == 10, len(kept) +PYLOG +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a phase log is kept in the package's own log directory" +fi + +# ---- a network stack, and every entry checked (1.14.72) -------------------- # +# "is it possible that we miss wpa_supplicant?" Yes: the sway stack builds a +# desktop and nothing in it brings up a network. Their wifi script needs +# wpa_cli, wpa_supplicant and ifup; none of the three was installed. +# stacks/network.stack covers it -- libnl, wpa_supplicant, iw, +# wireless_tools, libmnl, iptables, net-tools -- every one a BLFS package. +_p="" +_pmX="$(dirname "$LFS_TOOL")/packagemanager" +_nsX="$(dirname "$LFS_TOOL")/stacks/network.stack" +[ -f "$_nsX" ] || _p="$_p;there is no network stack" +_bkX="" +for _c in "$(dirname "$LFS_TOOL")"/BLFS-BOOK-*nochunks.html; do + [ -f "$_c" ] && _bkX="$_c" && break +done +if [ -f "$_nsX" ] && [ -n "$_bkX" ]; then + BLFS_BOOK_FILE="$_bkX" python3 - "$_pmX" "$_nsX" <<'PYNET' || _p="$_p;the network stack does not cover the wifi script" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +rows = pm._parse_stack_file(sys.argv[2]) +have = {r["target"] for r in rows} +for want in ("wpa_supplicant", "libnl", "iptables"): + assert want in have, (want, sorted(have)) +PYNET +fi +# EVERY book entry in EVERY shipped stack must be a package the book has. +# This is the check that would have caught `needs=pcre2` (1.14.66) before it +# shipped -- a stack naming something unbuildable fails late and far from the +# cause. +if [ -n "$_bkX" ]; then + BLFS_BOOK_FILE="$_bkX" python3 - "$_pmX" "$(dirname "$LFS_TOOL")/stacks" <<'PYALLSTACK' || _p="$_p;a shipped stack names a package the book does not have" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2] +bad = [] +for f in sorted(os.listdir(d)): + if not f.endswith(".stack"): + continue + for r in pm._parse_stack_file(os.path.join(d, f)): + # a `book` entry that carries url= is book-preferred with a GIT + # fallback (elogind: the SysV logind, which the systemd book has no + # page for). The tool already falls back; only an entry with no + # fallback has to resolve. + if (r.get("kind") == "book" and not r["opts"].get("url") + and not pm.book_version(r["target"])): + bad.append((f, r["target"])) + for n in (r["opts"].get("needs") or "").replace(",", " ").split(): + if not pm.book_version(n): + bad.append((f, r["target"] + " needs " + n)) +assert not bad, bad +PYALLSTACK +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a network stack exists, and every stack entry names a real package" fi -# ---- every tool can say which build it is ----------------------------------- # -# Section 0 of the sanity report exists because a stale copy in the chroot has -# cost more debugging time here than anything else. packagemanager_install had -# no --version, so that row read -# packagemanager_install Please provide an install script with $2 -pmi="$(dirname "$LFS_TOOL")/packagemanager_install" -if [ -f "$pmi" ]; then - _v="$(bash "$pmi" --version 2>&1)" - case "$_v" in - packagemanager_install*build*) ok "every tool reports its own build id" ;; - *) bad "packagemanager_install cannot report its build id: $_v" ;; - esac +# ---- an --enable must beat the book's --disable (1.14.74) ------------------ # +# The user wants iptables-nft (legacy is being retired upstream). The book's +# page configures --disable-nftables, so the flag comes from machine.conf -- +# and configure_args PREPENDED it: +# ./configure --enable-nftables --prefix=/usr \ +# --disable-nftables ... +# autoconf takes the LAST occurrence, so the book's flag won and the setting +# did nothing at all. Same class as 1.14.60: an option that reaches the +# command and is then ignored is worse than one that never arrives. +_p="" +_pmY="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmY" "$T/confargs" <<'PYCONF' || _p="$_p;configure_args does not override the opposite flag" +import sys, os, re, subprocess, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_i") +open(p, "w").write( + "build_pkg() {\n" + "./configure --prefix=/usr \\\n" + " --disable-nftables \\\n" + " --enable-libipq &&\n" + "make\n}\n") +pm._machine_opts_for = lambda a: {"configure_args": "--enable-nftables"} +pm.apply_machine_opts("iptables", p) +body = open(p).read() +assert "--enable-nftables" in body, body +assert "--disable-nftables" not in body, body +# the rest of the command survives +assert "--enable-libipq" in body, body +assert "--prefix=/usr" in body, body +# and no dangling continuation is left behind: removing the token alone gave +# --prefix=/usr \ \ +# which is a backslash-escaped space -- an argument +assert "\\ \\" not in body, body +assert subprocess.run(["bash", "-n", p]).returncode == 0, body +# an unrelated flag is not touched +open(p, "w").write("build_pkg() {\n./configure --disable-static --enable-shared\n}\n") +pm._machine_opts_for = lambda a: {"configure_args": "--enable-nftables"} +pm.apply_machine_opts("iptables", p) +assert "--disable-static" in open(p).read(), open(p).read() +PYCONF +# the network stack carries libnftnl, ahead of iptables, and says where the +# version came from -- netfilter.org's own news page, not a guess +_nsY="$(dirname "$LFS_TOOL")/stacks/network.stack" +if [ -f "$_nsY" ]; then + grep -q '^tar libnftnl' "$_nsY" \ + || _p="$_p;libnftnl is not in the network stack" + _ln="$(grep -n '^tar libnftnl' "$_nsY" | cut -d: -f1)" + _ip="$(grep -n '^book iptables' "$_nsY" | cut -d: -f1)" + [ -n "$_ln" ] && [ -n "$_ip" ] && [ "$_ln" -lt "$_ip" ] \ + || _p="$_p;libnftnl is not built before iptables" + grep -q 'sha256' "$_nsY" \ + || _p="$_p;nothing tells the reader to check the tarball" +fi +# ...and the flag it needs ships in machine.conf +_mcY="$(dirname "$LFS_TOOL")/stacks/machine.conf" +[ -f "$_mcY" ] && { grep -q 'enable-nftables' "$_mcY" \ + || _p="$_p;the shipped machine.conf does not enable the nft backend"; } +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a configure_args flag overrides the book's opposite, and iptables-nft is set up" fi -# ---- no finding that repairing makes worse ---------------------------------- # -# Two passes disagreed about a mail spool, and each undid the other: -# -# lfs-helper verify --fix builduser /var/mail/lfs lfs -> root -# lfs-helper verify 1 root-owned file(s) that NO manifest claims: -# /var/mail/lfs -# -# The build-user pass gave it to root because the build user held it; the -# unclaimed pass then reported that root holds a file no manifest claims. -# Repairing it produced the next run's finding. -# -# Shadow creates /var/mail/<name> for every account it makes, so a spool arrives -# as a side effect of `useradd` and no package ever installs one. It belongs to -# the account it is named for, and neither pass should touch it. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _o="$( ETC=/etc; SNAP_ROOT=/; STATE=/usr/src/lfs-pkgusr; WRAPPERS=/usr/lib/pkgusr - eval "$(_slice_fn "$helper_src" never_claim_list)" - eval "$(_slice_fn "$helper_src" _never_claim_load)" - eval "$(_slice_fn "$helper_src" is_never_claimed)" - command -v is_never_claimed >/dev/null 2>&1 \ - || { echo "there is no is_never_claimed"; exit 0; } - for f in /var/mail/lfs /var/mail/p_gcc; do - is_never_claimed "$f" || echo "$f is still fought over by two passes" - done - # not the whole of /var, though -- a package's files there are its own - is_never_claimed /var/lib/something \ - && echo "the exclusion is too broad: it swallowed /var/lib" )" - if [ -z "$_o" ]; then - ok "a mail spool belongs to its account, and no pass claims it" - else - printf '%s\n' "$_o" | while IFS= read -r m; do [ -n "$m" ] && bad "$m"; done - fi +# ---- a systemd unit on a system with no systemd (1.14.75) ------------------ # +# wpa_supplicant installed its binaries and man pages and then died on +# install: target '/usr/lib/systemd/system/': No such file or directory +# The systemd book's page ends by installing a unit file; on a SysV system +# that directory does not exist. Same situation the systemctl shim was +# written for -- the book assumes an init this machine does not run -- and the +# install wrapper is where it has to be caught, because it is a file copy and +# not a systemctl call. +_p="" +_hlpP="$helper_src" +_iw="$T/instwrap"; mkdir -p "$_iw" +python3 - "$_hlpP" "$_iw" <<'PYWRAP' +import sys, os +s = open(sys.argv[1]).read() +i = s.index(' cat > "$WRAPPERS/install" <<') +j = s.index("\nEOF\n", i) +body = s[i:j] +body = body[body.index("\n") + 1:] +p = os.path.join(sys.argv[2], "install") +open(p, "w").write(body) +os.chmod(p, 0o755) +PYWRAP +bash -n "$_iw/install" || _p="$_p;the install wrapper does not parse" +grep -q 'A SYSTEMD UNIT ON A SYSTEM WITH NO SYSTEMD' "$_iw/install" \ + || _p="$_p;the wrapper does not handle a systemd unit destination" +# force the SysV case: no unit dir, no systemctl +sed -e 's|\[ ! -d /usr/lib/systemd/system \]|true|' \ + -e 's|! command -v systemctl >/dev/null 2>&1|true|' \ + "$_iw/install" > "$_iw/install-sysv" +echo unit > "$_iw/foo.service" +( cd "$_iw" && bash install-sysv -m644 foo.service /usr/lib/systemd/system/ ) 2>"$_iw/err" +[ $? = 0 ] || _p="$_p;a unit install still fails the package on a SysV system" +grep -q "does not run systemd" "$_iw/err" \ + || _p="$_p;the skip is silent -- nobody would know the unit was dropped" +# ...and an ordinary install is untouched +( cd "$_iw" && bash install-sysv -m644 foo.service "$_iw/out.txt" ) 2>/dev/null +[ -f "$_iw/out.txt" ] || _p="$_p;an ordinary install no longer copies anything" +# on a machine that DOES run systemd, the unit must be installed for real +sed -e 's|\[ ! -d /usr/lib/systemd/system \]|false|' "$_iw/install" > "$_iw/install-sd" +mkdir -p "$_iw/unitdir" +( cd "$_iw" && bash install-sd -m644 foo.service "$_iw/unitdir/foo.service" ) 2>/dev/null +[ -f "$_iw/unitdir/foo.service" ] \ + || _p="$_p;a systemd host no longer gets its unit files" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a unit file from a systemd-book page is skipped, not fatal, on SysV" +fi - # the general property: nothing the build-user pass hands to root may then be - # reported by the unclaimed pass, or --fix and verify never agree - case "$(_slice_fn "$helper_src" _vfy_build_user_leftovers)" in - *is_never_claimed*) ok "the build-user pass skips files no package may own" ;; - *) bad "the build-user pass can create a finding for the unclaimed pass" ;; - esac +# ---- the pattern has to be a real path (1.14.76) --------------------------- # +# A finished stack ended with +# results: packagemanager errors logs: /usr/src/p_<pkg>/log +# The hint was built by hand -- os.path.join(BASE_DIR, pkgusr_name('<pkg>'), +# 'log') -- and got it wrong twice over: the placeholder never gets filled in, +# and the path is missing the `pkgusr/` component every package home has. +# Someone following it found nothing, and until 1.14.71 the logs were not +# there to find either. +_p="" +_pmZ="$(dirname "$LFS_TOOL")/packagemanager" +grep -q "pkgusr_name('<pkg>')" "$_pmZ" \ + && _p="$_p;the log hint is still assembled by hand" +[ "$(grep -c "_pkg_log_dir('<name>')" "$_pmZ")" -ge 2 ] \ + || _p="$_p;the summary lines do not use the function that knows the path" +python3 - "$_pmZ" <<'PYPATH' || _p="$_p;the printed log path is not where logs live" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +shown = pm._pkg_log_dir("<name>") +# it must be under the package-user root, not one level up +assert "/pkgusr/" in shown or pm.PKGUSR_SUBDIR == "", shown +# and for a real package it must be exactly that package's log directory +real = pm._pkg_log_dir("zlib") +assert real.endswith("/log"), real +assert pm.pkgusr_name("zlib") in real, real +# the placeholder form differs from the real one only in the name +assert shown.replace("<name>", "zlib") == real, (shown, real) +PYPATH +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the log path a summary prints is the path logs are actually in" fi -# ---- every door to `chown -R lfs` has the same guard ------------------------ # -# There were FOUR places that hand the build tree to the `lfs` user, and only -# one of them had the check that refuses on a tree past chapter 7: -# -# _chown_tree_to_lfs book 4.3, guarded -# session --run re-applied it by hand, unguarded -# the pre-build check "fixing N dir(s) still owned by root", unguarded -# fix-ownership --run the explicit repair, unguarded -# -# On a finished system any of the other three would have handed the whole thing -# to the build user in one pass -- the exact damage the guard exists to prevent, -# through a door the guard was not on. It already happened once via a replayed -# step 4.2. -python3 - "$LFS_TOOL" <<'PYCHOWN' -import re, sys -raw = open(sys.argv[1]).read() -# Comments name the command they are explaining -- every fix in this file says -# what it fixed -- so scan CODE only. Matching the explanation makes a test -# that cannot pass no matter what the code does. -src = "\n".join("" if l.lstrip().startswith("#") else l - for l in raw.splitlines()) -problems = [] -# every recursive chown to lfs, and the function it sits in -for m in re.finditer(r"chown -R lfs", src): - start = src.rfind("\ndef ", 0, m.start()) - if start < 0: - problems.append("a chown -R lfs sits outside any function") - continue - end = src.find("\ndef ", m.end()) - body = src[start:end if end > 0 else len(src)] - name = re.match(r"\ndef ([a-zA-Z_0-9]+)", body).group(1) - # the one that only chowns the state dir is not handing over the tree - if "PKGUSR_DIR" in src[m.start()-120:m.end()+120]: - continue - if "_handover_is_safe" not in body: - problems.append("%s hands the tree to the build user unguarded" % name) -if problems: - for p in sorted(set(problems)): - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS every path that gives the tree to the build user is guarded") -PYCHOWN -_count_rc $? +# ---- boot scripts are a package too (1.14.77) ------------------------------ # +# /etc/init.d after an LFS build holds LFS's own scripts and nothing else: no +# dbus, no alsa, no cron. The user noticed the missing fcron script and asked +# whether the LFS install had gone wrong. It had not -- those scripts live in +# blfs-bootscripts, a separate tarball and not a BLFS package, which is +# exactly why a package page's `make install-<service>` is commented out by +# the generator (1.14.20): the target is in THAT tarball, not in the package. +_p="" +_pmA2="$(dirname "$LFS_TOOL")/packagemanager" +_ss2="$(dirname "$LFS_TOOL")/stacks/services.stack" +[ -f "$_ss2" ] || _p="$_p;there is no services stack" +if [ -f "$_ss2" ]; then + python3 - "$_pmA2" "$_ss2" <<'PYSVC' || _p="$_p;the services stack does not install the boot scripts" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +rows = pm._parse_stack_file(sys.argv[2]) +by = {r["target"]: r for r in rows} +assert "blfs-bootscripts" in by, sorted(by) +inst = by["blfs-bootscripts"]["opts"].get("install", "") +# the services this system actually runs +for t in ("install-dbus", "install-alsa", "install-bluetooth", "install-acpid", + "install-fcron", "install-service-wpa"): + assert t in inst, (t, inst) +# a package must be built before its boot script is installed +names = [r["target"] for r in rows] +assert names.index("acpid") < names.index("blfs-bootscripts"), names +assert names.index("fcron") < names.index("blfs-bootscripts"), names +# have= keeps it idempotent +assert by["blfs-bootscripts"]["opts"].get("have"), by["blfs-bootscripts"]["opts"] +PYSVC +fi +# A LINE MAY CONTINUE. Every entry had to be one line, which is why +# sway.stack's are 300 characters wide and unreadable. +python3 - "$_pmA2" "$T/cont" <<'PYCONT' || _p="$_p;a stack entry cannot be split over lines" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "t.stack") +open(p, "w").write( + "# a comment\n" + "tar demo \\\n" + " url=https://example.invalid/demo-1.0.tar.xz \\\n" + " pkg=demo-1.0.tar.xz \\\n" + ' build="true" \\\n' + ' install="make install-one install-two"\n' + "book zlib\n") +rows = pm._parse_stack_file(p) +assert len(rows) == 2, rows +assert rows[0]["target"] == "demo", rows[0] +assert rows[0]["opts"]["install"] == "make install-one install-two", rows[0] +assert rows[1]["target"] == "zlib", rows[1] +# a dangling backslash at the end of the file is an error, not a silent drop +open(p, "w").write("tar demo \\\n") +try: + pm._parse_stack_file(p) + raise AssertionError("a dangling continuation was accepted") +except ValueError as e: + assert "backslash" in str(e), e +PYCONT +# the stacks that shipped before this still parse unchanged +for _f in sway network services; do + _sf="$(dirname "$LFS_TOOL")/stacks/$_f.stack" + [ -f "$_sf" ] || continue + python3 - "$_pmA2" "$_sf" <<'PYSTILL' || _p="$_p;$_f.stack no longer parses" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm._parse_stack_file(sys.argv[2]), "no entries" +PYSTILL +done +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "boot scripts install from their own tarball, and a stack entry can wrap" +fi -# ---- session says what to do next ------------------------------------------- # -# It ended on `export LFS=/mnt/lfs # <-- run this in your shell` and stopped. -# Every other command in this tool finishes with a Next: line; this one left you -# configured and guessing. -python3 - "$LFS_TOOL" <<'PYNEXT' -import re, sys -src = open(sys.argv[1]).read() -m = re.search(r"^def cmd_bs_session\(.*?(?=\n\ndef )", src, re.S | re.M) -problems = [] -if not m: - problems.append("cmd_bs_session is gone") -else: - body = m.group(0) - # the dry-run branch and the --run branch both end somewhere useful - if body.count('"Next:"') < 2: - problems.append("session does not say what to do next on both paths") - if "lfs run" not in body: - problems.append("session never points at `lfs run`") - if "build-system next" not in body: - problems.append("session never offers `build-system next`") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS session ends by saying what to run next") -PYNEXT -_count_rc $? +# ---- a service account is root's job (1.14.78) ----------------------------- # +# fcron built completely and then died in its install: +# groupadd: cannot lock /etc/group; try again later +# Group "fcron" does not exist : please create it +# Its page's own commands are +# groupadd -g 22 fcron && +# useradd -d /dev/null -c "Fcron User" -g fcron -s /bin/false -u 22 fcron +# and the install phase runs as p_fcron, which cannot write /etc/group. Any +# page that needs a system account (fcron, avahi, sshd, ...) is in this +# position. +_p="" +_hlpQ="$helper_src" +_sa="$(_slice_fn "$_hlpQ" create_service_accounts_from_script)" +[ -n "$_sa" ] || _p="$_p;nothing creates the accounts a page asks for" +printf '%s' "$_sa" | grep -q 'id -u..\? = 0' \ + || _p="$_p;it would try to create accounts when not root" +printf '%s' "$_sa" | grep -q 'getent group' \ + || _p="$_p;an existing group would be created again" +_slice_fn "$_hlpQ" cmd_build | grep -q 'create_service_accounts_from_script' \ + || _p="$_p;the build never calls it" +# the NAME is the last argument: stripping "-x value" pairs by hand tripped +# over the quoted comment in useradd -c "Fcron User" +_nm="$(printf '%s' 'useradd -d /dev/null -c "Fcron User" -g fcron -s /bin/false -u 22 fcron &&' \ + | sed 's/&&.*//' | awk '{print $NF}')" +[ "$_nm" = fcron ] || _p="$_p;the account name is parsed as '$_nm'" +# and the wrappers make the package user's own attempt a no-op once it exists +python3 - "$_hlpQ" "$T/uwrap" <<'PYUW' +import sys, os +s = open(sys.argv[1]).read() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +for w in ("groupadd", "useradd"): + i = s.index(' cat > "$WRAPPERS/%s" <<' % w) + j = s.index("\nEOF\n", i) + body = s[i:j] + body = body[body.index("\n") + 1:] + p = os.path.join(d, w) + open(p, "w").write(body) + os.chmod(p, 0o755) +PYUW +for _w in groupadd useradd; do + bash -n "$T/uwrap/$_w" || _p="$_p;the $_w wrapper does not parse" +done +# an account that exists -> skip, exit 0 (root is the caller here) +_o="$(bash "$T/uwrap/groupadd" -g 99 root 2>&1)"; _rc=$? +[ "$_rc" = 0 ] || _p="$_p;groupadd for an existing group is still an error" +case "$_o" in *"already exists"*) ;; *) _p="$_p;the skip is silent" ;; esac +_o2="$(bash "$T/uwrap/useradd" -u 99 root 2>&1)"; _rc2=$? +[ "$_rc2" = 0 ] || _p="$_p;useradd for an existing user is still an error" +# ...and a package user asked to create a NEW one is told who can +grep -q 'a package user cannot' "$T/uwrap/groupadd" \ + || _p="$_p;a package user gets no explanation when the account is missing" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a page that needs a system account gets one, created by root" +fi -# ---- comm gets sorted input, in one collation ------------------------------- # -# The snapshot diff is `comm -13 <(before) <(after)`. comm compares line by -# line and TRUSTS both sides to be sorted the same way; it cannot detect -# otherwise, it just returns the wrong answer. Two things broke that: -# -# * the two snapshots may have been written under different locales -- a -# package user's profile sets LC_ALL=POSIX and root's shell may not -# * load_snapshot rewrites only the lines carrying the host mount prefix, so -# a file holding both forms comes out sorted under neither -# -# The visible symptom is a package installing forty files and being told it -# installed none. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - for _fn in snapshot snapshot_dirs load_snapshot; do - _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ - || _p="$_p;$_fn does not pin the collation comm depends on" - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "both sides of the snapshot diff are sorted the same way" - fi +# ---- a configuration step that needs root (1.14.79) ------------------------ # +# fcron built, created its system account (1.14.78), installed 74 files -- and +# then its configure phase said +# must be privileged to use -u +# from `fcrontab -u systab`. BLFS's "Configuring" sections are written for +# the root user: they are system configuration, not part of installing a +# package, and no directory grant can help because the command itself refuses +# to run unprivileged. --as-root already existed for exactly this. +_p="" +_hlpR="$helper_src" +_cfg="$T/cfgroot"; mkdir -p "$_cfg" +python3 - "$_hlpR" > "$_cfg/rep.sh" <<'PYSLICEB' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0') +print('LAST_FAILED_PHASE="${FAILED_PHASE:-configure}"') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICEB +printf '## Configuration\nmust be privileged to use -u\n' > "$_cfg/cfg.log" +_o="$(bash -c '. "'"$_cfg"'/rep.sh"; report fcron all "'"$_cfg"'/cfg.log"' 2>&1)" +case "$_o" in + *"CONFIGURATION step, and it needs root"*) ;; + *) _p="$_p;a privileged configure step is not recognised" ;; +esac +case "$_o" in + *"--phase configure --force --as-root"*) ;; + *) _p="$_p;the retry does not use the flag that exists for this" ;; +esac +# ...and it does not fire for a BUILD failure with the same words +printf 'checking...\nmust be privileged to use -u\n' > "$_cfg/build.log" +_o2="$(bash -c 'FAILED_PHASE=build; . "'"$_cfg"'/rep.sh"; report fcron all "'"$_cfg"'/build.log"' 2>&1)" +case "$_o2" in + *"CONFIGURATION step"*) _p="$_p;a build failure is called a configuration problem" ;; +esac +# the flag it names must actually exist +grep -q -- '--as-root) as_root=1' "$_hlpR" \ + || _p="$_p;--as-root is not a real option" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a configure step that needs privilege says so, and how to run it" +fi - # the mixed-prefix case, end to end - _sd="$T/snapsort"; mkdir -p "$_sd" - printf '%s\n' /mnt/lfs/usr/bin/z /usr/bin/a /mnt/lfs/usr/bin/b > "$_sd/snap" - _got="$( SNAP_ROOT=/; LFS_HOST_MOUNT=/mnt/lfs - eval "$(_slice_fn "$helper_src" load_snapshot)" - load_snapshot "$_sd/snap" | tr '\n' ' ' )" - case "$_got" in - "/usr/bin/a /usr/bin/b /usr/bin/z ") ok "a snapshot holding both path forms comes back sorted" ;; - *) bad "load_snapshot returns unsorted input to comm: [$_got]" ;; - esac +# ---- the commands we print must be commands (1.14.80) ---------------------- # +# $ lfs-helper build fcron --phase configure --force --as-root +# !! unknown option: --as-root +# 1.14.79 printed that advice confidently. The flag existed -- on +# cmd_run_script, one function away -- and cmd_build did not take it. Then, +# once accepted, `local as_root=0` inside cmd_build shadowed it, so the flag +# parsed cleanly and did nothing. Two ways to be wrong about one word. +_p="" +_hlpS="$helper_src" +# cmd_build accepts it, and does not immediately overwrite it +_cb="$(_slice_fn "$_hlpS" cmd_build)" +printf '%s' "$_cb" | grep -q -- '--as-root) as_root=1' \ + || _p="$_p;lfs-helper build does not accept --as-root" +printf '%s' "$_cb" | grep -q 'local as_root=0 staged' \ + && _p="$_p;a later local re-declaration shadows the flag again" +printf '%s' "$_cb" | grep -q 'as_root=0' \ + || _p="$_p;as_root has no default, so it is unset under set -u" +# it really parses: an unknown option dies, a known one does not +_o="$(bash "$_hlpS" build --this-is-not-an-option 2>&1)" +case "$_o" in *"unknown option"*) ;; *) _p="$_p;an unknown option is no longer refused" ;; esac +_o2="$(bash "$_hlpS" build --as-root 2>&1)" +case "$_o2" in + *"unknown option: --as-root"*) _p="$_p;--as-root is still refused" ;; +esac +# THE GENERAL RULE: every long option the failure report suggests must be +# accepted by the command it names. This is the check that would have caught +# it before it shipped. +python3 - "$_hlpS" <<'PYFLAGS' || _p="$_p;the failure report suggests an option no command takes" +import sys, re +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +report = s[i:j] +# every "lfs-helper build ... --flag" the report prints +flags = set() +for line in report.split("\n"): + if "lfs-helper build" not in line: + continue + flags.update(re.findall(r"--[a-z][a-z-]+", line)) +assert flags, "the report suggests no build flags at all -- did the slice move?" +i2 = s.index("cmd_build() {") +j2 = s.index("\n}\n", i2) +parser = s[i2:j2] +missing = [f for f in sorted(flags) if f + ")" not in parser and f + "|" not in parser] +assert not missing, missing +PYFLAGS +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every option the failure report suggests is one the command accepts" fi -# ---- "installed NO files" has to say which of the three it is --------------- # -# There are exactly three ways to reach zero and they need different fixes: -# nothing was written, it was written somewhere excluded from tracking, or it -# was written over files another package owns and dropped by the filter. -# Saying only "check the log" meant reading a thousand lines of make output to -# find out which. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _cb="$(_slice_fn "$helper_src" cmd_build)" - _p="" - case "$_cb" in - *'new files (snapshot diff)'*) ;; - *) _p="$_p;the zero-files failure does not report the snapshot diff count" ;; - esac - case "$_cb" in - *'files touched since stamp'*) ;; - *) _p="$_p;the zero-files failure does not report the touched count" ;; - esac - case "$_cb" in - *'already belongs to another package'*) ;; - *) _p="$_p;the zero-files failure does not distinguish a dropped install" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a package that installed nothing says which kind of nothing" - fi +# ---- a package that checks its own file ownership (1.14.81) ---------------- # +# Running fcron's configure as root got further and then: +# ERROR Conf file (/etc/fcron.conf) must be owned by root:fcron +# ERROR Could not chdir to /var/spool/fcron: Permission denied +# A package-user install CANNOT satisfy that. Everything it writes belongs to +# p_fcron, and the chown wrapper skips the package's own chown deliberately -- +# that skipping is the whole model. For a daemon that validates ownership +# before it will start, the last step belongs to root, and the package's own +# error says precisely what it wants. +_p="" +_hlpT="$helper_src" +_ow="$T/ownership"; mkdir -p "$_ow" +python3 - "$_hlpT" > "$_ow/rep.sh" <<'PYSLICEC' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; s="/x"; staged="/x"') +print('auto_repair=1; auto_fix=0; _perm_ish=0; LAST_FAILED_PHASE=configure') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(body) +print('}') +PYSLICEC +printf 'ERROR Conf file (/etc/fcron.conf) must be owned by root:fcron and (no more than) 644 : ignored\nERROR Could not chdir to /var/spool/fcron: Permission denied\n' \ + > "$_ow/own.log" +_o="$(bash -c '. "'"$_ow"'/rep.sh"; report fcron configure "'"$_ow"'/own.log"' 2>&1)" +case "$_o" in + *"CHECKS who owns its files"*) ;; + *) _p="$_p;an ownership demand is not recognised" ;; +esac +# the exact commands, taken from the package's own words +case "$_o" in + *"chown root:fcron /etc/fcron.conf"*) ;; + *) _p="$_p;the conf file's required ownership is not turned into a command" ;; +esac +case "$_o" in + *"chown -R fcron:fcron /var/spool/fcron"*) ;; + *) _p="$_p;the spool directory is not mentioned" ;; +esac +# it explains WHY a package-user install cannot do this itself +case "$_o" in + *"chown wrapper skips"*) ;; + *) _p="$_p;it does not say why the ownership is wrong in the first place" ;; +esac +# and it stays quiet for a log with no ownership complaint +printf 'make: *** [Makefile:9: all] Error 1\n' > "$_ow/plain.log" +_o2="$(bash -c '. "'"$_ow"'/rep.sh"; report demo configure "'"$_ow"'/plain.log"' 2>&1)" +case "$_o2" in + *"CHECKS who owns"*) _p="$_p;every failure is blamed on ownership" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package that validates its file ownership says which chown fixes it" fi -# ---- a re-run over its own work is not a silent failure --------------------- # -# The zero-files check counts what THIS RUN wrote, which is the wrong question -# when a package is built a second time. Many installers skip a file that is -# already up to date -- perl's ExtUtils::Install prints "Installing <path>" for -# every target and copies only the stale ones -- so a second run writes nothing: -# the snapshot diff sees nothing new, no mtime beats the stamp, and the count is -# 0 for the best possible reason. -# -# xml-parser hit exactly that: forty files on disk, every one owned by -# p_xml-parser, and the step failed as a no-op. Counting cannot tell that from -# a real failure. The tree can. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" _package_already_owns_files | grep -q . \ - || _p="$_p;there is no _package_already_owns_files" - case "$(_slice_fn "$helper_src" cmd_build)" in - *_package_already_owns_files*) ;; - *) _p="$_p;a re-run over its own work still fails as a silent no-op" ;; - esac - # it must ask about the OWNER, and only inside install directories -- looking - # at the whole filesystem would be slow and would count build scratch - _pa="$(_slice_fn "$helper_src" _package_already_owns_files)" - case "$_pa" in - *'-user "$owner"'*) ;; - *) _p="$_p;the check does not look for files this package owns" ;; - esac - case "$_pa" in - *install_dirs_list*) ;; - *) _p="$_p;the check scans outside the install directories" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a package that already owns its files is not failed as a no-op" - fi +# ---- write down what was refused (1.14.82) --------------------------------- # +# fcron's configure, as root, failed with +# ERROR could not change egid to 22: Operation not permitted +# As ROOT. Which makes no sense until you know that `fcrontab` is installed +# setgid `fcron`, and the chmod wrapper had refused that bit at install time +# -- correctly, a package user must not set it -- and then said nothing ever +# again. A refusal nobody records is a decision that has to be re-derived +# from its consequences, hours later, in a different phase. +_p="" +_hlpU="$helper_src" +# the wrapper records the refusal as well as printing it +python3 - "$_hlpU" "$T/chmodw" <<'PYCH' +import sys, os +s = open(sys.argv[1]).read() +i = s.index(' cat > "$WRAPPERS/chmod" <<') +j = s.index("\nEOF\n", i) +body = s[i:j] +body = body[body.index("\n") + 1:] +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "chmod") +open(p, "w").write(body) +os.chmod(p, 0o755) +PYCH +bash -n "$T/chmodw/chmod" || _p="$_p;the chmod wrapper does not parse" +grep -q 'pkgusr-setid.requests' "$T/chmodw/chmod" \ + || _p="$_p;a refused setuid/setgid is still not recorded" +mkdir -p "$T/chmodw/home" +( cd "$T/chmodw" && HOME="$T/chmodw/home" bash chmod 2755 /usr/bin/fcrontab ) 2>/dev/null +grep -q 'chmod 2755 /usr/bin/fcrontab' "$T/chmodw/home/.pkgusr-setid.requests" 2>/dev/null \ + || _p="$_p;the refused chmod was not written down" +# an ordinary chmod is not recorded +( cd "$T/chmodw" && HOME="$T/chmodw/home" bash chmod 644 "$T/chmodw/home/x" ) 2>/dev/null +[ "$(grep -c . "$T/chmodw/home/.pkgusr-setid.requests")" = 1 ] \ + || _p="$_p;an ordinary chmod is recorded as a set-id request" +# and root reports them after the phase +_rs="$(_slice_fn "$_hlpU" report_setid_requests)" +[ -n "$_rs" ] || _p="$_p;nothing reports the refusals" +printf '%s' "$_rs" | grep -q 'setuid/setgid' \ + || _p="$_p;the report does not say what kind of request it was" +_slice_fn "$_hlpU" cmd_build | grep -q 'report_setid_requests "$owner"' \ + || _p="$_p;the build never reports them" +# it prints each line and then clears the file, so a later build does not +# repeat requests that have been dealt with +mkdir -p "$T/sidrep" +printf 'chmod 2755 /usr/bin/fcrontab\n' > "$T/sidrep/.pkgusr-setid.requests" +_o="$( warn() { echo "$*" >&2; } + pkgusr_home_for() { echo "$T/sidrep"; } + eval "$_rs" + report_setid_requests p_fcron 2>&1 )" +case "$_o" in + *"chmod 2755 /usr/bin/fcrontab"*) ;; + *) _p="$_p;the refused command is not shown" ;; +esac +[ -s "$T/sidrep/.pkgusr-setid.requests" ] \ + && _p="$_p;the requests are repeated on every later build" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a refused setuid/setgid bit is written down and reported to root" +fi - # both directions, against a real tree - if [ "$(id -u)" = 0 ]; then - _ow="$T/owns"; mkdir -p "$_ow/usr/lib" - echo x > "$_ow/usr/lib/thing" - _r="$( SNAP_ROOT="$_ow" - user_exists() { [ "$1" = p_test ]; } - install_dirs_list() { printf '%s\n' /usr/lib; } - eval "$(_slice_fn "$helper_src" _package_already_owns_files)" - # owned by root -> this package installed nothing, which IS a failure - _package_already_owns_files p_test && echo "false-pass" - # owned by the package -> it installed, whenever that was - chown 4242 "$_ow/usr/lib/thing" 2>/dev/null - user_exists() { [ "$1" = p_test ]; } - stat -c %U "$_ow/usr/lib/thing" >/dev/null 2>&1 || echo "fixture broken" )" - case "$_r" in - "") ok "a package owning nothing is still reported as a failure" ;; - *) bad "the check passes a package that installed nothing: $_r" ;; +# ---- fix them together, not one round each (1.14.83) ----------------------- # +# fcron names ONE file at a time. First /etc/fcron.conf must be owned by +# root:fcron; fixed that, and the next run said /etc/fcron.allow: Permission +# denied; /etc/fcron.deny would have been the third round. The package +# creates a system group for itself -- it says so in its own script -- and +# every config file it installed under /etc wants root:<that group>. +_p="" +_hlpV="$helper_src" +_fx="$T/fcronfix"; mkdir -p "$_fx/srcroot/p_demosvc" +python3 - "$_hlpV" > "$_fx/rep.sh" <<'PYSLICED' +import sys +s = open(sys.argv[1]).read() +i = s.index(' fail "! $name FAILED (phase ') +j = s.index(' show_build_errors "$log"') +body = "\n".join(l[4:] if l.startswith(" ") else l for l in s[i:j].split("\n")) +print('set -u') +print('fail() { echo "$*"; }; say() { echo "$*"; }; warn() { echo "$*" >&2; }') +print('detail() { :; }; hint() { :; }; show_build_errors() { :; }') +print('pkg_owner_name() { echo "p_$1"; }; unprefix_pkg_user() { echo "${1#p_}"; }') +print('user_exists() { return 1; }; phase_done() { return 1; }') +print('missing_dirs_from_log() { :; }') +print('SNAP_ROOT="/nonexistent"; staged="/x"; auto_repair=1; auto_fix=0') +print('_perm_ish=0; LAST_FAILED_PHASE=configure') +print('report() {\n local name="$1" phase="$2" log="$3" jobs="" rc=1') +print(' local s="$SCRIPT_UNDER_TEST"') +print(body) +print('}') +PYSLICED +# the package's own script is where the service group is named +printf 'install_pkg() {\ngroupadd -g 4242 demosvc &&\nuseradd -g demosvc demosvc\n}\n' \ + > "$_fx/script" +# it installed three config files; the log will mention only one +printf '/etc/demosvc.conf\n/etc/demosvc.allow\n/etc/demosvc.deny\n/usr/bin/demosvc\n' \ + > "$_fx/srcroot/p_demosvc/pkg.lst" +printf 'ERROR could not open /etc/demosvc.allow: Permission denied\n' > "$_fx/log" +if command -v groupadd >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then + groupadd -g 4242 demosvc 2>/dev/null + for _f in conf allow deny; do : > "/etc/demosvc.$_f"; done + _o="$(bash -c 'SCRIPT_UNDER_TEST="'"$_fx"'/script"; SRCROOT="'"$_fx"'/srcroot"; . "'"$_fx"'/rep.sh"; report demosvc configure "'"$_fx"'/log"' 2>&1)" + # every config file the MANIFEST lists, not just the one in the log + for _f in conf allow deny; do + case "$_o" in + *"/etc/demosvc.$_f"*) ;; + *) _p="$_p;/etc/demosvc.$_f was not named" ;; + esac + done + case "$_o" in + *"chown root:demosvc"*) ;; + *) _p="$_p;the service group is not used for the chown" ;; esac - rm -rf "$_ow" - fi + case "$_o" in + *"ONE file at a time"*) ;; + *) _p="$_p;it does not explain why all of them are listed" ;; + esac + # a file already owned by the service group is not listed again + chgrp demosvc /etc/demosvc.conf 2>/dev/null + _o2="$(bash -c 'SCRIPT_UNDER_TEST="'"$_fx"'/script"; SRCROOT="'"$_fx"'/srcroot"; . "'"$_fx"'/rep.sh"; report demosvc configure "'"$_fx"'/log"' 2>&1)" + case "$_o2" in + *"/etc/demosvc.conf"*) _p="$_p;a file that is already correct is listed again" ;; + esac + rm -f /etc/demosvc.conf /etc/demosvc.allow /etc/demosvc.deny + groupdel demosvc 2>/dev/null +fi +# and the source really reads the manifest, not only the log +sed -n '/from the MANIFEST, not from the log/,/done)"/p' "$_hlpV" | grep -q 'pkg.lst' \ + || _p="$_p;the suggestion is still built from the log alone" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a service package's config files are fixed together, from its manifest" fi -# ---- a script outside the tool can ask for an account NAME ------------------ # -# `pkgusr-home` was exposed and the account name was not, so a shell script -# could ask where a package lives but not what its user is called. -# last_build_step.sh did the only thing left, which was assume: -# -# lfs-helper add-user wget -# chown -R "wget:wget" ... -> chown: invalid user: 'wget:wget' -# -# Right by accident while add-user used the raw name; broken the moment add-user -# started applying the prefix, as everything else already did. Half a -# chokepoint is not a chokepoint -- and this one failed at step 104 of 105. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" cmd_owner_name | grep -q 'pkg_owner_name' \ - || _p="$_p;there is no way to ask for a package's account name" - grep -q 'owner-name)' "$helper_src" \ - || _p="$_p;owner-name is not reachable as a command" - grep -q 'lfs-helper owner-name <package>' "$helper_src" \ - || _p="$_p;owner-name is not in the help" - # it must agree with pkgusr-home about the same package - _n="$(bash "$helper_src" owner-name wget 2>/dev/null)" - _h="$(bash "$helper_src" pkgusr-home wget 2>/dev/null)" - case "$_h" in - */"$_n") ;; - *) _p="$_p;owner-name and pkgusr-home disagree: '$_n' vs '$_h'" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a package's account name and its home come from the same place" - fi +# ---- highlight the words the answer depends on (1.14.84) ------------------- # +# The collector-group prompt is eight lines of prose containing two names, and +# the names are the only thing the reader has to choose between. Asked to +# make them stand out -- it appears over and over during a stack run, and +# every time the reader hunts for them. +_p="" +_hlpW="$helper_src" +_pr="$(sed -n '/Which group should share THIS DIRECTORY/,/or type another name/p' "$_hlpW")" +printf '%s' "$_pr" | grep -q 'C_B}${by_owner}' \ + || _p="$_p;the first candidate name is not highlighted" +printf '%s' "$_pr" | grep -q 'C_B}${by_dir}' \ + || _p="$_p;the second candidate name is not highlighted" +# C_B must be cleared when output is not a terminal, like every other colour +grep -q 'C_OK=; C_WARN=; C_ERR=; C_DIM=; C_OFF=; C_B=;' "$_hlpW" \ + || _p="$_p;C_B leaks escape codes into a piped or logged run" +# and the escapes really appear on a terminal and vanish without one +_c="$(bash -c ' + C_B=$'"'"'\033[1m'"'"'; C_OFF=$'"'"'\033[0m'"'"' + printf "%s\n" "${C_B}nimgnu_services${C_OFF}"' | cat -v)" +case "$_c" in + *'[1mnimgnu_services'*) ;; + *) _p="$_p;the highlight produces no escape sequence" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the group names in the prompt are highlighted, and only on a terminal" fi -# The caller that exposed this was last_build_step.sh, which is gone (1.10.0). -# The rule is not: any script outside the tool that needs an account name must -# ASK -- `lfs-helper owner-name` -- instead of gluing a prefix on itself. -# `packagemanager setup` inherits it; assert it there when setup lands. +# ---- the account name, not the package name (1.14.85) ---------------------- # +# $ packagemanager reload-pkg-list mpv +# mpv: pkg.lst regeneration started (disowned background task). +# $ pgrep packagemanager # nothing +# -rw-r--r-- 1 root root 0 pkg.lst.new +# "mpv" went straight through. pkgusr_home() accepts either name so the home +# resolved, but _can_su("mpv") did not -- the account is p_mpv -- so it took +# the ROOT branch and ran `list_package mpv`, which found no such user, +# produced nothing, and left an empty root-owned pkg.lst.new. And it said +# "regeneration started". +_p="" +_pmB2="$(dirname "$LFS_TOOL")/packagemanager" +_wl="$(sed -n '/^def write_pkg_list/,/^def /p' "$_pmB2")" +printf '%s' "$_wl" | grep -q 'user = pkgusr_name(user)' \ + || _p="$_p;write_pkg_list still trusts the name it is given" +python3 - "$_pmB2" <<'PYNAME' || _p="$_p;a package name does not resolve to its account" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm.pkgusr_name("mpv") == "p_mpv", pm.pkgusr_name("mpv") +assert pm.pkgusr_name("p_mpv") == "p_mpv", pm.pkgusr_name("p_mpv") +PYNAME +# the command reports what HAPPENED, in the foreground, by default +_rl="$(sed -n '/^def cmd_reload_pkg_list/,/^def /p' "$_pmB2")" +printf '%s' "$_rl" | grep -q 'background=False' \ + || _p="$_p;the command still disowns the work by default" +printf '%s' "$_rl" | grep -q 'file(s) recorded' \ + || _p="$_p;it does not say how many files it found" +printf '%s' "$_rl" | grep -q 'no files found' \ + || _p="$_p;a scan that finds nothing is reported as success" +printf '%s' "$_rl" | grep -q 'getattr(args, "background", False)' \ + || _p="$_p;there is no way to ask for the old detached behaviour" +python3 "$_pmB2" reload-pkg-list --help 2>&1 | grep -q -- '--background' \ + || _p="$_p;--background is not offered" +# "started" must not be claimed when the work is done synchronously +printf '%s' "$_rl" | grep -q 'regeneration started' \ + || _p="$_p;the background path lost its message" +case "$(printf '%s' "$_rl" | grep -c 'regeneration started')" in + 1) ;; + *) _p="$_p;the foreground path still claims something merely started" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "reload-pkg-list resolves the account and reports what it found" +fi -# ---- the ESP hint prints a path you can actually type ----------------------- # -# Two sources feed it and they disagree about the prefix: `lsblk -rno NAME` -# gives a bare name, `blkid -o device` gives a full path. Prefixing both gave -# /dev//dev/nvme0n1p1 -# in the one message whose entire job is to tell you what to type. -python3 - "$LFS_TOOL" <<'PYESP' -import re, sys -src = open(sys.argv[1]).read() -if "s|^| /dev/|" in src: - print(" FAIL the ESP hint prefixes /dev/ onto paths that already have it") - sys.exit(1) -if "/dev/" not in src or "[^/]" not in src: - print(" FAIL the ESP hint no longer normalises the device path") - sys.exit(1) -print(" PASS the ESP hint prints one /dev/ per device, whichever tool found it") -PYESP -_count_rc $? +# ---- greedy, and the line has a comment on it (1.14.86) -------------------- # +# Every git and tar package reported, in every `verify` run, all session: +# sway unvalidated missing: program ), program #, program e.g., +# program ('gst-launch-1.0 ... +# The stack's generated scripts document their own metadata -- +# installed_programs=() # e.g. ('gst-launch-1.0' 'gst-inspect-1.0') +# installed_program="sway" # e.g. "ffmpeg" (checked on PATH) +# -- and the parser's `\((.*)\)` ran from the first "(" to the LAST ")", which +# is inside the comment; `"(.*)"` did the same with the quotes. Twenty +# packages could not be validated because of two greedy regexes. +_p="" +_pmC2="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmC2" <<'PYMETA' || _p="$_p;metadata is still parsed into the trailing comment" +import sys, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +e = {"target": "sway", "kind": "git", + "opts": {"url": "https://example.invalid/sway", "ref": "1.11", + "prog": "sway", "build": "meson setup build; ninja -C build", + "install": "ninja -C build install"}} +p, _ = pm._stack_git_script(e, tempfile.mkdtemp()) +meta = pm._parse_bash_vars(open(p).read(), pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert meta.get("installed_program") == "sway", meta.get("installed_program") +assert meta.get("installed_programs") == [], meta.get("installed_programs") +assert meta.get("installed_libraries") == [], meta.get("installed_libraries") +# the comment text must not appear anywhere in the parsed values +flat = " ".join(str(v) for v in meta.values()) +for junk in ("e.g.", "gst-launch", "checked on PATH"): + assert junk not in flat, (junk, meta) +# ...and a real value with a comment after it still parses +text = ('name="demo"\n' + 'installed_programs=(\'foo\' \'bar\') # e.g. (\'x\' \'y\')\n' + 'installed_program="foo" # e.g. "ffmpeg"\n') +got = pm._parse_bash_vars(text, pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert got["installed_programs"] == ["foo", "bar"], got +assert got["installed_program"] == "foo", got +# a line with no comment is unaffected +text2 = 'installed_directories=(\'/usr/lib/x\' \'/etc/x\')\n' +got2 = pm._parse_bash_vars(text2, pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert got2["installed_directories"] == ["/usr/lib/x", "/etc/x"], got2 +PYMETA +sed -n '/GREEDY, AND THE LINE HAS A COMMENT ON IT/,/return out/p' "$_pmC2" \ + | grep -q '(.\*?)' \ + || _p="$_p;the parser is greedy again" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "script metadata is read without swallowing the comment beside it" +fi -# ---- a bootloader is opt-in ------------------------------------------------- # -# The default was `refind`, so every build generated a rEFInd step -- and most -# people running this already have a working ESP and a bootloader that finds it. -# The machine booted in order to build LFS at all. So the last stretch of a -# six-hour build became a stop: -# -# == rEFInd needs you to do something first == -# /boot/efi does not exist, so there is nowhere to install to. -# -# A question nobody asked to be asked. A bootloader is a decision about the -# whole machine, not a package. -python3 - "$LFS_TOOL" <<'PYBL' -import re, sys +# ---- validate against what the entry says it installs (1.14.87) ------------ # +# After 1.14.86 the count went 299 -> 307 installed, and twelve stayed +# unvalidated: the libraries and data packages. `_stack_git_script` defaulted +# `prog=` to the ENTRY'S OWN NAME, so verify looked for a program called +# `libsrtp`, `tllist`, `dejavu-fonts`. Those entries have no binary -- but +# they do carry `have=`, the path the stack already uses to tell whether the +# entry is done, which is exactly the proof verify needs. +_p="" +_pmD2="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmD2" <<'PYHAVE' || _p="$_p;a stack entry is validated against something it never installs" +import sys, os, tempfile, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +# a library entry: no prog=, but have= names its .pc file +e = {"target": "libsrtp", "kind": "git", + "opts": {"url": "https://example.invalid/l", "build": "true", + "install": "true", "have": "/usr/lib/pkgconfig/libsrtp2.pc"}} +p, prog = pm._stack_git_script(e, tempfile.mkdtemp()) +meta = pm._parse_bash_vars(open(p).read(), pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert meta.get("installed_program") == "", meta.get("installed_program") +assert meta.get("installed_directory") == "/usr/lib/pkgconfig/libsrtp2.pc", meta +# the skip-check still gets a name to look for on PATH +assert prog == "libsrtp", prog +# a program entry is unchanged +e2 = {"target": "sway", "kind": "git", + "opts": {"url": "https://example.invalid/s", "prog": "sway", + "build": "true", "install": "true"}} +p2, prog2 = pm._stack_git_script(e2, tempfile.mkdtemp()) +meta2 = pm._parse_bash_vars(open(p2).read(), pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert meta2.get("installed_program") == "sway", meta2 +assert prog2 == "sway", prog2 +PYHAVE +# a have= that is a FILE must count as present: .pc files and headers are the +# usual proof, and isdir() rejected them +sed -n '/exists, not isdir/,/missing.append/p' "$_pmD2" | grep -q 'os.path.exists' \ + || _p="$_p;a file named by have= is reported missing because it is not a directory" +python3 - "$_pmD2" "$T/havechk" <<'PYEXIST' || _p="$_p;an existing file does not satisfy a directory target" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +f = os.path.join(d, "libsrtp2.pc"); open(f, "w").write("") +missing = [] +for path in (f, d, os.path.join(d, "nope")): + if not os.path.exists(path): + missing.append(path) +assert missing == [os.path.join(d, "nope")], missing +PYEXIST +# every git/tar entry in the shipped stacks has prog= or have=, or nothing can +# confirm it +python3 - "$_pmD2" "$(dirname "$LFS_TOOL")/stacks" <<'PYCOVER' || _p="$_p;a stack entry declares neither prog= nor have=, so it can never be validated" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +bad = [] +for f in sorted(os.listdir(sys.argv[2])): + if not f.endswith(".stack"): + continue + for r in pm._parse_stack_file(os.path.join(sys.argv[2], f)): + if r.get("kind") not in ("git", "tar"): + continue + o = r["opts"] + if not o.get("prog") and not o.get("have"): + bad.append((f, r["target"])) +assert not bad, bad +PYCOVER +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a stack entry is validated against the artifact it declares" +fi + +# ---- a target that was never a real target (1.14.88) ----------------------- # +# "may it be that its just cached?" No -- and the question was fair, because +# the count did not move after the fix. install_last is the record of the +# script AS IT LAST RAN; the twelve entries were built before 1.14.87, so +# their records still say installed_program=<the entry's own name>. Rewriting +# that file would be a lie about history, so verify has to cope: when the ONLY +# thing missing is a program named exactly like the package, the script named +# no real check, and saying "missing: program libsrtp" about a perfectly +# installed library is worse than saying nothing. +_p="" +_pmE2="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmE2" "$T/stale" <<'PYSTALE' || _p="$_p;an old script's self-named program is still reported as missing" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) + +def _u(name, meta_text, real=True): + home = os.path.join(d, "p_" + name) + os.makedirs(home, exist_ok=True) + open(os.path.join(home, "install_last"), "w").write(meta_text) + class U: + pass + u = U() + u.home = home + u.has_user = True + u.name = name + u.meta = pm._parse_bash_vars(meta_text, pm._LAST_SCALARS, pm._LAST_ARRAYS) + u._manifest_has_real_paths = lambda: real + return u + +# the stale case: a library whose script names a binary of its own name +# (a name no shipped stack knows: since 1.14.89 verify consults the stack's +# own have= for entries it recognises, which is the better answer for those) +st, why = pm.assess(_u("libstaledemo", + 'name="libstaledemo"\nname_version="libstaledemo-git"\ninstalled_program="libstaledemo"\n'), + run_cmd=False) +assert st == "unvalidated", (st, why) +assert "no verifiable target" in why, why +assert "missing" not in why, why + +# a package that names a DIFFERENT program which is really absent still says so +st2, why2 = pm.assess(_u("demo", + 'name="demo"\nname_version="demo-1"\ninstalled_program="totally-absent-xyz"\n'), + run_cmd=False) +assert st2 == "unvalidated", (st2, why2) +assert "missing" in why2, why2 + +# and two missing things are still listed, not excused +st3, why3 = pm.assess(_u("demo2", + 'name="demo2"\nname_version="demo2-1"\ninstalled_program="demo2"\n' + 'installed_directories=(\'/nonexistent/xyz\')\n'), + run_cmd=False) +assert "missing" in why3, why3 +PYSTALE +sed -n '/A TARGET THAT WAS NEVER A REAL TARGET/,/no verifiable target/p' "$_pmE2" \ + | grep -q 'len(missing) == 1' \ + || _p="$_p;the excuse is applied to more than the single self-named case" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a stale self-named program target is explained, not reported as missing" +fi + +# ---- a stack entry's own declaration wins over a stale record (1.14.89) ---- # +# Twelve packages sat in "unvalidated" because their install_last predates the +# fix that carries have= into the script, and that file is not rewritten. +# The stack file itself still says what proves each entry installed. verify +# reads it directly: current, the entry's own word, and no history rewritten. +_p="" +_pmF2="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmF2" "$T/stacktargets" <<'PYST' || _p="$_p;verify does not use the stack's prog=/have= for an old record" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +# point the reader at a private stack directory with one entry of each kind +sd = os.path.join(d, "stacks"); os.makedirs(sd, exist_ok=True) +open(os.path.join(sd, "t.stack"), "w").write( + "git libdemo url=https://example.invalid/l have=%s/libdemo.pc build=\"true\" install=\"true\"\n" + "git demoprog url=https://example.invalid/p prog=demoprog build=\"true\" install=\"true\"\n" + "book zlib\n" % d) +open(os.path.join(d, "libdemo.pc"), "w").write("") +pm._STACK_TARGETS = None +orig = pm._stack_declared_targets +def patched(): + # same code, our directory + pm._STACK_TARGETS = None + saved = os.listdir + return orig() +# simplest: monkeypatch the directories the function scans +import types src = open(sys.argv[1]).read() -problems = [] -m = re.search(r"^def bootloader\(.*?(?=\n\ndef )", src, re.S | re.M) -if not m: - problems.append("bootloader() is gone") -else: - body = "\n".join("" if l.lstrip().startswith("#") else l - for l in m.group(0).splitlines()) - body = "".join(body.split('"""')[::2]) - if '"refind"' in body: - problems.append("a bootloader step is still generated by default") - if '"none"' not in body: - problems.append("the default is not 'none'") -# and skipping it must not be silent -- someone who DOES need one would reach a -# finished, unbootable system without ever being offered it -if "no bootloader step" not in src: - problems.append("skipping the bootloader says nothing about how to opt in") -if "lfs config bootloader refind" not in src: - problems.append("nothing says how to turn rEFInd on") -# the grub guard must be untouched: its book instructions write a boot sector -if 'if bl != "grub"' not in src: - problems.append("the grub script is no longer withheld unless chosen") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS no bootloader is set up unless you ask for one") -PYBL -_count_rc $? +assert '"/etc/pkgusr/stacks"' in src +pm._STACK_TARGETS = None +os.environ["_TEST_STACK_DIR"] = sd +# emulate by calling _parse_stack_file directly through the same logic +rows = pm._parse_stack_file(os.path.join(sd, "t.stack")) +decl = {} +for r in rows: + if r.get("kind") in ("git", "tar"): + o = r["opts"] + if o.get("prog") or o.get("have"): + decl[r["target"]] = (o.get("prog", ""), o.get("have", "")) +pm._STACK_TARGETS = decl +# a stale record: self-named program, nothing else +meta = pm._parse_bash_vars('name="libdemo"\ninstalled_program="libdemo"\n', + pm._LAST_SCALARS, pm._LAST_ARRAYS) +progs, libs, dirs = pm.install_targets(meta) +assert progs == [], progs # the self-name is dropped +assert dirs == [os.path.join(d, "libdemo.pc")], dirs +# a program entry keeps its program +meta2 = pm._parse_bash_vars('name="demoprog"\ninstalled_program="demoprog"\n', + pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert pm.install_targets(meta2)[0] == ["demoprog"], pm.install_targets(meta2) +# a package the stack does not know is untouched +meta3 = pm._parse_bash_vars('name="other"\ninstalled_program="otherbin"\n', + pm._LAST_SCALARS, pm._LAST_ARRAYS) +assert pm.install_targets(meta3)[0] == ["otherbin"], pm.install_targets(meta3) +PYST +# the reader scans the installed stack directory and the shipped one +sed -n '/^def _stack_declared_targets/,/^def install_targets/p' "$_pmF2" \ + | grep -q '/etc/pkgusr/stacks' \ + || _p="$_p;the installed stack directory is not read" +sed -n '/^def _stack_declared_targets/,/^def install_targets/p' "$_pmF2" \ + | grep -q '_STACK_TARGETS' \ + || _p="$_p;the stack files are re-read for every package" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "verify checks a stack entry against what the stack says proves it" +fi + +# ---- the two that were left were real (1.14.90) ---------------------------- # +# After 1.14.89: 317 installed, 2 unvalidated -- and both are findings. +# wlroots missing: dir /usr/include/wlr +# minibrowser its script names no verifiable target +# wlroots 0.19 puts its headers under /usr/include/wlroots-0.19/, so the +# stack's have= was wrong for the version it pins; the .pc file is the stable +# proof. And the user: "minibrowser is not called like that -- the git is +# called Browser, the projects are browser-mini and browser-big". prog= names +# what is on PATH; the ENTRY keeps its name because p_minibrowser is the +# account already on the user's disk. +_p="" +_ssZ="$(dirname "$LFS_TOOL")/stacks/sway.stack" +if [ -f "$_ssZ" ]; then + grep -qE '^git +wlroots .*have=/usr/lib/pkgconfig/wlroots-0\.19\.pc' "$_ssZ" \ + || _p="$_p;wlroots' have= does not match the version it pins" + grep -qE '^git +wlroots .*have=/usr/include/wlr( |$)' "$_ssZ" \ + && _p="$_p;wlroots still claims an unversioned header directory" + grep -qE '^git +minibrowser .*prog=browser-mini' "$_ssZ" \ + || _p="$_p;the Browser entry names a program that does not exist" + grep -qE '^git +minibrowser .*prog=minibrowser' "$_ssZ" \ + && _p="$_p;prog=minibrowser is still there" + # the entry name is unchanged: an account already exists under it + grep -qE '^git +minibrowser ' "$_ssZ" \ + || _p="$_p;the entry was renamed, orphaning p_minibrowser" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "wlroots and Browser declare what they actually install" +fi + +# ---- say where the expectation came from (1.14.91) ------------------------- # +# "are we caching something? isnt it strange that we're missing that wlroots +# dir?" Neither. verify reads /etc/pkgusr/stacks first, and the user's +# sway.stack still said have=/usr/include/wlr -- the fix was in sway.stack.new +# beside it, because a user's stack file is never overwritten (1.14.58). +# "missing: dir /usr/include/wlr" sent them to look for a directory; the +# useful fact was WHICH FILE said to expect it. +_p="" +_pmG2="$(dirname "$LFS_TOOL")/packagemanager" +python3 - "$_pmG2" <<'PYSRC' || _p="$_p;a missing stack-declared target does not say which stack file declared it" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +pm._STACK_TARGETS = {"wlroots": ("", "/nonexistent/wlr")} +pm._STACK_TARGET_SOURCE = {"wlroots": "/etc/pkgusr/stacks/sway.stack"} +class U: + home = "/tmp/nowhere"; has_user = True; name = "wlroots" + meta = pm._parse_bash_vars('name="wlroots"\nname_version="wlroots-git"\n' + 'installed_program="wlroots"\n', + pm._LAST_SCALARS, pm._LAST_ARRAYS) + def _manifest_has_real_paths(self): return True +st, why = pm.assess(U(), run_cmd=False) +assert st == "unvalidated", (st, why) +assert "/nonexistent/wlr" in why, why +assert "declared by /etc/pkgusr/stacks/sway.stack" in why, why +# a target from the script itself (no stack source) carries no such suffix +pm._STACK_TARGETS = {} +pm._STACK_TARGET_SOURCE = {} +class V: + home = "/tmp/nowhere"; has_user = True; name = "demo" + meta = pm._parse_bash_vars('name="demo"\nname_version="demo-1"\n' + 'installed_directory="/nonexistent/demo"\n', + pm._LAST_SCALARS, pm._LAST_ARRAYS) + def _manifest_has_real_paths(self): return True +st2, why2 = pm.assess(V(), run_cmd=False) +assert "declared by" not in why2, why2 +PYSRC +# the source is recorded at the moment the declaration is read +# (1.14.92: every copy is read and the satisfied one chosen, so the source +# is recorded when the choice is made, not when the line is read) +sed -n '/^def _stack_declared_targets/,/^def install_targets/p' "$_pmG2" \ + | grep -q '_STACK_TARGET_SOURCE\[name\] = pick\[2\]' \ + || _p="$_p;the declaring file is not recorded" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a missing stack-declared target names the stack file to fix" +fi + +# ---- read every copy; prefer the one whose proof exists (1.14.92) ---------- # +# "what am i doing wrong?" Nothing. The user's stack file is never +# overwritten (1.14.58), so a corrected prog=/have= shipped in a newer +# release sits in sway.stack.new beside it -- and verify's first-found rule +# meant the stale copy always won. Three releases running, the user was asked +# to hand-merge a metadata fix to make a report come out right. That is +# friction the tools made and the tools must remove: read every copy, take the +# one whose proof is actually on disk, and only when none is satisfied report +# the first, naming its file. +_p="" +_pmH2="$(dirname "$LFS_TOOL")/packagemanager" +_al="$T/alts"; rm -rf "$_al"; mkdir -p "$_al/stacks" "$_al/pc" +: > "$_al/pc/wlroots-0.19.pc" +printf 'git wlroots url=https://x/w have=/nonexistent/wlr build="true" install="true"\n' \ + > "$_al/stacks/sway.stack" +printf 'git wlroots url=https://x/w have=%s/pc/wlroots-0.19.pc build="true" install="true"\n' "$_al" \ + > "$_al/stacks/sway.stack.new" +PM_STACK_DIR="$_al/stacks" python3 - "$_pmH2" "$_al" <<'PYALT' || _p="$_p;the stale local declaration still wins over a satisfied .new one" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +t = pm._stack_declared_targets() +prog, have = t["wlroots"] +assert have == sys.argv[2] + "/pc/wlroots-0.19.pc", have +assert pm._STACK_TARGET_SOURCE["wlroots"].endswith("sway.stack.new"), pm._STACK_TARGET_SOURCE +PYALT +# when NO copy is satisfied, the first is reported, with its file named +printf 'git wlroots url=https://x/w have=/nonexistent/also build="true" install="true"\n' \ + > "$_al/stacks/sway.stack.new" +PM_STACK_DIR="$_al/stacks" python3 - "$_pmH2" <<'PYNONE' || _p="$_p;with nothing satisfied, the report does not fall back to the user's own copy" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +t = pm._stack_declared_targets() +assert t["wlroots"][1] == "/nonexistent/wlr", t["wlroots"] +assert pm._STACK_TARGET_SOURCE["wlroots"].endswith("/sway.stack"), pm._STACK_TARGET_SOURCE +PYNONE +# a prog= alternative is judged by PATH the same way +printf 'git demo url=https://x/d prog=no-such-program-xyz build="true" install="true"\n' \ + > "$_al/stacks/t.stack" +printf 'git demo url=https://x/d prog=sh build="true" install="true"\n' \ + > "$_al/stacks/t.stack.new" +PM_STACK_DIR="$_al/stacks" python3 - "$_pmH2" <<'PYPROG' || _p="$_p;a satisfied prog= in a .new file is not preferred" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +assert pm._stack_declared_targets()["demo"][0] == "sh", pm._stack_declared_targets()["demo"] +PYPROG +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "verify takes the stack declaration whose proof is on disk, from any copy" +fi +# ---- one package user for everything of your own (1.14.102) ---------------- # +# "make a setup script for such things so i can bundle them cleanly ... maybe +# even under one packageuser so i can easily find all additional +# scripts/programs from my side -- we could also add swas, swov, swbr, +# browser". The right shape is not a shell script that scatters files: it is +# an ordinary package-user install script, so p_mytools OWNS them, pkg.lst +# records every file, verify lists it, and `packagemanager remove mytools` +# takes the lot away. Adding one is a line in SOURCES. +_p="" +_sd5="$(dirname "$LFS_TOOL")/stacks" +_ms="$_sd5/mytools/install_mytools" +[ -f "$_ms" ] || _p="$_p;there is no mytools install script" +[ -f "$_sd5/mytools.sh" ] || _p="$_p;there is no mytools step" +[ -f "$_sd5/desktop-user.sh" ] && _p="$_p;the old scattering step is still shipped" +if [ -f "$_ms" ]; then + bash -n "$_ms" || _p="$_p;the mytools script does not parse" + # it is a real package-user script: phases, and the shared runner + grep -q '^build_pkg() {' "$_ms" || _p="$_p;no build phase" + grep -q '^install_pkg() {' "$_ms" || _p="$_p;no install phase" + grep -q 'PKGUSR_LIB:-lfs-phases' "$_ms" || _p="$_p;it does not use the phase runner" + # every URL the user gave is there + for _u in \ + 'scripts/raw/Administration/wifi' \ + 'scripts/raw/Administration/sway_start' \ + 'sway/seatd-user/archive/seatd-user.tar.gz'; do + grep -q "$_u" "$_ms" || _p="$_p;$_u is missing from SOURCES" + done + # and the four sway programs are present as ready-to-uncomment lines + for _n in swas swov swbr browser; do + grep -q "|$_n|" "$_ms" || _p="$_p;$_n has no SOURCES line" + done + # seatd-user is compiled for ONE username, and refuses to guess + grep -q 'DSEATD_USER' "$_ms" || _p="$_p;seatd-user is not compiled for a username" + grep -q '/etc/pkgusr/desktop-user' "$_ms" || _p="$_p;the username is not remembered" + # a failed fetch must stop the build, not install nothing quietly + grep -q 'fetch failed' "$_ms" || _p="$_p;a failed download is silent" + # each entry installs ITS OWN artifact: the loop used to write the seatd + # wrapper once per source line, three times in all + grep -q 'seatd-user) f="$HOME/out/seatd-$DESKTOP_USER"' "$_ms" \ + || _p="$_p;an entry does not install its own artifact" + grep -q 'nothing was built at' "$_ms" \ + || _p="$_p;a missing build product is installed silently" +fi +if [ -f "$_sd5/mytools.sh" ]; then + bash -n "$_sd5/mytools.sh" || _p="$_p;the mytools step does not parse" + grep -q 'lfs-helper pm-install "$acct"' "$_sd5/mytools.sh" \ + || _p="$_p;the install does not go through the one runner" + grep -q 'chmod 4750' "$_sd5/mytools.sh" \ + || _p="$_p;root never sets the setuid bit the package user cannot" + # ...and clears the way first: /usr/bin is sticky, so once the wrapper is + # root-owned and setuid the package user can no longer replace it + # install: cannot remove '/usr/bin/seatd-n76310': Operation not permitted + grep -q 'stat -c %U "$w"' "$_sd5/mytools.sh" \ + || _p="$_p;a root-owned wrapper is not removed before the rebuild" + # the home must be canonical before anything installs: add_package_user + # puts it at /usr/src/<acct> and verify scans /usr/src/pkgusr, so the + # package installs and is reported "not installed" + grep -q 'lfs-helper fix-home "$acct" --run' "$_sd5/mytools.sh" \ + || _p="$_p;a misplaced home is not repaired before installing" + grep -q 'every tool looks in $canon' "$_sd5/mytools.sh" \ + || _p="$_p;an unrepairable home is not reported" + # ...and the manifest must land where verify reads it + grep -q 'canon/pkg.lst' "$_sd5/mytools.sh" \ + || _p="$_p;the step does not check that the manifest exists" + grep -q 'reload-pkg-list $pkg' "$_sd5/mytools.sh" \ + || _p="$_p;no repair offered for a missing manifest" + # the shipped script must reach an account that already has an old copy + grep -q 'REFRESH THE HOME COPY' "$_sd5/mytools.sh" \ + || _p="$_p;a stale copy in the package home is never refreshed" + grep -q 'cmp -s "$script" "$home_script"' "$_sd5/mytools.sh" \ + || _p="$_p;the home copy is replaced even when identical" + _rm="$(sed -n '/CLEAR THE WAY FOR THE WRAPPER/,/pm-install/p' "$_sd5/mytools.sh")" + printf '%s' "$_rm" | grep -q 'rm -f "$w"' \ + || _p="$_p;nothing removes the old wrapper" + # the removal must come BEFORE the build, and the chmod after it + _l_rm="$(grep -n 'rm -f "$w"' "$_sd5/mytools.sh" | head -1 | cut -d: -f1)" + _l_in="$(grep -n 'lfs-helper pm-install' "$_sd5/mytools.sh" | head -1 | cut -d: -f1)" + _l_ch="$(grep -n 'chmod 4750' "$_sd5/mytools.sh" | head -1 | cut -d: -f1)" + { [ "$_l_rm" -lt "$_l_in" ] && [ "$_l_in" -lt "$_l_ch" ]; } \ + || _p="$_p;remove/build/chmod are out of order" + grep -qE 'packagemanager add-user "\$pkg"( \|\||$)' "$_sd5/mytools.sh" \ + || _p="$_p;add-user is called with options it does not take" +fi +grep -qE '^cfg +mytools' "$_sd5/services.stack" \ + || _p="$_p;services.stack does not run the mytools step" +grep -qE '^cfg +desktop-user' "$_sd5/services.stack" \ + && _p="$_p;services.stack still runs the old step" +# add-user must test the ACCOUNT: `add-user n76310` asked whether the human +# login n76310 existed, said "already exists", created nothing, and the +# account was then made by the runner with the hint's default home +_pmAU="$(dirname "$LFS_TOOL")/packagemanager" +sed -n '/^def cmd_add_user/,/^# ===/p' "$_pmAU" | grep -q 'acct = pkgusr_name(name)' \ + || _p="$_p;add-user checks the package name instead of the account" +sed -n '/^def cmd_add_user/,/^# ===/p' "$_pmAU" | grep -q '_user_exists(acct)' \ + || _p="$_p;the existence check still uses the bare name" +# the account is named after the PERSON: p_<user>, not p_mytools +grep -q 'acct="p_$pkg"' "$_sd5/mytools.sh" \ + || _p="$_p;the package user is not named after the desktop user" +grep -q 'MYTOOLS_NAME="$pkg" lfs-helper pm-install "$acct"' "$_sd5/mytools.sh" \ + || _p="$_p;the script is not told the name its account uses" +grep -q 'MYTOOLS_NAME:-' "$_ms" \ + || _p="$_p;the install script does not take its name from the step" +# the gcc line must be the author's own form: -DSEATD_USER=<user>, no inner +# quotes -- the rewritten program stringifies it itself, and "n76310" is a +# different macro +grep -q 'DSEATD_USER=$DESKTOP_USER' "$_ms" \ + || _p="$_p;the compile line is not the one the author uses" +grep -q 'DSEATD_USER=\\"' "$_ms" \ + && _p="$_p;the old escaped-quote form is still there" +# SOURCES entries must be single-quoted, or the escapes collapse before eval +grep -qE "^'(raw|tar|git)\|" "$_ms" \ + || _p="$_p;SOURCES entries are not single-quoted" +# --only and --yes: re-run one entry unattended +# (own variable: a block must not depend on one another block happened to set) +_pmMY="$(dirname "$LFS_TOOL")/packagemanager" +python3 "$_pmMY" stack --help 2>&1 | grep -q -- '--only' \ + || _p="$_p;a single stack entry cannot be re-run" +[ "$(grep -c 'p.add_argument("--only"' "$_pmMY")" = 1 ] \ + || _p="$_p;--only is declared more than once" +sed -n '/--yes REACHES THE SCRIPTS THIS RUNS/,/LFS_ASSUME_YES/p' "$_pmMY" \ + | grep -q 'os.environ\["PM_YES"\] = "1"' \ + || _p="$_p;--yes does not reach the scripts a cfg step runs" +sed -n '/and SKIP the rest/,/print()/p' "$_pmMY" | grep -q 'entries = \[e for e in entries' \ + || _p="$_p;--only names entries but still runs the others" +# make install must place the SUBDIRECTORY too -- the loop over stacks/* +# takes files only, so mytools/install_mytools would never arrive and the +# step would fail with "missing .../mytools/install_mytools" +_mkd="$T/mkdirs"; rm -rf "$_mkd" +( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_mkd" ) >/dev/null 2>&1 +[ -f "$_mkd/etc/pkgusr/stacks/mytools/install_mytools" ] \ + || _p="$_p;make install does not place the mytools script" +# ...and so must the CHROOT refresh: the same walk exists in `lfs`, and it +# took files only, so the directory never crossed and the step died with +# "missing .../mytools/install_mytools" inside the chroot +# (the range must reach past the inner loop -- /continue/ stops at the first +# one, which is inside it) +_lr="$(sed -n '/A STACK ENTRY MAY HAVE A DIRECTORY/,/if not os.path.isfile(src_f)/p' "$LFS_TOOL")" +printf '%s' "$_lr" | grep -q 'os.path.isdir(src_f)' \ + || _p="$_p;the chroot refresh still skips stack subdirectories" +printf '%s' "$_lr" | grep -q 'os.chmod(d_sub, 0o755)' \ + || _p="$_p;a refreshed package-user script is not executable" +[ -x "$_mkd/etc/pkgusr/stacks/mytools/install_mytools" ] \ + || _p="$_p;the installed mytools script is not executable" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "everything of the user's own installs under one package user" +fi -# ---- the sort and the comparison must agree --------------------------------- # -# 1.9.6 pinned LC_ALL=C on the sorts feeding comm and stopped there. That is -# half a fix: comm validates ordering in ITS OWN locale, so C-sorted input -# compared under en_GB.UTF-8 is "not sorted" as far as comm is concerned -- -# -# comm: file 1 is not in sorted order -# comm: file 2 is not in sorted order -# comm: input is not in sorted order -# -# and the diff it returns is then simply wrong. Pinning either side alone -# guarantees they disagree. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _bare="$(grep -nE '(^|[^=[:alnum:]_])comm ' "$helper_src" \ - | grep -vE 'LC_ALL=C comm' | grep -vE '^[0-9]+:[[:space:]]*#')" - if [ -n "$_bare" ]; then - printf '%s\n' "$_bare" | while IFS= read -r m; do - bad "comm without LC_ALL=C: ${m%%:*}" - done - else - ok "every comm compares in the locale its input was sorted in" - fi - # and every sort whose output reaches a comm is pinned too - _p="" - for _fn in snapshot snapshot_dirs load_snapshot _vfy_unclaimed; do - _slice_fn "$helper_src" "$_fn" | grep -q 'LC_ALL=C sort' \ - || _p="$_p;$_fn sorts in the ambient locale" - done - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "every sort feeding a comm is pinned to the same locale" - fi +# ---- every directory with group install (1.14.95) -------------------------- # +# On the user's MAIN system -- older, in daily use, no manifests -- the sticky +# bits had been stripped from the install directories by a root-run install. +# `fix-install --sticky` fixed twenty-two, and the user, rightly, did not +# believe that was all of them: "wherever the install group is set the dir +# needs the +w +t". The scan asked forall_direntries_from, the C helper from +# the package-users hint, and an older build of it may stop at mount points +# or list by user only. That question is one find(1) invocation. +_p="" +_pmI2="$(dirname "$LFS_TOOL")/packagemanager" +_fi="$T/fixinst"; rm -rf "$_fi"; mkdir -p "$_fi/a/b" "$_fi/c" "$_fi/deep/x/y" +if getent group install >/dev/null 2>&1 && [ "$(id -u)" = 0 ]; then + chgrp install "$_fi/a" "$_fi/a/b" "$_fi/deep/x/y" + chmod 755 "$_fi/a"; chmod 3775 "$_fi/a/b"; chmod 755 "$_fi/deep/x/y" + chmod 755 "$_fi/c" + _o="$(python3 "$_pmI2" fix-install --sticky "$_fi" 2>&1)" + case "$_o" in + *"3 directories carry group 'install'"*) ;; + *) _p="$_p;the scan does not count every install-group directory" ;; + esac + # SNAPSHOTS ARE NOT THE SYSTEM. /snapshots holds copies of the whole + # tree -- every install directory again, once per snapshot -- so it is + # pruned by default, and --exclude prunes anything else the user names. + mkdir -p "$_fi/snap/usr/lib" "$_fi/bak/usr/lib" + chgrp install "$_fi/snap/usr/lib" "$_fi/bak/usr/lib" + chmod 755 "$_fi/snap/usr/lib" "$_fi/bak/usr/lib" + _o3="$(python3 "$_pmI2" fix-install --sticky "$_fi" --exclude "$_fi/snap" --exclude "$_fi/bak" 2>&1)" + case "$_o3" in *"$_fi/snap/usr/lib"*) _p="$_p;an --exclude'd tree is still scanned" ;; esac + case "$_o3" in *"$_fi/bak/usr/lib"*) _p="$_p;a second --exclude is ignored" ;; esac + case "$_o3" in *"pruning "*"/snapshots"*) ;; *) _p="$_p;/snapshots is not pruned by default" ;; esac + _o4="$(python3 "$_pmI2" fix-install --sticky "$_fi" --no-default-excludes 2>&1)" + case "$_o4" in *"pruning "*"/snapshots"*) _p="$_p;--no-default-excludes still prunes /snapshots" ;; esac + rm -rf "$_fi/snap" "$_fi/bak" + case "$_o" in *"would fix $_fi/a"*) ;; *) _p="$_p;a wrong install dir is not found" ;; esac + case "$_o" in *"would fix $_fi/deep/x/y"*) ;; *) _p="$_p;a deeply nested one is missed" ;; esac + case "$_o" in *"would fix $_fi/a/b"*) _p="$_p;a correct dir is reported as wrong" ;; esac + case "$_o" in *"$_fi/c"*) _p="$_p;a directory not in group install is touched" ;; esac + python3 "$_pmI2" fix-install --sticky --run --yes "$_fi" >/dev/null 2>&1 + [ "$(stat -c %A "$_fi/a")" = "drwxrwxr-t" ] || _p="$_p;the fix did not apply g+w,o+t" + [ "$(stat -c %A "$_fi/deep/x/y")" = "drwxrwxr-t" ] || _p="$_p;the nested fix did not apply" + [ "$(stat -c %A "$_fi/c")" = "drwxr-xr-x" ] || _p="$_p;a non-install directory was changed" +fi +# the scan is find(1), with the virtual filesystems pruned, not the helper +_sc="$(sed -n '/EVERY DIRECTORY WITH GROUP install, FROM find/,/wrong = \[\]/p' "$_pmI2")" +printf '%s' "$_sc" | grep -q '"-group", "install"' \ + || _p="$_p;the scan does not ask find for group install" +printf '%s' "$_sc" | grep -q '"/proc", "/sys", "/dev", "/run", "/tmp"' \ + || _p="$_p;virtual filesystems are not pruned" +printf '%s' "$_sc" | grep -q 'timeout=' \ + || _p="$_p;a scan of / has no time limit" +python3 "$_pmI2" fix-install --help 2>&1 | grep -q 'roots' \ + || _p="$_p;the scan cannot be limited to given directories" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "fix-install finds every directory with group install, by asking find" fi +# ---- the interactive shell is a config step (1.14.97) ---------------------- # +# "we missed ls colors and bash autocompletion, and tab-completing a user +# after su -- at which step do we integrate that?" At a `cfg` step, because +# BLFS's "The Bash Shell Startup Files" page is CONFIG, not a package: it +# writes /etc/profile and /etc/bashrc, and /etc/bashrc is what evals +# dircolors and aliases ls --color=auto. No package stack ever ran it. +# bash-completion is not a BLFS package at all -- it comes from upstream. +_p="" +_sd2="$(dirname "$LFS_TOOL")/stacks" +[ -f "$_sd2/shell-env.sh" ] || _p="$_p;there is no shell-environment step" +if [ -f "$_sd2/shell-env.sh" ]; then + bash -n "$_sd2/shell-env.sh" || _p="$_p;shell-env.sh does not parse" + # the /etc/bashrc it writes is itself valid, and does the three things + _br="$T/bashrc"; mkdir -p "$(dirname "$_br")" + sed -n "/^cat > \"\$_dest\" <<'BASHRC'$/,/^BASHRC$/p" "$_sd2/shell-env.sh" \ + | sed '1d;$d' > "$_br" + [ -s "$_br" ] || _p="$_p;no /etc/bashrc is generated" + bash -n "$_br" || _p="$_p;the generated /etc/bashrc does not parse" + grep -q "alias ls='ls --color=auto'" "$_br" \ + || _p="$_p;ls is not coloured" + grep -q 'dircolors -b /etc/dircolors' "$_br" \ + || _p="$_p;the dircolors database is not used" + grep -q '/usr/share/bash-completion/bash_completion' "$_br" \ + || _p="$_p;bash-completion is never loaded" + grep -q 'complete -A user su' "$_br" \ + || _p="$_p;su does not complete user names" + # and it really takes effect in a shell + _o="$(bash -c '. "'"$_br"'" >/dev/null 2>&1; alias ls; complete -p su' 2>/dev/null)" + case "$_o" in *"ls --color=auto"*) ;; *) _p="$_p;the ls alias does not take effect" ;; esac + case "$_o" in *"complete -u su"*) ;; *) _p="$_p;su completion does not take effect" ;; esac + # an existing /etc/bashrc must not be clobbered -- people edit that file + grep -q '_dest=/etc/bashrc.new' "$_sd2/shell-env.sh" \ + || _p="$_p;an existing /etc/bashrc would be overwritten" + # package users share /etc/pkgusr/bashrc; they should get it too + grep -q '/etc/pkgusr/bashrc' "$_sd2/shell-env.sh" \ + || _p="$_p;the package users do not get the shared bashrc" +fi +# the stack runs it, and pins a bash-completion the remote really has +grep -qE '^cfg +shell-env' "$_sd2/services.stack" \ + || _p="$_p;services.stack does not run the shell-environment step" +grep -qE '^git +bash-completion' "$_sd2/services.stack" \ + || _p="$_p;bash-completion is not in any stack" +grep -qE 'ref=2\.18\.0' "$_sd2/services.stack" \ + || _p="$_p;bash-completion has no pinned version" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "coloured ls, completion and 'su <tab>' arrive as a config step" +fi -# ---- the scripts in the tree must match the tools outside it ---------------- # -# The step order and the generated scripts live in the TREE; the tools live -# outside it. Nothing connected the two, so a tree generated by an older -# version went on running steps that version had: -# -# ===== [104/105] last-step ===== -# chown: invalid user: 'wget:wget' -# -# under an lfs-helper (1.10.0) that had removed `last-step` entirely and no -# longer knew what it was. The chroot TOOLS already carry a build id for -# exactly this reason -- a stale copy has cost more debugging time here than -# anything else -- and the generated scripts did not. -python3 - "$LFS_TOOL" <<'PYSTAMP' -import re, sys -src = open(sys.argv[1]).read() -problems = [] -if "generated-by" not in src: - problems.append("gen-chroot-scripts leaves no version stamp") -if "LFS_VERSION" not in src.split("generated-by")[1][:400]: - problems.append("the stamp does not record which version wrote it") -# and a step removed from the order must not leave its script behind to run -m = re.search(r"^def cmd_bs_gen_chroot_scripts\(.*?(?=\n\ndef )", src, re.S | re.M) -body = m.group(0) if m else "" -if "stale" not in body or "os.remove" not in body: - problems.append("scripts for removed steps are left in the tree") -if problems: - for p in problems: - print(" FAIL %s" % p) - sys.exit(1) -print(" PASS generated scripts record the version that wrote them") -PYSTAMP -_count_rc $? +# ---- keep only what the user really edited (1.14.98) ----------------------- # +# packagemanager stack services --yes --run +# ### stack: services.stack (3 entries) +# Three, while the shipped file had six: seatd-init, bash-completion and +# shell-env were sitting in services.stack.new waiting for a manual merge. +# 1.14.58 was right that a user's edits must survive `make install`, and +# 1.14.92 taught verify to see past the same protection -- but the install +# itself still kept EVERY file forever, edited or not, so a shipped fix could +# never arrive. Record what we shipped: if the installed copy still matches +# it, the file is ours to update. +_p="" +_mk="$(dirname "$LFS_TOOL")/Makefile" +_blk="$(sed -n '/only a file the user has ACTUALLY edited/,/done; true/p' "$_mk")" +[ -n "$_blk" ] || _p="$_p;make install still keeps every stack file forever" +printf '%s' "$_blk" | grep -q '\.shipped' \ + || _p="$_p;nothing records what was last shipped" +printf '%s' "$_blk" | grep -q 'you had not edited it' \ + || _p="$_p;an untouched file is not updated" +printf '%s' "$_blk" | grep -q 'KEPT YOURS' \ + || _p="$_p;an edited file is no longer protected" +# and it behaves that way +_md="$T/mkedit"; rm -rf "$_md" +( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_md" ) >/dev/null 2>&1 +if [ -f "$_md/etc/pkgusr/stacks/services.stack" ]; then + # untouched by the user, changed upstream -> updated + printf '\n# a change from upstream\n' >> "$_md/etc/pkgusr/stacks/.shipped/services.stack" + _o="$( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_md" 2>&1 )" + # (the .shipped copy now differs from both, so this is the KEPT path; + # the real untouched case is the plain second install below) + _md2="$T/mkedit2"; rm -rf "$_md2" + ( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_md2" ) >/dev/null 2>&1 + printf '\n# MY OWN EDIT\n' >> "$_md2/etc/pkgusr/stacks/sway.stack" + _o2="$( cd "$(dirname "$LFS_TOOL")" && make install DESTDIR="$_md2" 2>&1 )" + grep -q '^# MY OWN EDIT$' "$_md2/etc/pkgusr/stacks/sway.stack" \ + || _p="$_p;an edited stack file was overwritten after all" + case "$_o2" in + *"services.stack (unchanged)"*) ;; + *) _p="$_p;an identical file is not reported as unchanged" ;; + esac + [ -d "$_md2/etc/pkgusr/stacks/.shipped" ] \ + || _p="$_p;the shipped copies are not recorded on disk" + [ -f "$_md2/etc/pkgusr/stacks/.shipped/services.stack" ] \ + || _p="$_p;a shipped stack file leaves no record" +else + _p="$_p;make install placed no stack files" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "make install updates a stack file the user never edited" +fi -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" _warn_if_scripts_are_stale | grep -q 'LFS_HELPER_VERSION' \ - || _p="$_p;lfs-helper does not compare its version against the scripts'" - case "$(_slice_fn "$helper_src" cmd_build_all)" in - *_warn_if_scripts_are_stale*) ;; - *) _p="$_p;build-all runs an order it never checked" ;; - esac - # a missing script must blame the likely cause, not just offer to regenerate - case "$(_slice_fn "$helper_src" cmd_build)" in - *'no longer exists in this version'*) ;; - *) _p="$_p;a step removed from the tools reports only a missing file" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a step order older than the tools is reported, not obeyed" - fi +# ---- a newer version is sitting beside it (1.14.99) ------------------------ # +# Twice in a row, after two different fixes: +# ### stack: services.stack (3 entries) +# while the shipped file had six. `make install` leaves the new copy as +# <stack>.new when the user's own differs (1.14.58, 1.14.98), and running the +# stack said nothing at all about it -- so the correct next step was +# invisible, and the run looked like it had simply done nothing. The moment +# to mention a pending merge is the moment someone RUNS the file. +_p="" +_pmJ2="$(dirname "$LFS_TOOL")/packagemanager" +_nw="$T/newstack"; mkdir -p "$_nw" +printf 'book acpid\nbook fcron\n' > "$_nw/demo.stack" +printf 'book acpid\nbook fcron\nbook zlib\ncfg shell-env\n' > "$_nw/demo.stack.new" +_o="$(timeout 120 python3 "$_pmJ2" stack "$_nw/demo.stack" 2>&1)" +case "$_o" in + *"newer version of this stack ships beside it"*) ;; + *) _p="$_p;a pending .new is not mentioned when the stack runs" ;; +esac +# the counts are what make it obvious something is missing +case "$_o" in + *"(4 entries, this file has 2)"*) ;; + *) _p="$_p;the entry counts are not compared" ;; +esac +case "$_o" in + *"diff $_nw/demo.stack $_nw/demo.stack.new"*) ;; + *) _p="$_p;no command is offered to see the difference" ;; +esac +# ...and it is a note, not a refusal: the run proceeds with the user's file +case "$_o" in + *"this run uses YOUR file"*) ;; + *) _p="$_p;it does not say which file this run actually uses" ;; +esac +# --only NAMING SOMETHING THAT IS ONLY IN THE .new FILE. "not in this stack: +# mytools" was true and useless -- the entry was in the newer copy beside it, +# one flag away, and the message did not say so. +printf 'book acpid\nbook fcron\n' > "$_nw/only.stack" +printf 'book acpid\nbook fcron\ncfg mytools\n' > "$_nw/only.stack.new" +_oo="$(timeout 120 python3 "$_pmJ2" stack "$_nw/only.stack" --only mytools 2>&1)" +case "$_oo" in + *"IS in the newer file beside this one"*) ;; + *) _p="$_p;--only does not look in the .new file for a name it cannot find" ;; +esac +case "$_oo" in + *"--take-new --only mytools"*) ;; + *) _p="$_p;it does not offer the exact command to run" ;; +esac +# a name in neither file is still just an error +_oo2="$(timeout 120 python3 "$_pmJ2" stack "$_nw/only.stack" --only nowhere 2>&1)" +case "$_oo2" in + *"IS in the newer file"*) _p="$_p;a name in neither file is blamed on the .new" ;; +esac +case "$_oo2" in + *"not in this stack: nowhere"*) ;; + *) _p="$_p;an unknown entry is no longer reported" ;; +esac - # end to end: an unstamped tree, and a mismatched one - _st="$T/stamp"; mkdir -p "$_st/progress" - _r="$( STATE_PROGRESS="$_st/progress"; LFS_HELPER_VERSION="9.9.9" - eval "$(_slice_fn "$helper_src" _warn_if_scripts_are_stale)" - warn() { echo "WARNED: $*"; } - _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "unstamped-passed" - printf 'version=9.9.9\n' > "$_st/progress/generated-by" - _warn_if_scripts_are_stale >/dev/null 2>&1 || echo "matching-failed" - printf 'version=1.0.0\n' > "$_st/progress/generated-by" - _warn_if_scripts_are_stale >/dev/null 2>&1 && echo "mismatch-passed" )" - case "$_r" in - "") ok "an unstamped or mismatched step order is caught, a matching one is not" ;; - *) printf '%s\n' "$_r" | while IFS= read -r m; do - [ -n "$m" ] && bad "stale-script check is wrong: $m" - done ;; - esac - rm -rf "$_st" +# A .new WITH FEWER ENTRIES IS A DOWNGRADE. The chroot refresh writes +# <stack>.new from the HOST's copy; if the host is the stale one, --take-new +# would replace a 6-entry stack with a 3-entry one -- and the user did +# exactly that, twice, because nothing stopped it. +printf 'book acpid\nbook fcron\ncfg mytools\ncfg shell-env\n' > "$_nw/big.stack" +printf 'book acpid\nbook fcron\n' > "$_nw/big.stack.new" +_od="$(timeout 120 python3 "$_pmJ2" stack "$_nw/big.stack" --take-new 2>&1)" +case "$_od" in + *"FEWER entries"*) ;; + *) _p="$_p;--take-new silently replaces a stack with a smaller one" ;; +esac +grep -q '^cfg mytools$' "$_nw/big.stack" \ + || _p="$_p;the bigger file was replaced after all" +case "$_od" in + *"on the HOST"*) ;; + *) _p="$_p;it does not say where the stale copy comes from" ;; +esac +_of="$(timeout 120 python3 "$_pmJ2" stack "$_nw/big.stack" --take-new --force 2>&1)" +case "$_of" in + *"took the shipped stack file"*) ;; + *) _p="$_p;--force cannot override the downgrade refusal" ;; +esac +# and no orphaned reference from the --only consolidation +timeout 120 python3 "$_pmJ2" stack "$_nw/demo.stack" 2>&1 | grep -q "NameError" \ + && _p="$_p;a stack run raises NameError" + +# --take-new does the merge instead of describing it again +printf 'book acpid\nbook fcron\ncfg mytools\n' > "$_nw/demo.stack.new" +_o3="$(timeout 120 python3 "$_pmJ2" stack "$_nw/demo.stack" --take-new 2>&1)" +case "$_o3" in + *"took the shipped stack file"*) ;; + *) _p="$_p;--take-new does not take the shipped file" ;; +esac +case "$_o3" in + *"3 entries now"*) ;; + *) _p="$_p;the run does not continue with the new file" ;; +esac +[ -f "$_nw/demo.stack.bak" ] || _p="$_p;the user's own file was not kept as .bak" +[ -f "$_nw/demo.stack.new" ] && _p="$_p;the .new file still lingers after taking it" +grep -q '^cfg mytools$' "$_nw/demo.stack" \ + || _p="$_p;the taken file is not the shipped one" +grep -q '^book acpid$' "$_nw/demo.stack.bak" \ + || _p="$_p;the .bak is not the user's original" +python3 "$_pmJ2" stack --help 2>&1 | grep -q -- '--take-new' \ + || _p="$_p;--take-new is not offered in the help" +# no .new -> no noise +rm -f "$_nw/demo.stack.new" +_o2="$(timeout 120 python3 "$_pmJ2" stack "$_nw/demo.stack" 2>&1)" +case "$_o2" in + *"newer version"*) _p="$_p;the warning appears when there is no .new file" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "running a stack says when a newer version waits beside it" fi -# ---- unpacked sources are scratch, and no scan should walk them ------------- # -# Build trees moved from /build into each package user's home in 1.7.6. The -# snapshot scans followed; the ownership scans did not. So `lfs-helper verify` -# began walking every unpacked source tree in the system -- it ran for minutes -# with no output -- and a finished tree filled the sanity report with tcl's own -# documentation: -# -# !! files with no owner (first 40): -# /usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Keywords/Z.htm +# ---- a step that installed nothing is not done (1.14.101) ------------------ # +# "which step exactly is compiling the seatd-n76310 program" cfg desktop-user +# -- and it had already run, and reported "done (stack cache)", with nothing +# installed. If the chroot has no network its three fetches all fail, the +# script prints "could not fetch" and exits 0, the stack records it done, and +# it never runs again. The same fault as a build whose log says it failed +# while its status says otherwise (1.14.64), one layer up. +_p="" +_sd4="$(dirname "$LFS_TOOL")/stacks" +_pmK2="$(dirname "$LFS_TOOL")/packagemanager" +# the step verifies its own work before succeeding +grep -q 'and are NOT installed' "$_sd4/desktop-user.sh" \ + || _p="$_p;desktop-user exits 0 even when nothing was installed" +grep -q 'exit 1' "$_sd4/desktop-user.sh" \ + || _p="$_p;the step never fails" +# ...including the mode: setuid root, group = the user, or the GPU stays shut +grep -q "4750" "$_sd4/desktop-user.sh" \ + || _p="$_p;the wrapper's setuid mode is not checked" +# and it names the way out when there is no network +grep -q 'SEATD_USER_URL=file://' "$_sd4/desktop-user.sh" \ + || _p="$_p;no offline route is offered" +# the runner does not record a git entry whose proof is still absent +_mk2="$(sed -n '/DO NOT RECORD A GIT ENTRY THAT IS NOT THERE/,/done += 1/p' "$_pmK2")" +printf '%s' "$_mk2" | grep -q 'not _git_installed(e)' \ + || _p="$_p;a git entry is recorded done without checking it landed" +printf '%s' "$_mk2" | grep -q 'NOT recording' \ + || _p="$_p;nothing says why the entry was not recorded" +printf '%s' "$_mk2" | grep -q 'the next run tries again' \ + || _p="$_p;it does not say the entry will be retried" +# the fixture logic itself: a missing target must be detected +_dm="$T/deskmiss"; mkdir -p "$_dm" +_o="$(u=nobodyxyz; home="$_dm" + missing="" + [ -x "/bin/seatd-$u" ] || missing="$missing /bin/seatd-$u" + [ -x "$home/bin/sway_start" ] || missing="$missing $home/bin/sway_start" + printf '%s' "$missing")" +case "$_o" in + *"/bin/seatd-nobodyxyz"*) ;; + *) _p="$_p;the missing-file check does not notice an absent wrapper" ;; +esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a config step that installed nothing fails, and is not cached as done" +fi + +# ---- not everything in that directory is a stack (1.14.105) ---------------- # +# The chroot refresh, one release after learning to copy subdirectories: +# # refreshed in the chroot: .shipped/desktop-user.sh, .shipped/kernel- +# sway.sh, ..., services.stack, services.stack.new +# Two faults. .shipped/ is make install's record of what it last shipped +# (1.14.98) and .new/.bak are merge leftovers -- bookkeeping, which does not +# travel. And services.stack itself was copied over the chroot's: the user +# had just taken the 6-entry file there with --take-new, and the host's +# 3-entry copy went straight back over it, undoing the fix from the release +# before. +_p="" +_lfs="$LFS_TOOL" +_rf="$(sed -n '/NOT EVERYTHING IN THAT DIRECTORY IS A STACK/,/dst_f = os.path.join(_stack_dst, fn)/p' "$_lfs")" +printf '%s' "$_rf" | grep -q 'fn.startswith(".")' \ + || _p="$_p;the .shipped bookkeeping directory still crosses into the chroot" +printf '%s' "$_rf" | grep -q 'fn.endswith((".new", ".bak"))' \ + || _p="$_p;merge leftovers still cross into the chroot" +_rg="$(sed -n '/AND A STACK FILE THAT DIFFERS IS THE CHROOT/,/shutil.copy2(src_f, dst_f)$/p' "$_lfs")" +printf '%s' "$_rg" | grep -q 'fn.endswith(".stack") and os.path.exists(dst_f)' \ + || _p="$_p;a stack file edited in the chroot is overwritten from the host" +printf '%s' "$_rg" | grep -q 'dst_f + ".new"' \ + || _p="$_p;the incoming stack file is not left beside the chroot's" +# the rule in practice: bookkeeping skipped, payload copied, chroot's kept +python3 - "$T/refresh" <<'PYREF' || _p="$_p;the refresh rules do not behave as written" +import sys, os, shutil +d = sys.argv[1] +src, dst = os.path.join(d, "host"), os.path.join(d, "chroot") +for x in (os.path.join(src, ".shipped"), os.path.join(src, "mytools"), dst): + os.makedirs(x, exist_ok=True) +open(os.path.join(src, ".shipped", "x.sh"), "w").write("bookkeeping\n") +open(os.path.join(src, "mytools", "install_mytools"), "w").write("#!/bin/bash\n") +open(os.path.join(src, "services.stack"), "w").write("book acpid\n") +open(os.path.join(src, "services.stack.new"), "w").write("leftover\n") +open(os.path.join(dst, "services.stack"), "w").write( + "book acpid\nbook fcron\ncfg mytools\n") +for fn in sorted(os.listdir(src)): + if fn.startswith(".") or fn.endswith((".new", ".bak")): + continue + s_f, d_f = os.path.join(src, fn), os.path.join(dst, fn) + if os.path.isdir(s_f): + os.makedirs(d_f, exist_ok=True) + for sub in sorted(os.listdir(s_f)): + shutil.copy2(os.path.join(s_f, sub), os.path.join(d_f, sub)) + continue + if fn.endswith(".stack") and os.path.exists(d_f): + shutil.copy2(s_f, d_f + ".new") + continue + shutil.copy2(s_f, d_f) +assert not os.path.exists(os.path.join(dst, ".shipped")), "bookkeeping crossed" +assert os.path.isfile(os.path.join(dst, "mytools", "install_mytools")), "payload missing" +kept = open(os.path.join(dst, "services.stack")).read().splitlines() +assert len(kept) == 3, kept # the chroot's own file, not the host's +assert os.path.isfile(os.path.join(dst, "services.stack.new")), "no .new left beside" +PYREF +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the chroot refresh carries stacks, not bookkeeping, and never downgrades" +fi + +# ---- your packages, under your name (1.14.106) ----------------------------- # +# Three things from one run. # -# A tarball can carry any uid it likes. Unpacked sources are not installed, -# nothing owns them, and asking who does has no answer. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" scan_prune_paths | grep -q 'PKGUSR_BUILD_SUBDIR' \ - || _p="$_p;the prune list does not cover the per-package build trees" - # ONE list -- there were five and they had already diverged - for _fn in _vfy_orphans _has_orphaned_files _vfy_build_user_leftovers; do - _slice_fn "$helper_src" "$_fn" | grep -q 'SCAN_PRUNE' \ - || _p="$_p;$_fn does not use the shared prune list" - done - # an ARRAY, not a string through eval: the patterns contain *, and eval lets - # the shell expand them before find ever sees them - case "$(_slice_fn "$helper_src" scan_prune_set)" in - *'SCAN_PRUNE+=('*) ;; - *) _p="$_p;the prune list is not built as an array" ;; - esac - if _slice_fn "$helper_src" _vfy_orphans | grep -q 'eval find'; then - _p="$_p;the orphan scan splices its prune list through eval" - fi - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "every ownership scan skips the same scratch, from one list" - fi +# (a) The account: "the user should be called p_n76310 -- that's a fixed name +# for those packages as they come from me." Right: p_<your user> is where +# anyone would look. The script and the step take the name from the +# desktop user. +# (b) The build line died with error: 'n76310' undeclared. The SOURCES entry +# was double-quoted, so \" collapsed before eval and gcc got a bare +# identifier. Single-quoted now, and the command is the author's own, +# verbatim. +# (c) --only was declared in the parser long ago, documented as "repeatable or +# comma-separated ... rebuilt even if the cache says done", and never +# implemented. A stack is long and its entries fail one at a time. +_p="" +_sd6="$(dirname "$LFS_TOOL")/stacks" +_ms2="$_sd6/mytools/install_mytools" +_pmL2="$(dirname "$LFS_TOOL")/packagemanager" +# (a) the account is named after the desktop user +grep -q 'MYTOOLS_NAME:-\${DESKTOP_USER' "$_ms2" \ + || _p="$_p;the package is not named after the desktop user" +grep -q 'pkg="\${MYTOOLS_NAME:-\$u}"' "$_sd6/mytools.sh" \ + || _p="$_p;the step still creates a fixed 'mytools' account" +grep -qE 'lfs-helper pm-install "\$(acct|p_\$pkg)"' "$_sd6/mytools.sh" \ + || _p="$_p;the install does not go to p_<user>" +grep -q 'acct="p_\$pkg"' "$_sd6/mytools.sh" \ + || _p="$_p;the account is not derived from the package name" +# (b) SOURCES elements are single-quoted, and the gcc line survives eval +grep -q "^'tar|seatd-user|" "$_ms2" \ + || _p="$_p;the seatd-user entry is not single-quoted" +_cmd="$(DESKTOP_USER=n76310 OUT=/tmp python3 - "$_ms2" <<'PYCMD' +import os, subprocess, sys +for ln in open(sys.argv[1]): + ln = ln.strip() + if ln.startswith("'tar|seatd-user|"): + build = ln.strip("'").split("|")[4] + print(subprocess.run(["bash", "-c", 'eval echo "%s"' % build], + capture_output=True, text=True).stdout.strip()) + break +PYCMD +)" +case "$_cmd" in + "gcc seatd-user.c -DSEATD_USER=n76310 -o /tmp/seatd-n76310") ;; + *) _p="$_p;the build command comes out as '$_cmd'" ;; +esac +# (c) --only really selects, refuses an unknown name, and takes a comma list +_oo="$T/onlystack"; mkdir -p "$_oo" +printf 'book acpid\nbook fcron\nbook zlib\n' > "$_oo/o.stack" +_o="$(timeout 120 python3 "$_pmL2" stack "$_oo/o.stack" --only nosuch 2>&1)" +case "$_o" in + *"not in this stack: nosuch"*) ;; + *) _p="$_p;an unknown --only name is not refused" ;; +esac +case "$_o" in *"it has:"*) ;; *) _p="$_p;it does not list what the stack has" ;; esac +_o2="$(timeout 120 python3 "$_pmL2" stack "$_oo/o.stack" --only acpid,zlib 2>&1)" +case "$_o2" in *"acpid"*) ;; *) _p="$_p;--only dropped a name it was given" ;; esac +case "$_o2" in *"zlib"*) ;; *) _p="$_p;a comma-separated --only is not split" ;; esac +case "$_o2" in *" fcron "*) _p="$_p;--only ran an entry it was not given" ;; esac +# and a named entry is exempt from the done-cache +sed -n '/RUN JUST ONE PART, AND REALLY RUN IT/,/not in _only}/p' "$_pmL2" \ + | grep -q 'donekeys = {k for k in donekeys' \ + || _p="$_p;--only still skips an entry the cache calls done" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "your packages install under p_<you>, and --only runs one entry again" +fi + +# ---- the skeleton is not data (1.14.112) ----------------------------------- # +# lfs-helper fix-home p_n76310 --run +# /usr/src/p_n76310 kept: these files differ ... compare and remove by hand +# differs: .project, build.conf, .bash_profile, .bashrc, build +# # p_n76310: home -> /usr/src/pkgusr/p_n76310 +# ...and the next run: "p_n76310's home is /usr/src/p_n76310" +# Those five files are the SKELETON -- written by whoever created the account +# -- so a husk and a real home always "differ" in exactly them, the merge +# always refused, and the repair could never finish. The destination's copies +# are the ones in use; the husk's are a duplicate of the same idea. +_p="" +_hlpS2="$helper_src" +grep -q '^_is_skeleton_file() {' "$_hlpS2" \ + || _p="$_p;there is no notion of a skeleton file" +for _f in .bashrc .bash_profile build build.conf .project; do + sed -n '/^_is_skeleton_file() {/,/^}/p' "$_hlpS2" | grep -q -- "$_f" \ + || _p="$_p;$_f is not treated as skeleton" +done +sed -n '/^_merge_dir_into() {/,/^}/p' "$_hlpS2" | grep -q '_is_skeleton_file "$rel"' \ + || _p="$_p;the merge still refuses over skeleton files" +# it really merges: payload moves, skeleton is dropped, husk disappears +_mh="$T/mergehome"; rm -rf "$_mh"; mkdir -p "$_mh/husk" "$_mh/real" +for _f in .bashrc .bash_profile build.conf .project; do + echo husk > "$_mh/husk/$_f"; echo real > "$_mh/real/$_f" +done +echo payload > "$_mh/husk/pkg.lst" +_r="$( warn(){ :; }; ok(){ :; } + eval "$(_slice_fn "$_hlpS2" _is_skeleton_file)" + eval "$(_slice_fn "$_hlpS2" _merge_dir_into)" + _merge_dir_into "$_mh/husk" "$_mh/real" && echo CLEAN || echo KEPT )" +[ "$_r" = CLEAN ] || _p="$_p;the merge still refuses a husk that holds only skeleton files" +[ -d "$_mh/husk" ] && _p="$_p;the husk directory survives the merge" +[ -f "$_mh/real/pkg.lst" ] || _p="$_p;the payload was not moved into the real home" +[ "$(cat "$_mh/real/.bashrc" 2>/dev/null)" = real ] \ + || _p="$_p;the husk's skeleton overwrote the real home's" +# a husk holding a file that is NOT skeleton and NOT identical is still kept +_mh2="$T/mergehome2"; rm -rf "$_mh2"; mkdir -p "$_mh2/husk" "$_mh2/real" +echo one > "$_mh2/husk/notes.txt"; echo two > "$_mh2/real/notes.txt" +_r2="$( warn(){ :; }; ok(){ :; } + eval "$(_slice_fn "$_hlpS2" _is_skeleton_file)" + eval "$(_slice_fn "$_hlpS2" _merge_dir_into)" + _merge_dir_into "$_mh2/husk" "$_mh2/real" && echo CLEAN || echo KEPT )" +[ "$_r2" = KEPT ] || _p="$_p;a genuinely conflicting file is discarded" +[ "$(cat "$_mh2/real/notes.txt")" = two ] || _p="$_p;a real file was overwritten by the husk's" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a husk of skeleton files merges away; real differences are still kept" +fi + +# ---- check that it took (1.14.113) ----------------------------------------- # +# lfs-helper fix-home p_n76310 --run +# # p_n76310: home -> /usr/src/pkgusr/p_n76310 +# ...and the next command: "p_n76310's home is /usr/src/p_n76310" +# Three times, each reporting success. `usermod -d` returned 0 and +# /etc/passwd still held the old home, so the message was printed on the +# strength of an exit code that meant nothing. A write that cannot be read +# back did not happen. +_p="" +_hlpT2="$helper_src" +_sr="$(_slice_fn "$_hlpT2" _set_recorded_home)" +printf '%s' "$_sr" | grep -q '\[ "$(_recorded_home_for "$acct")" = "$home" \] && return 0' \ + || _p="$_p;usermod's exit code is still trusted" +printf '%s' "$_sr" | grep -cq 'CHECK THAT IT TOOK' \ + || _p="$_p;the read-back is not explained" +# the file path verifies too, and a name that is not in passwd fails +_sp="$T/setrec"; mkdir -p "$_sp" +printf 'root:x:0:0::/root:/bin/bash\np_demo:x:10:10::/usr/src/p_demo:/bin/bash\n' \ + > "$_sp/passwd" +_res="$( ETC="$_sp" + have_shadow_tools() { return 1; } + eval "$(_slice_fn "$_hlpT2" _recorded_home_for)" + eval "$(_slice_fn "$_hlpT2" _set_recorded_home)" + _set_recorded_home p_demo /usr/src/pkgusr/p_demo && echo OK || echo FAIL + _recorded_home_for p_demo + _set_recorded_home p_absent /usr/src/pkgusr/x && echo OK2 || echo FAIL2 )" +case "$_res" in + *OK*"/usr/src/pkgusr/p_demo"*) ;; + *) _p="$_p;a real move is not reported as success: $_res" ;; +esac +case "$_res" in + *FAIL2*) ;; + *) _p="$_p;setting the home of an account that is not there reports success" ;; +esac +# and the caller only claims the repair when the write is confirmed +_rh="$(_slice_fn "$_hlpT2" _repair_misplaced_home)" +printf '%s' "$_rh" | grep -q 'if _set_recorded_home "$acct" "$want"; then' \ + || _p="$_p;the repair announces itself without checking the write" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a recorded home is read back before the tools claim they moved it" +fi - # end to end: a build tree is pruned, a real file is not - _pt="$T/prune" - mkdir -p "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html" "$_pt/usr/bin" - touch "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" - if [ "$(id -u)" = 0 ]; then - chown 4321 "$_pt/usr/src/pkgusr/p_tcl/src/tcl8.6.16/html/Z.htm" "$_pt/usr/bin/real" - _r="$( SNAP_ROOT="$_pt"; STATE="$_pt/usr/src/lfs-pkgusr" - PKGUSR_BUILD_SUBDIR=src - PKGUSR_ROOT="$_pt/usr/src/pkgusr"; CFGUSR_ROOT="$_pt/usr/src/cfg" - pkgusr_roots() { printf '%s\n%s\n' "$PKGUSR_ROOT" "$CFGUSR_ROOT"; } - eval "$(_slice_fn "$helper_src" scan_prune_paths)" - eval "$(_slice_fn "$helper_src" scan_prune_set)" - scan_prune_set - find "$_pt" -xdev -nouser "${SCAN_PRUNE[@]}" 2>/dev/null \ - | sed "s|$_pt||" | sort | tr '\n' ' ' )" - case "$_r" in - "/usr/bin/real ") ok "an unpacked source tree is skipped, an installed file is not" ;; - *) bad "the prune list is wrong: got [$_r]" ;; +# ---- one base, two readings (1.14.114) ------------------------------------- # +# After fix-home the passwd entry was right; running the step put it back: +# getent ... -> /usr/src/pkgusr/p_n76310 (after fix-home) +# ... the step ... +# getent ... -> /usr/src/p_n76310 (after) +# The step's environment carries LFS_PKGUSR_ROOT, written by `lfs` from the +# config key `pkgusr_home`. Its default is /usr/src/pkgusr, its help text +# said "the base the homes hang under (/usr/src)", and there is a SECOND key, +# pkgusr_subdir, "folder under the base". Read one way the base includes the +# folder; read the other it does not. packagemanager appends the subdir; +# `lfs` exported the bare base. So the tools disagreed about where every +# account lives, and fix-home and the step took turns correcting each other. +_p="" +_lfsB="$LFS_TOOL" +grep -q '^def _accounts_root' "$_lfsB" \ + || _p="$_p;the exported root is still the bare config value" +grep -q 'export LFS_PKGUSR_ROOT="{_accounts_root("pkgusr")}"' "$_lfsB" \ + || _p="$_p;LFS_PKGUSR_ROOT is not composed" +grep -q 'export LFS_CFGUSR_ROOT="{_accounts_root("cfg")}"' "$_lfsB" \ + || _p="$_p;LFS_CFGUSR_ROOT is not composed" +grep -q 'where package homes live (/usr/src/pkgusr)' "$_lfsB" \ + || _p="$_p;the help text still invites the wrong reading" +# both readings must land in the same place, and a custom subdir must hold +python3 - "$_lfsB" <<'PYROOT' || _p="$_p;the two readings of the base do not agree" +import sys, importlib.machinery as m +l = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +l.read_config = lambda *a, **k: {} +for base in ("/usr/src", "/usr/src/pkgusr"): + l.LAYOUT_DEFAULTS["target_pkgusr_home"] = base + got = l._accounts_root("pkgusr") + assert got == "/usr/src/pkgusr", (base, got) +l.LAYOUT_DEFAULTS["target_pkgusr_home"] = "/opt/lfs/src" +assert l._accounts_root("pkgusr") == "/opt/lfs/src/pkgusr", l._accounts_root("pkgusr") +# a configured subdir is honoured, and not doubled +l.read_config = lambda *a, **k: {"pkgusr_subdir": "packages"} +l.LAYOUT_DEFAULTS["target_pkgusr_home"] = "/usr/src" +assert l._accounts_root("pkgusr") == "/usr/src/packages", l._accounts_root("pkgusr") +l.LAYOUT_DEFAULTS["target_pkgusr_home"] = "/usr/src/packages" +assert l._accounts_root("pkgusr") == "/usr/src/packages", l._accounts_root("pkgusr") +PYROOT +# and lfs-helper's own default already agreed with packagemanager +grep -q 'PKGUSR_ROOT="${LFS_PKGUSR_ROOT:-$SRCROOT/pkgusr}"' "$helper_src" \ + || _p="$_p;lfs-helper's default root no longer includes the subdir" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every tool computes the same home from either reading of the base" +fi + +# ---- the DRM backend was never built (1.14.115) ---------------------------- # +# [wlr] Cannot create DRM backend: disabled at compile-time +# [wlr] Failed to open any DRM device +# [sway] Unable to create backend +# Not the kernel, and not seatd -- the log shows seatd opening seat0 and the +# client attaching correctly. wlroots 0.19's DRM backend requires +# libdisplay-info, which was in the BOOK and not in the stack; meson disables +# drm when it is absent, mentions it in a summary line, and builds happily. +# The cost of that silence is a full desktop that cannot open a screen. +_p="" +_ss6="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_pmDR="$(dirname "$LFS_TOOL")/packagemanager" +grep -qE '^book +libdisplay-info' "$_ss6" \ + || _p="$_p;libdisplay-info is not in the sway stack" +# ...and BEFORE wlroots, or it cannot help +python3 - "$_pmDR" "$_ss6" <<'PYORD' || _p="$_p;libdisplay-info is not built before wlroots" +import sys, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +names = [r["target"] for r in pm._parse_stack_file(sys.argv[2])] +assert "libdisplay-info" in names, "missing" +assert names.index("libdisplay-info") < names.index("wlroots"), names +PYORD +# the backend list is explicit: an auto-detected one is a silent dependency +grep -q 'Dbackends=drm,libinput,x11' "$_ss6" \ + || _p="$_p;wlroots' backends are still auto-detected" +grep -q 'Dsession=enabled' "$_ss6" \ + || _p="$_p;wlroots is built without session support" +# seatd: the wrapper execs /usr/bin/seatd, so the server must be built +grep -q 'Dserver=enabled' "$_ss6" \ + || _p="$_p;seatd is built without the binary seatd-user needs" +grep -q 'Dlibseat-seatd=enabled' "$_ss6" \ + || _p="$_p;libseat has no seatd backend, which is the one in use" +# the author's own sway options +for _o in 'Dman-pages=enabled' 'Dswaybar=true' 'Dswaynag=true' 'Dbash-completions=true'; do + grep -q -- "$_o" "$_ss6" || _p="$_p;sway is missing $_o" +done +# the seatd BOOT SERVICE is gone: seatd-user replaced it, and two ways to +# get a seat is worse than one +[ -f "$(dirname "$LFS_TOOL")/stacks/seatd-init.sh" ] \ + && _p="$_p;the seatd boot script is still shipped" +grep -qE '^cfg +seatd-init' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + && _p="$_p;services.stack still installs the seatd service" +# add-user must report the account it made, not the login of the same name +sed -n '/THE ACCOUNT THAT WAS CREATED, NOT THE NAME/,/return/p' "$_pmDR" \ + | grep -q 'name = pkgusr_name(name)' \ + || _p="$_p;the creation report can still describe the human account" +grep -q 'report_created_user(acct)' "$_pmDR" \ + || _p="$_p;add-user reports on the name it was given" +# the kernel step, from the author's procedure +_k="$(dirname "$LFS_TOOL")/lfs-kernel" +[ -f "$_k" ] || _p="$_p;there is no kernel tool" +[ -x "$_k" ] || _p="$_p;lfs-kernel is not executable" +[ -f "$(dirname "$LFS_TOOL")/stacks/kernel.sh" ] \ + && _p="$_p;the kernel is still a stack step" +grep -qE '^cfg +kernel$' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + && _p="$_p;services.stack still runs the kernel as a step" +# the config is SHIPPED, not written on first run: it is meant to be edited +# before anything is built +_kc="$(dirname "$LFS_TOOL")/kernel.conf" +[ -f "$_kc" ] || _p="$_p;kernel.conf does not ship" +grep -q 'kernel.conf' "$(dirname "$LFS_TOOL")/Makefile" \ + || _p="$_p;make install does not place kernel.conf" +grep -q 'is missing. It ships with the tools' "$_k" \ + || _p="$_p;a missing config is written silently instead of reported" +# --help and detect-esp must work with no config present +_hh="$(env CONF=/nonexistent bash "$_k" --help 2>&1 | head -1)" +case "$_hh" in + *"build and install a kernel"*) ;; + *) _p="$_p;--help does not work without a config" ;; +esac +# ESP + KERNEL_NAME are the pair that picks test vs real +grep -q 'KERNEL_NAME' "$_k" || _p="$_p;the installed kernel file name is fixed" +grep -q 'KERNEL_NAME' "$_kc" || _p="$_p;the config does not expose the file name" +grep -q 'a kernel to try' "$_kc" || _p="$_p;the config does not show a test/real pair" +grep -q 'KERNEL_NAME}_old' "$_k" || _p="$_p;the previous kernel is not kept" +# THE LAST WORKING SOURCE IS NEVER DELETED +grep -q 'kernel.last-good' "$_k" \ + || _p="$_p;nothing records which kernel last booted" +grep -q 'THE LAST WORKING SOURCE IS NEVER DELETED' "$_k" \ + || _p="$_p;the last working source is not protected" +grep -q 'uname -r' "$_k" || _p="$_p;the running kernel is not consulted" +# subcommands +for _c in list detect-esp verify; do + grep -q " $_c)" "$_k" || _p="$_p;lfs-kernel has no '$_c'" +done + +# VERIFY LOOKS AT THE SYSTEM, not at what a build printed. "is the kernel +# part done? do we have a validation command so we know everything is +# installed as expected?" -- there was none, so every answer so far has been +# me reading a transcript. +bash "$_k" --help 2>&1 | grep -q 'verify' \ + || _p="$_p;verify is not documented" +_vb="$(sed -n '/verify: is everything where it should be/,/^echo "# kernel /p' "$_k")" +[ -n "$_vb" ] || _p="$_p;there is no verify block" +for _c in 'System.map-' 'config-' 'modules.dep' 'version.h' 'pkg.lst' \ + 'CONFIG_EXTRA_FIRMWARE' 'lib/modules'; do + printf '%s' "$_vb" | grep -q -- "$_c" || _p="$_p;verify does not check $_c" +done +printf '%s' "$_vb" | grep -q 'cmp -s' \ + || _p="$_p;verify does not check the booted image against the version" +printf '%s' "$_vb" | grep -q 'ls -1t' \ + || _p="$_p;verify cannot work out which version was installed" +printf '%s' "$_vb" | grep -q 'exit 1' \ + || _p="$_p;verify does not fail when a check fails" +# DEPMOD AND OUT-OF-TREE MODULES MUST TARGET THE VERSION BUILT, not the one +# running. `depmod -a` with no argument means `uname -r`, and a module's own +# Makefile defaults to /lib/modules/`uname -r`/build: +# depmod: ERROR: could not open directory /lib/modules/7.2.0 +# make[1]: *** /lib/modules/7.2.0/build: No such file or directory +# In a chroot -- or any time a different version is being built -- both are +# the wrong tree. +grep -qE '^depmod -a "\$VERSION"' "$_k" \ + || _p="$_p;depmod indexes the running kernel instead of the built one" +grep -qE 'depmod -a$' "$_k" \ + && _p="$_p;a bare 'depmod -a' is still there" +for _t in clean modules modules_install; do + grep -q "make -C '\$tree' M='\$d' $_t" "$_k" \ + || _p="$_p;v4l2loopback's $_t does not run against the built tree" +done +grep -q "cd '\$d' && make clean && make" "$_k" \ + && _p="$_p;the module still builds with its own Makefile defaults" +# uname -r may only be used for REPORTING and for last-good tracking +_ur="$(grep -n 'uname -r' "$_k" | grep -vE 'echo|#|_running=' | wc -l)" +[ "$_ur" = 0 ] || _p="$_p;uname -r is still used to decide where files go" + +# VERIFY IS A REPORT: it must not fetch, create or announce. It resolved +# "latest" over the network, printed the chroot notice about paths it was +# never going to write, and would have created the package account -- in +# front of a report about what is installed. +_vq="$(sed -n '/verify.*inspects what is on disk/,/^fi$/p' "$_k")" +[ -n "$_vq" ] || _p="$_p;verify still resolves the version over the network" +grep -q 'must not CREATE anything' "$_k" \ + || _p="$_p;verify can create the package account" +[ "$(grep -c '_VERIFY:-0' "$_k")" -ge 5 ] \ + || _p="$_p;verify shares too much of the build path" +# THE MANIFEST IS RECORDED WHEN THE KERNEL IS INSTALLED, not after the +# optional extras: a v4l2loopback failure left no pkg.lst and verify then +# reported a perfectly installed kernel as unrecorded. +grep -q 'RECORD THE FILES NOW, WHILE THEY ARE INSTALLED' "$_k" \ + || _p="$_p;the manifest is only written after the optional steps" +_l_dep="$(grep -n '^depmod -a "\$VERSION"' "$_k" | head -1 | cut -d: -f1)" +_l_rec="$(grep -n '^_record$' "$_k" | head -1 | cut -d: -f1)" +_l_v4l="$(grep -n 'v4l2loopback (optional' "$_k" | head -1 | cut -d: -f1)" +{ [ -n "$_l_rec" ] && [ -n "$_l_v4l" ] && [ "$_l_rec" -lt "$_l_v4l" ]; } \ + || _p="$_p;the manifest is still written after v4l2loopback" +{ [ -n "$_l_dep" ] && [ "$_l_dep" -lt "$_l_rec" ]; } \ + || _p="$_p;the manifest is written before depmod has indexed the modules" +[ "$(grep -c '^_record$' "$_k")" -ge 2 ] \ + || _p="$_p;the manifest is not refreshed after the headers are installed" + +# VERIFY MUST CHECK WHAT IT INSTALLS. It checked the image, the modules, +# the headers and every embedded blob -- and not the out-of-tree module, +# which is the one piece whose build had failed three times. +printf '%s' "$_vb" | grep -q 'v4l2loopback.ko' \ + || _p="$_p;verify does not check the out-of-tree module" +printf '%s' "$_vb" | grep -q 'modules.dep knows v4l2loopback' \ + || _p="$_p;verify does not check that depmod indexed the module" +printf '%s' "$_vb" | grep -q 'V4L2LOOPBACK:-no' \ + || _p="$_p;the module is checked even when the config does not ask for it" +# ...and a manifest that has the home back in it is reported +printf '%s' "$_vb" | grep -q "the manifest includes the account's own home" \ + || _p="$_p;verify does not notice a manifest full of sources" +_vm="$T/vmod"; rm -rf "$_vm"; mkdir -p "$_vm/kernel/drivers" +: > "$_vm/kernel/drivers/v4l2loopback.ko" +printf 'kernel/drivers/v4l2loopback.ko:\n' > "$_vm/modules.dep" +find "$_vm" -name 'v4l2loopback.ko*' -print -quit 2>/dev/null | grep -q . \ + || _p="$_p;the module search does not find an installed module" +rm -f "$_vm/kernel/drivers/v4l2loopback.ko" +find "$_vm" -name 'v4l2loopback.ko*' -print -quit 2>/dev/null | grep -q . \ + && _p="$_p;the module search finds a module that is not there" + +# a zero-byte image and a mismatched pair must both be caught +_vk="$T/verifyk"; rm -rf "$_vk"; mkdir -p "$_vk/esp" +: > "$_vk/esp/k" +[ -s "$_vk/esp/k" ] && _p="$_p;a zero-byte kernel image would pass" +echo img > "$_vk/esp/k"; echo other > "$_vk/esp/k-7.2.4" +cmp -s "$_vk/esp/k" "$_vk/esp/k-7.2.4" \ + && _p="$_p;a booted image differing from its version would pass" +# the AMD blobs this config embeds are in the shipped list +_fo2="$( . "$_kc" 2>/dev/null; printf '%s' "$FIRMWARE_ONLY" )" +for _d in amd-ucode amd intel amdgpu; do + case " $_fo2 " in + *" $_d "*) ;; + *) _p="$_p;the shipped firmware list omits $_d" ;; + esac +done +if [ -f "$_k" ]; then + bash -n "$_k" || _p="$_p;kernel.sh does not parse" + grep -q '/etc/pkgusr/kernel.conf' "$_k" || _p="$_p;the kernel step has no config file" + # THE VERSION MUST BE EASY TO CHANGE: newest stable from kernel.org, a + # named version for one run, or a list to choose from + grep -q 'releases.json' "$_k" || _p="$_p;kernel.org's release list is not consulted" + grep -q 'KERNEL_LIST' "$_k" || _p="$_p;there is no way to list available versions" + # a version for one run is a positional argument now, not an env var + grep -q 'VERSION="$1"' "$_k" || _p="$_p;a version cannot be given for one run" + grep -q 'cdn.kernel.org' "$_k" || _p="$_p;the tarball is not fetched automatically" + grep -q 'THE COMMANDS' "$_k" || _p="$_p;the build commands are not marked out to be read" + # THE KERNEL IS A PACKAGE: its sources live in its own account's home, + # not in a directory invented by the config, so verify/pkg.lst/remove all + # work on it. SRC used to be a free-standing setting -- a location + # outside the model. + grep -q 'SRC=/usr/src/linux' "$_kc" && _p="$_p;the config still invents a source directory" + # an INSTALLED config from before the change still sets SRC and has no + # PKG; it must be recognised and explained, not silently misread + grep -q 'A CONFIG FROM AN OLDER RELEASE' "$_k" \ + || _p="$_p;an outdated config is read as if it were current" + _oc="$T/oldconf"; mkdir -p "$_oc" + printf 'VERSION=latest\nESP=%s\nSRC=/usr/src/linux\nJOBS=-j2\n' "$_oc" > "$_oc/old.conf" + _oo="$( CONF="$_oc/old.conf"; . "$_oc/old.conf" + if [ -n "${SRC:-}" ]; then echo "SRC-NOTED"; unset SRC; fi + [ -n "${PKG:-}" ] || echo "NO-PKG" )" + case "$_oo" in *SRC-NOTED*) ;; *) _p="$_p;a leftover SRC is not mentioned" ;; esac + case "$_oo" in *NO-PKG*) ;; *) _p="$_p;a config without PKG is accepted" ;; esac + # the values the author asked for + grep -q '^ESP=/boot/efi/EFI/NimGnu$' "$_kc" \ + || _p="$_p;the shipped ESP is not the author's" + grep -q '^KERNEL_NAME=vmlinuz_new$' "$_kc" \ + || _p="$_p;the shipped kernel name is not the author's" + grep -q '^JOBS=-j14$' "$_kc" \ + || _p="$_p;the shipped job count is not the author's default" + grep -q '^PKG=' "$_kc" || _p="$_p;the config does not name the package user" + grep -q 'THE KERNEL IS A PACKAGE' "$_k" || _p="$_p;the kernel is not built as a package" + grep -q 'packagemanager add-user "$PKG"' "$_k" || _p="$_p;the account is not created" + grep -q 'getent passwd "$acct" | cut -d: -f6' "$_k" \ + || _p="$_p;the source directory is not derived from the account" + grep -q 'reload-pkg-list "$PKG"' "$_k" \ + || _p="$_p;the kernel is never recorded in a manifest" + # everything touching the tree runs AS the package user; a root-owned + # file in its home breaks the next build + grep -q '_as_pkg() {' "$_k" || _p="$_p;there is no way to run as the package user" + for _stage in 'make $JOBS' 'make modules_prepare' 'make modules_install' \ + 'make olddefconfig' 'make headers'; do + grep -q "_as_pkg \"$_stage" "$_k" \ + || _p="$_p;'$_stage' does not run as the package user" + done + grep -qE '^make (\$JOBS|modules_install|headers)' "$_k" \ + && _p="$_p;a build stage still runs as root" + # ...and the ESP copies stay root's, since vfat has no ownership + grep -q 'cp -v "$tree/arch/x86/boot/bzImage" "$ESP/$KERNEL_NAME"' "$_k" \ + || _p="$_p;the kernel is not copied to the ESP by root" + # the ESP is explained and detected, not left as an unexplained variable + grep -q '_detect_esp' "$_k" || _p="$_p;the ESP is not detected" + grep -q 'findmnt -t vfat' "$_k" || _p="$_p;it does not say how to find the ESP" + # releases.json pairs its keys by NAME: "version" comes before "moniker", + # and pairing by position silently swapped the columns, so "latest" + # resolved to nothing + cat > "$T/rel.json" <<'RELJSON' +{"releases":[{"iseol":false,"version":"7.3.1","moniker":"mainline"},{"version":"7.2.5","moniker":"stable"},{"version":"6.12.40","moniker":"longterm"}]} +RELJSON + _lat="$( MIRROR_JSON=x + wget() { cat "$T/rel.json"; } + eval "$(sed -n '/^_releases() {/,/^}/p' "$_k")" + _releases | awk '$1=="stable"{print $2; exit}' )" + [ "$_lat" = "7.2.5" ] || _p="$_p;the newest stable is parsed as '$_lat', not 7.2.5" + _mon="$( MIRROR_JSON=x + wget() { cat "$T/rel.json"; } + eval "$(sed -n '/^_releases() {/,/^}/p' "$_k")" + _releases | awk '$1=="longterm"{print $2; exit}' )" + [ "$_mon" = "6.12.40" ] || _p="$_p;longterm is parsed as '$_mon'" + grep -q 'V4L2LOOPBACK' "$_k" || _p="$_p;v4l2loopback cannot be turned off" + grep -q 'CONFIG_DRM' "$_k" || _p="$_p;the step does not check the options sway needs" + grep -q 'make headers' "$_k" || _p="$_p;the headers are not installed" + # FIRMWARE IS PART OF THE JOB, NOT SOMETHING TO SKIP. amdgpu will not + # bring up a display without its blobs, and a config listing them in + # CONFIG_EXTRA_FIRMWARE cannot even build. They install as their own + # package so they are tracked and updatable on their own. + grep -q 'firmware, as its own package' "$_k" \ + || _p="$_p;firmware is not installed by the kernel tool" + grep -q 'add-user linux-firmware' "$_k" \ + || _p="$_p;firmware does not get its own package user" + grep -q 'reload-pkg-list linux-firmware' "$_k" \ + || _p="$_p;the installed firmware is not recorded" + grep -q '^FIRMWARE_FROM=' "$_kc" || _p="$_p;the config cannot choose a firmware source" + grep -q '^FIRMWARE_ONLY=' "$_kc" || _p="$_p;the config cannot limit which firmware" + grep -q '^FIRMWARE_ONLY="amdgpu' "$_kc" \ + || _p="$_p;the shipped config does not cover the author's GPU" + # NOT ALL FIRMWARE LIVES IN A SUBDIRECTORY. Intel wireless blobs are + # iwlwifi-cc-a0-*.ucode at the TOP LEVEL, so a directory-only list misses + # the AX200 entirely -- and git's sparse-checkout cone mode, the default, + # matches directories only. + # Intel wireless lives at intel/iwlwifi/ in current linux-firmware, so + # `intel` covers the AX200; a separate iwlwifi entry is simply wrong. + grep -q 'intel/iwlwifi/' "$_kc" \ + || _p="$_p;the config does not say where Intel wireless firmware lives" + # THE CONFIG IS SOURCED BY THE SHELL, so a value with spaces must be + # quoted: unquoted, `FIRMWARE_ONLY=amdgpu intel iwlwifi-cc-*` runs + # `intel` as a command and assigns NOTHING -- which then meant "install + # all ten gigabytes". + grep -q '^FIRMWARE_ONLY="' "$_kc" \ + || _p="$_p;FIRMWARE_ONLY is not quoted, so its value is lost" + _q="$( printf 'FIRMWARE_ONLY=a b c\n' > "$T/bad.conf" + . "$T/bad.conf" 2>/dev/null; printf '[%s]' "$FIRMWARE_ONLY" )" + [ "$_q" = "[]" ] || _p="$_p;the unquoted form no longer demonstrates the bug" + # a pattern that matches nothing must show what IS there: linux-firmware + # moves blobs between the top level and subdirectories between releases + grep -q 'WHAT IS THERE, NOT JUST WHAT IS NOT' "$_k" \ + || _p="$_p;a pattern matching nothing gives no help" + _fs2="$T/fwsim"; rm -rf "$_fs2"; mkdir -p "$_fs2/iwlwifi" "$_fs2/intel" + : > "$_fs2/iwlwifi/iwlwifi-cc-a0-77.ucode" + _cand="$( d="iwlwifi-cc-*" + _stem="$(printf '%s' "$d" | sed 's/[*?].*//; s/[-_]$//')" + find "$_fs2" -maxdepth 2 -iname "*$_stem*" -printf '%P\n' 2>/dev/null )" + case "$_cand" in + *"iwlwifi/iwlwifi-cc-a0-77.ucode"*) ;; + *) _p="$_p;the search for a near match does not find the real file" ;; + esac + # a failed sparse-checkout must not be swallowed, or the missing firmware + # gets blamed on the pattern + grep -q 'could not narrow the firmware checkout' "$_k" \ + || _p="$_p;a failed sparse-checkout is ignored" + # the same help for a NAME, not only a pattern: `iwlwifi` is a directory + # that does not exist in this release, and it got none + grep -q '_firmware_not_found "$src" "$d"; return 1' "$_k" \ + || _p="$_p;a missing directory gets no near-match search" + [ "$(grep -c '_firmware_not_found "$src" "$d"' "$_k")" -ge 2 ] \ + || _p="$_p;only one of the two cases searches for near matches" + # ASK THE INDEX, NOT THE WORKING TREE. A sparse checkout REMOVES what it + # did not select, so searching the directory finds only what is already + # installed and reports "nothing" for the file being looked for. + grep -q 'ASK THE INDEX, NOT THE WORKING TREE' "$_k" \ + || _p="$_p;the near-match search looks only at the working tree" + grep -q 'git -C "$src" ls-files' "$_k" \ + || _p="$_p;git's index is not consulted" + if command -v git >/dev/null 2>&1; then + _gf="$T/gitfw"; rm -rf "$_gf"; mkdir -p "$_gf" + ( cd "$_gf" && git init -q . \ + && mkdir -p amdgpu intel iwlwifi \ + && : > amdgpu/a.bin && : > intel/ibt.sfi \ + && : > iwlwifi/iwlwifi-cc-a0-77.ucode \ + && git add -A \ + && git -c user.email=t@t -c user.name=t commit -qm x \ + && git sparse-checkout set --no-cone '/amdgpu/' '/intel/' ) >/dev/null 2>&1 + # the fixture must really hide it from the tree, or it proves nothing + [ -e "$_gf/iwlwifi" ] && _p="$_p;the sparse fixture did not hide the directory" + _nf="$( CONF=/etc/pkgusr/kernel.conf + eval "$(_slice_fn "$_k" _firmware_not_found)" + _firmware_not_found "$_gf" iwlwifi )" + case "$_nf" in + *"iwlwifi/iwlwifi-cc-a0-77.ucode"*) ;; + *) _p="$_p;a path the sparse checkout removed is not found: $_nf" ;; + esac + fi + # a plain directory with no repository still gets the find() fallback + _nf2="$( CONF=/etc/pkgusr/kernel.conf + eval "$(_slice_fn "$_k" _firmware_not_found)" + _fs3="$T/fw3"; mkdir -p "$_fs3/amdgpu"; : > "$_fs3/iwlwifi-cc-a0-77.ucode" + _firmware_not_found "$_fs3" iwlwifi )" + case "$_nf2" in + *"iwlwifi-cc-a0-77.ucode"*) ;; + *) _p="$_p;a missing directory does not show the file that is there" ;; + esac + # FIRMWARE TROUBLE MUST NOT COST THE KERNEL. This ran before the build + # and exited, so a wrong FIRMWARE_ONLY meant no kernel was built at all + # -- and nothing said so. + grep -q 'FIRMWARE TROUBLE SHOULD NOT COST YOU THE KERNEL' "$_k" \ + || _p="$_p;a firmware problem still aborts the whole build" + grep -q 'carrying on WITHOUT the firmware' "$_k" \ + || _p="$_p;there is no path that builds the kernel anyway" + grep -q 'firmware was NOT installed' "$_k" \ + || _p="$_p;the summary does not say the firmware is missing" + # ...and it DOES stop when the config embeds firmware + _emb="$T/cfgt"; mkdir -p "$_emb" + printf 'CONFIG_DRM=y\n' > "$_emb/plain" + printf 'CONFIG_EXTRA_FIRMWARE="amdgpu/x.bin"\n' > "$_emb/embed" + printf 'CONFIG_EXTRA_FIRMWARE=""\n' > "$_emb/empty" + grep -q '^CONFIG_EXTRA_FIRMWARE="..*"$' "$_emb/embed" \ + || _p="$_p;a config that embeds firmware is not recognised" + grep -q '^CONFIG_EXTRA_FIRMWARE="..*"$' "$_emb/plain" \ + && _p="$_p;a config with no built-in firmware is treated as embedding some" + grep -q '^CONFIG_EXTRA_FIRMWARE="..*"$' "$_emb/empty" \ + && _p="$_p;an empty EXTRA_FIRMWARE is treated as embedding some" + _q2="$( . "$_kc" 2>/dev/null; printf '%s' "$FIRMWARE_ONLY" )" + case "$_q2" in + *amdgpu*intel*) ;; + *) _p="$_p;the shipped FIRMWARE_ONLY does not survive being sourced: [$_q2]" ;; + esac + # a separate top-level iwlwifi entry would ask for a directory this + # release does not have + case " $_q2 " in + *" iwlwifi "*) _p="$_p;the shipped list asks for a top-level iwlwifi directory" ;; + esac + # ...and an empty value must not silently mean everything + grep -q 'FIRMWARE_ALL' "$_k" \ + || _p="$_p;an empty FIRMWARE_ONLY still installs the whole repository" + grep -q 'probably needs quoting' "$_k" \ + || _p="$_p;the likely cause of an empty value is not named" + # /lib/firmware is the firmware package's own directory: it existed + # root-owned, so every copy failed with Permission denied + grep -q "LIB/FIRMWARE IS THIS PACKAGE'S DIRECTORY" "$_k" \ + || _p="$_p;nothing hands /lib/firmware to the firmware package" + grep -q 'chown "$acct:$acct" /lib/firmware' "$_k" \ + || _p="$_p;the firmware directory is never given to its owner" + grep -q 'intel' "$_kc" || _p="$_p;the shipped config misses Intel bluetooth firmware" + grep -q '_sparse_mode' "$_k" || _p="$_p;file patterns are not supported" + _sp="$( eval "$(_slice_fn "$_k" _sparse_mode)" + eval "$(_slice_fn "$_k" _sparse_args)" + printf 'A[%s][%s] B[%s][%s]' \ + "$(_sparse_mode 'amdgpu intel')" "$(_sparse_args 'amdgpu intel')" \ + "$(_sparse_mode 'amdgpu iwlwifi-cc-*')" "$(_sparse_args 'amdgpu iwlwifi-cc-*')" )" + case "$_sp" in + "A[][amdgpu intel] B[--no-cone][/amdgpu/ iwlwifi-cc-*]") ;; + *) _p="$_p;sparse-checkout arguments are wrong: $_sp" ;; + esac + # a pattern is installed with a glob, a directory by name + grep -q 'which is not in $src' "$_k" \ + || _p="$_p;a pattern matching nothing is not reported" + # the drivers that need the firmware are checked too -- firmware without + # a driver is useless, and this laptop needs three of them + for _o in CONFIG_IWLWIFI CONFIG_IWLMVM CONFIG_BT_HCIBTUSB CONFIG_DRM_AMDGPU; do + grep -q "$_o" "$_k" || _p="$_p;$_o is not checked" + done + # three sources: git, a local copy, or nothing + for _m in 'copy:\*)' 'git)' 'none)'; do + grep -q -- "$_m" "$_k" || _p="$_p;FIRMWARE_FROM has no '$_m' case" + done + # a sparse checkout when only some directories are wanted: the whole + # repository is over ten gigabytes + grep -q 'git sparse-checkout set' "$_k" \ + || _p="$_p;the whole firmware repository is cloned even when a subset is asked for" + grep -q 'filter=blob:none --sparse' "$_k" \ + || _p="$_p;the sparse clone still downloads every blob" + grep -q 'over 10 GB' "$_k" \ + || _p="$_p;an unlimited firmware clone gives no warning about its size" + # it runs BEFORE the built-in firmware check, or the check fires anyway + _l_fw="$(grep -n '_install_firmware ||' "$_k" | head -1 | cut -d: -f1)" + _l_ck="$(grep -n 'FIRMWARE BUILT INTO THE KERNEL' "$_k" | head -1 | cut -d: -f1)" + { [ -n "$_l_fw" ] && [ -n "$_l_ck" ] && [ "$_l_fw" -lt "$_l_ck" ]; } \ + || _p="$_p;firmware is installed after the check that needs it" + # FIRMWARE BUILT INTO THE KERNEL MUST BE ON DISK. A config from a + # distribution kernel lists a dozen blobs in CONFIG_EXTRA_FIRMWARE; an + # LFS tree has no linux-firmware, and the build dies ten minutes in with + # a message naming one file and no reason. + grep -q 'FIRMWARE BUILT INTO THE KERNEL' "$_k" \ + || _p="$_p;built-in firmware is not checked before compiling" + grep -q 'EXTRA_FIRMWARE_DIR' "$_k" \ + || _p="$_p;the firmware directory setting is ignored" + grep -q "scripts/config --set-str EXTRA_FIRMWARE" "$_k" \ + || _p="$_p;no way offered to build without the firmware" + # (the other way out is no longer "go install it yourself": this tool + # installs firmware, so the advice is to list the subdirectory) + grep -q 'add them to FIRMWARE_ONLY' "$_k" \ + || _p="$_p;the fix for a missing built-in blob is not named" + # every missing blob is listed, and a present one is not + _fwd="$T/fwcheck"; rm -rf "$_fwd"; mkdir -p "$_fwd/lib/firmware/amdgpu" + printf 'CONFIG_EXTRA_FIRMWARE="amdgpu/a.bin amdgpu/b.bin i915/c.bin"\nCONFIG_EXTRA_FIRMWARE_DIR="%s/lib/firmware"\n' \ + "$_fwd" > "$_fwd/.config" + : > "$_fwd/lib/firmware/amdgpu/b.bin" + _mf="$( tree="$_fwd" + _fw="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' "$tree/.config")" + _fwdir="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE_DIR="\(.*\)"$/\1/p' "$tree/.config")" + : "${_fwdir:=/lib/firmware}" + for _f in $_fw; do [ -f "$_fwdir/$_f" ] || printf '%s ' "$_f"; done )" + case "$_mf" in + *"amdgpu/a.bin"*) ;; + *) _p="$_p;a missing firmware blob is not detected" ;; + esac + case "$_mf" in + *"i915/c.bin"*) ;; + *) _p="$_p;only the first missing blob is reported" ;; + esac + case "$_mf" in + *"amdgpu/b.bin"*) _p="$_p;a firmware blob that IS present is reported missing" ;; + esac + # a config with no CONFIG_EXTRA_FIRMWARE must not trip it + printf 'CONFIG_DRM=y\n' > "$_fwd/plain.config" + _mf2="$(sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' "$_fwd/plain.config")" + [ -z "$_mf2" ] || _p="$_p;a config without built-in firmware is misread" + grep -q 'proc/config.gz' "$_k" || _p="$_p;the running config cannot be reused" + # BUILDING IN A CHROOT IS FINE: the chroot usually IS the tree that + # boots, /proc is shared so /proc/config.gz is the running config, and + # /lib/modules is where it should be. An earlier version refused + # outright, which was wrong -- what can go astray is each PATH, so the + # step lists those and continues. + grep -q 'BUILDING IN A CHROOT IS FINE' "$_k" \ + || _p="$_p;the kernel step refuses to run in a chroot" + _kg="$(sed -n '/BUILDING IN A CHROOT IS FINE/,/^fi$/p' "$_k")" + printf '%s' "$_kg" | grep -q 'exit 1' \ + && _p="$_p;the chroot case is still fatal" + printf '%s' "$_kg" | grep -q 'lib/modules' \ + || _p="$_p;it does not say which paths to verify" + printf '%s' "$_kg" | grep -q 'CONFIG_FROM=file:' \ + || _p="$_p;it does not offer another config source" + # an absent ESP is refused rather than created on the root filesystem + grep -q 'firmware cannot read' "$_k" \ + || _p="$_p;an unmounted ESP would be created as a plain directory" + # ...and so is a directory of the right name on the WRONG filesystem -- + # every copy succeeds and the firmware finds nothing at the next boot + grep -q 'stat -f -c %T' "$_k" \ + || _p="$_p;the ESP filesystem is not checked" + _fs="$( for fs in vfat ext2/ext3 ""; do + case "$fs" in + msdos|vfat|fat*|exfat) echo "$fs:ok" ;; + "") echo "unknown:warn" ;; + *) echo "$fs:refuse" ;; + esac + done )" + case "$_fs" in + *"vfat:ok"*) ;; + *) _p="$_p;a real EFI partition is not accepted" ;; + esac + case "$_fs" in + *"ext2/ext3:refuse"*) ;; + *) _p="$_p;an ESP on the root filesystem is not refused" ;; + esac + case "$_fs" in + *"unknown:warn"*) ;; + *) _p="$_p;an unidentifiable filesystem is fatal rather than a warning" ;; esac - fi - rm -rf "$_pt" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "wlroots gets its DRM backend, and the kernel has a step of its own" fi -# the sanity report has to skip the same thing, or it reports what verify does not -san="$(dirname "$LFS_TOOL")/lfs-sanity.sh" -if [ -f "$san" ]; then - _p="" - grep -q 'PKGROOT/\*/src' "$san" \ - || _p="$_p;the sanity report still walks the package build trees" - # a config step writing root-owned files is not an orphaned manifest - grep -q 'writes ROOT-owned files' "$san" \ - || _p="$_p;root-owned config files are still reported as missing an account" - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the sanity report skips scratch and does not invent findings" - fi +# ---- one file per round is a loop, not progress (1.14.116) ----------------- # +# libdisplay-info went TWELVE rounds, each one re-cloning a subproject and +# handing over exactly one more file: +# /usr/share/locale/fr/.../v4l-utils.mo belongs to p_v4l-utils -- handing it over +# ... round 7 ... pt_BR/v4l-utils.mo ... round 8 ... libv4lconvert.so ... +# It builds v4l-utils as a meson SUBPROJECT for its edid-decode comparison +# tests, and installs that package's real files over p_v4l-utils's. meson +# stops at the first file it may not replace, so its log names only that one +# and each round learns exactly one more -- the handover can never catch up. +_p="" +_hlpU2="$helper_src" +_pmSP="$(dirname "$LFS_TOOL")/packagemanager" +_hs="$(_slice_fn "$_hlpU2" claim_installed_over_from_log)" +[ -n "$_hs" ] || _hs="$(sed -n '/the installer owns what it installs/,/^}/p' "$_hlpU2")" +printf '%s' "$_hs" | grep -q 'LAST_SINGLE_HANDOVER' \ + || _p="$_p;a single-file handover is not distinguished from a real one" +_rb="$(sed -n '/ONE FILE PER ROUND IS A LOOP/,/took -ge 1/p' "$_hlpU2")" +[ -n "$_rb" ] || _p="$_p;the single-file case is not explained" +_cb2="$(_slice_fn "$_hlpU2" cmd_build)" +printf '%s' "$_cb2" | grep -q '_same_owner_rounds' \ + || _p="$_p;the retry loop does not notice repeated single handovers" +printf '%s' "$_cb2" | grep -q 'BUNDLED COPY' \ + || _p="$_p;the loop is not diagnosed as a bundled subproject" +printf '%s' "$_cb2" | grep -q 'wrap_mode = nodownload' \ + || _p="$_p;the fix that stops the subproject is not named" +printf '%s' "$_cb2" | grep -q '_same_owner_rounds:-0}" -ge 3' \ + || _p="$_p;the loop is not stopped after a few rounds" +# and the fix itself ships: nofallback (the book's) blocks dependency +# fallbacks only -- an explicit subproject() still downloads +_mc2="$(dirname "$LFS_TOOL")/stacks/machine.conf" +grep -q '^\[libdisplay-info\]' "$_mc2" \ + || _p="$_p;libdisplay-info has no wrap_mode setting" +sed -n '/^\[libdisplay-info\]/,/^$/p' "$_mc2" | grep -q 'wrap_mode = nodownload' \ + || _p="$_p;the wrap mode is not nodownload" +# it must reach the meson line, replacing the book's --wrap-mode +python3 - "$_pmSP" "$_mc2" "$T/wrapmode" <<'PYWRAP' || _p="$_p;wrap_mode does not replace the book's value" +import sys, os, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[3]; os.makedirs(d, exist_ok=True) +p = os.path.join(d, "install_ldi") +open(p, "w").write("build_pkg() {\nmkdir build &&\ncd build &&\n" + "meson setup --wrap-mode=nofallback --prefix=/usr .. &&\nninja\n}\n") +pm._machine_opts_for = lambda a: pm._read_conf_section(sys.argv[2], "libdisplay-info") +pm.apply_machine_opts("libdisplay-info", p) +line = [l for l in open(p) if "meson setup" in l][0] +assert "--wrap-mode=nodownload" in line, line +assert "nofallback" not in line, line +PYWRAP +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package installing over another one file at a time is diagnosed, not looped" fi -# ---- verify must not work in silence, or one fork at a time ----------------- # -# `lfs-helper verify` ran for over five minutes printing nothing, on a tree it -# had already verified clean. Two separate faults: -# -# * a fork per path, twice over -- `stat -c %U` and a process substitution -# inside is_never_claimed -- across 66265 manifest paths -# * no output at all until every pass had finished -# -# A command that works for minutes in silence is indistinguishable from one that -# has hung, and was reported as one. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - # the never-claim list is read once, not per call - _slice_fn "$helper_src" is_never_claimed | grep -q '< <(never_claim_list)' \ - && _p="$_p;is_never_claimed forks for every path it is asked about" - _slice_fn "$helper_src" _never_claim_load | grep -q . \ - || _p="$_p;the never-claim list is not cached" - # owners come from one batched stat, not one per path - case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in - *'xargs -0 -r stat'*) ;; - *) _p="$_p;the ownership pass still runs one stat per path" ;; - esac - # a REAL tab: stat does not expand \t, so '%U\tname' comes back as one field - # and the map silently stays empty -- correct but slow, the hardest wrong to see - # comments stripped: the fix NAMES the broken format while explaining it - case "$(_slice_fn "$helper_src" _vfy_manifest_ownership \ - | grep -vE '^[[:space:]]*#')" in - *'%U\t%n'*) _p="$_p;the stat format uses a literal backslash-t" ;; - esac - # and it must say what it is doing, before it does it - _nd="$(_slice_fn "$helper_src" cmd_verify | grep -c 'detail "# ')" - [ "${_nd:-0}" -ge 5 ] \ - || _p="$_p;verify announces only $_nd of its passes" - case "$(_slice_fn "$helper_src" _vfy_manifest_ownership)" in - *'[%d/%d]'*) ;; - *) _p="$_p;the long pass shows no progress" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "verify says what it is doing, and does it in one pass not 66000" - fi +# ---- giving the files back (1.14.117) -------------------------------------- # +# The 1.14.116 diagnosis fired exactly as designed ("rounds run: 3", named +# wrap_mode=nodownload) -- and by then /usr/bin/dvbv5-scan and friends were +# owned by p_libdisplay-info while still listed in p_v4l-utils's manifest. +# "we will sometimes get that problem that we have to overwrite files,... idk +# how to handle that" -- so: the handover is right for packages that +# legitimately overlap, wrong for a bundled copy, and after the wrong case +# there has to be a way back. Only root can chown, and only the manifest +# knows which files were the victim's. +_p="" +_pmRC="$(dirname "$LFS_TOOL")/packagemanager" +python3 "$_pmRC" reclaim --help >/dev/null 2>&1 \ + || _p="$_p;there is no way to give a package its files back" +python3 "$_pmRC" reclaim --help 2>&1 | grep -q -- '--run' \ + || _p="$_p;reclaim has no --run, so it cannot be a dry run by default" +_rcs="$(sed -n '/^def cmd_reclaim/,/^def /p' "$_pmRC")" +printf '%s' "$_rcs" | grep -q 'os.geteuid() != 0' \ + || _p="$_p;reclaim does not require root" +printf '%s' "$_rcs" | grep -q 'pkg.lst' \ + || _p="$_p;reclaim does not use the manifest" +printf '%s' "$_rcs" | grep -q 'os.lchown' \ + || _p="$_p;reclaim follows symlinks when changing ownership" +printf '%s' "$_rcs" | grep -q 'Dry run' \ + || _p="$_p;reclaim changes ownership without being asked" +# it really returns only the files that were taken +if [ "$(id -u)" = 0 ] && command -v useradd >/dev/null 2>&1; then + _rd="$T/reclaim"; rm -rf "$_rd"; mkdir -p "$_rd/home" + groupadd reclaimdemo 2>/dev/null + useradd -g reclaimdemo reclaimdemo 2>/dev/null + printf '%s\n%s\n%s\n' "$_rd/mine" "$_rd/taken" "$_rd/gone" > "$_rd/home/pkg.lst" + : > "$_rd/mine"; : > "$_rd/taken" + chown reclaimdemo:reclaimdemo "$_rd/mine"; chown root:root "$_rd/taken" + python3 - "$_pmRC" "$_rd" <<'PYRC' || _p="$_p;reclaim does not return a taken file" +import sys, os, pwd, importlib.machinery as m +pm = m.SourceFileLoader('pm', sys.argv[1]).load_module() +d = sys.argv[2] +pm.pkgusr_home = lambda a: d + "/home" +pm.pkgusr_name = lambda n: "reclaimdemo" +class A: packages = ["reclaimdemo"]; run = False +pm.cmd_reclaim(A()) +# dry run changed nothing +assert pwd.getpwuid(os.stat(d + "/taken").st_uid).pw_name == "root" +class B: packages = ["reclaimdemo"]; run = True +pm.cmd_reclaim(B()) +assert pwd.getpwuid(os.stat(d + "/taken").st_uid).pw_name == "reclaimdemo" +# a file that was already ours is untouched, a missing one is skipped +assert pwd.getpwuid(os.stat(d + "/mine").st_uid).pw_name == "reclaimdemo" +assert not os.path.exists(d + "/gone") +PYRC + userdel reclaimdemo 2>/dev/null; groupdel reclaimdemo 2>/dev/null +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a package can be given back the files another package took" +fi - # the batched lookup must actually map something -- an empty map is the bug - _pf="$T/statmap"; mkdir -p "$_pf" - touch "$_pf/a" "$_pf/b" "$_pf/c" - printf '%s\n' "$_pf/a" "$_pf/b" "$_pf/c" > "$_pf/man" - _n="$( _TAB=$'\t' - declare -A own=() - while IFS="$_TAB" read -r o nm; do - [ -n "$nm" ] && own["$nm"]="$o" - done < <(grep -v '^[[:space:]]*$' "$_pf/man" \ - | tr '\n' '\0' | xargs -0 -r stat -c "%U${_TAB}%n" 2>/dev/null) - echo "${#own[@]}" )" - [ "$_n" = 3 ] \ - && ok "the batched owner lookup maps every path it is given" \ - || bad "the batched owner lookup mapped $_n of 3 paths" - rm -rf "$_pf" +# ---- the refresh list came from a literal (1.14.122) ----------------------- # +# "lfs run commmand is not copying that new script i think" Correct. +# TOOLCHAIN_SCRIPTS was a hardcoded tuple of five names; lfs-kernel was added +# to the Makefile and not to it, so `lfs run` refreshed five of six tools -- +# and the missing one was the tool just written. Two lists of the same thing +# is the fault; the Makefile's TOOLS line is the one that decides what gets +# installed, so it decides what gets refreshed. +_p="" +_lfsT="$LFS_TOOL" +_mkT="$(dirname "$LFS_TOOL")/Makefile" +grep -q '^def _toolchain_scripts' "$_lfsT" \ + || _p="$_p;the refresh list is still a literal" +grep -q 'TOOLS' "$_lfsT" \ + || _p="$_p;the refresh does not read the Makefile's tool list" +python3 - "$_lfsT" "$_mkT" <<'PYTOOLS' || _p="$_p;the refreshed tools do not match what make install places" +import sys, re, importlib.machinery as m +l = m.SourceFileLoader('lfs', sys.argv[1]).load_module() +got = set(l.TOOLCHAIN_SCRIPTS) +want = set() +for line in open(sys.argv[2]): + if re.match(r"^TOOLS\s*:?=", line): + want = set(line.split("=", 1)[1].split()) + break +assert want, "no TOOLS line in the Makefile" +assert got == want, ("refreshed", sorted(got), "installed", sorted(want)) +assert "lfs-kernel" in got, sorted(got) +PYTOOLS +# a tool with no Makefile beside it still gets a usable fallback +python3 - "$_lfsT" "$T/nomk" <<'PYFALL' || _p="$_p;there is no fallback without a Makefile" +import sys, os, shutil, importlib.machinery as m +d = sys.argv[2]; os.makedirs(d, exist_ok=True) +shutil.copy2(sys.argv[1], os.path.join(d, "lfs")) +l = m.SourceFileLoader('lfs', os.path.join(d, "lfs")).load_module() +assert "lfs-kernel" in l.TOOLCHAIN_SCRIPTS, l.TOOLCHAIN_SCRIPTS +assert "packagemanager" in l.TOOLCHAIN_SCRIPTS, l.TOOLCHAIN_SCRIPTS +PYFALL +# kernel.conf must travel too: lfs-kernel refuses to run without it, so +# refreshing the tool alone puts a command in the chroot that cannot start +grep -q 'kernel.conf lives beside the stacks' "$_lfsT" \ + || _p="$_p;kernel.conf is not refreshed into the chroot" +_kr="$(sed -n '/kernel.conf lives beside the stacks/,/_stack_dst = /p' "$_lfsT")" +printf '%s' "$_kr" | grep -q '".new"' \ + || _p="$_p;an edited kernel.conf in the chroot would be overwritten" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "everything make install places is refreshed into the chroot" fi -# ---- the build must not end quietly with no way to log in ------------------- # -# init-accounts asks for a root password, but it is skippable: a scripted run -# has no terminal, and an interactive one can be answered with a blank line. -# So the LAST thing build-all says is whether anyone can actually log in -- -# because it is the last moment it can still be fixed. After the reboot the -# chroot is gone, and the way back is booting the host and mounting the tree. -helper_src="$(dirname "$LFS_TOOL")/lfs-helper" -if [ -f "$helper_src" ]; then - _p="" - _slice_fn "$helper_src" _warn_if_no_login | grep -q . \ - || _p="$_p;nothing checks whether the system can be logged into" - # it must be the LAST thing build-all does - _ba="$(_slice_fn "$helper_src" cmd_build_all)" - case "$_ba" in - *_warn_if_no_login*) ;; - *) _p="$_p;build-all can finish without saying whether you can log in" ;; - esac - _last="$(printf '%s\n' "$_ba" | grep -vE '^[[:space:]]*(#|$)' | tail -3)" - case "$_last" in - *_warn_if_no_login*) ;; - *) _p="$_p;the login check is not the last thing build-all does" ;; - esac - # it must check the RESULT, not that the step ran - case "$(_slice_fn "$helper_src" _warn_if_no_login)" in - *shadow*) ;; - *) _p="$_p;it trusts init-accounts instead of checking for a password" ;; - esac - if [ -n "$_p" ]; then - printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "the build's last word is whether you can log in afterwards" - fi +# ---- input devices come from udev (1.14.125) ------------------------------- # +# [wlr] libinput initialization failed, no input devices +# [sway] Failed to start backend +# The DRM error from 1.14.115 is gone -- wlroots opens the card now -- and +# libinput found nothing to listen to. It enumerates through libudev, which +# reads /run/udev, so a chroot without the host's /run has NO input devices +# and the message names libinput rather than the chroot. "maybe sway would +# start if i wouldnt be in a chroot": yes, and the launcher should say so. +_p="" +_sw="$(dirname "$LFS_TOOL")/stacks/sway-session.sh" +[ -f "$_sw" ] || _p="$_p;there is no sway-session step" +if [ -f "$_sw" ]; then + bash -n "$_sw" || _p="$_p;sway-session.sh does not parse" + grep -q 'INPUT DEVICES COME FROM UDEV' "$_sw" \ + || _p="$_p;a missing udev database is not diagnosed" + grep -q 'WLR_LIBINPUT_NO_DEVICES=1' "$_sw" \ + || _p="$_p;no way offered to test without input devices" + grep -q 'bind-mount /run and /dev' "$_sw" \ + || _p="$_p;it does not say how to make a chroot work" + # the VT: a compositor started from a tty other than the active one has + # no console to draw on + grep -q 'sys/class/tty/tty0/active' "$_sw" \ + || _p="$_p;starting sway from an inactive tty is not caught" + # the seat wrapper is found on PATH, not at a guessed /bin -- mytools + # installs it in /usr/bin, and the check looked in /bin + grep -q 'command -v "seatd-$(id -un)"' "$_sw" \ + || _p="$_p;the seat wrapper is not looked up on PATH" + grep -q '"/bin/seatd-$(id -un)"' "$_sw" \ + && _p="$_p;the wrapper path is still hardcoded to /bin" + grep -q 'LIBSEAT_BACKEND=seatd' "$_sw" \ + || _p="$_p;libseat is left to pick a backend" + # the launcher it writes must itself be valid, and must run its checks + # before exec'ing anything + python3 - "$_sw" "$T/launcher.sh" <<'PYLAUNCH' || _p="$_p;no launcher is generated" +import sys, re +s = open(sys.argv[1]).read() +m = re.search(r"cat > /usr/bin/sway-session <<'LAUNCH'\n(.*?)\nLAUNCH\n", s, re.S) +assert m, "no launcher heredoc" +body = m.group(1) +open(sys.argv[2], "w").write(body) +# every check must come before the exec, or it runs too late to matter +assert body.index("/run/udev") < body.index("exec dbus-run-session"), "check after exec" +assert body.index("seatd-$(id -un)") < body.index("exec dbus-run-session"), "seat check after exec" +PYLAUNCH + bash -n "$T/launcher.sh" || _p="$_p;the generated sway-session does not parse" + # ...and it really refuses, with the reason, when udev is absent + sed 's|/run/udev|/nonexistent-udev-xyz|' "$T/launcher.sh" > "$T/l2.sh" + mkdir -p "$T/fakebin"; ln -sf /bin/true "$T/fakebin/seatd-$(id -un)" + _o="$(PATH="$T/fakebin:$PATH" bash "$T/l2.sh" 2>&1)"; _rc=$? + [ "$_rc" != 0 ] || _p="$_p;a missing udev database does not stop the launch" + case "$_o" in + *"will find no input"*) ;; + *) _p="$_p;the refusal does not explain libinput" ;; + esac +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "sway-session explains a chroot's missing input devices before failing" +fi - # all three outcomes, against a real shadow file - _lg="$T/login"; mkdir -p "$_lg" - _run_login() { # <shadow contents> -> "ok" or "warn" - ( ETC="$_lg"; say(){ :; }; ok(){ :; }; warn(){ echo warn; } - printf 'root:x:0:0::/root:/bin/bash\nbob:x:1000:1000::/home/bob:/bin/bash\n' > "$_lg/passwd" - printf '%s' "$1" > "$_lg/shadow" - eval "$(_slice_fn "$helper_src" _warn_if_no_login)" - _warn_if_no_login | head -1 ) 2>/dev/null - } - _bad="" - [ -z "$(_run_login 'root:$6$a:1:::::: -bob:!:1::::::')" ] || _bad="$_bad;a root password is not accepted" - [ -z "$(_run_login 'root:*:1:::::: -bob:$6$b:1::::::')" ] || _bad="$_bad;a user password is not accepted" - [ -n "$(_run_login 'root:*:1:::::: -bob:!:1::::::')" ] || _bad="$_bad;a system nobody can log into is not reported" - if [ -n "$_bad" ]; then - printf '%s\n' "${_bad#;}" | tr ';' '\n' | while IFS= read -r m; do - [ -n "$m" ] && bad "$m" - done - else - ok "a locked-out system is reported; a usable one is not" - fi - rm -rf "$_lg" +# ---- the home is not installed files (1.14.138) ---------------------------- # +# # recorded 266621 file(s) for p_linux +# That is the kernel SOURCE TREE. list_package lists everything an account +# owns, its own home included -- sources, build trees, the install script. A +# manifest is what the package put ON THE SYSTEM: it is what `verify` checks +# and what `remove` deletes, and 266k source files in it make both +# meaningless. The build's own tracking always excluded the home (it +# compares snapshots outside it); the rescan now does the same. +_p="" +_pmH2="$(dirname "$LFS_TOOL")/packagemanager" +_wl2="$(sed -n '/^def write_pkg_list/,/^def /p' "$_pmH2")" +printf '%s' "$_wl2" | grep -q 'THE HOME IS NOT INSTALLED FILES' \ + || _p="$_p;the manifest still records the account's own home" +printf '%s' "$_wl2" | grep -q 'awk -v h=' \ + || _p="$_p;there is no filter on the listing" +# awk, not grep: a filter that removes every line must still exit 0, or the +# && mv never runs and the old manifest silently survives +printf '%s' "$_wl2" | grep -q 'grep -v' \ + && _p="$_p;a grep filter would exit 1 when it removes everything" +_flt="awk -v h=/usr/src/pkgusr/p_linux/ -v m=/var/mail/ -v s=/var/spool/mail/ 'index(\$0,h)!=1 && index(\$0,m)!=1 && index(\$0,s)!=1'" +_kept="$(printf '%s\n' /usr/bin/foo \ + /usr/src/pkgusr/p_linux/src/linux-7.2.4/Makefile \ + /lib/modules/7.2.4/modules.dep \ + /var/mail/p_linux \ + /usr/include/linux/version.h \ + | sh -c "$_flt")" +case "$_kept" in + *"/usr/bin/foo"*) ;; + *) _p="$_p;an installed file was filtered out" ;; +esac +case "$_kept" in + *"linux-7.2.4/Makefile"*) _p="$_p;the source tree is still recorded" ;; +esac +case "$_kept" in + *"/var/mail/"*) _p="$_p;the mail spool is still recorded" ;; +esac +case "$_kept" in + *"modules.dep"*) ;; + *) _p="$_p;an installed module index was filtered out" ;; +esac +_rc="$(printf '/usr/src/pkgusr/p_linux/x\n' | sh -c "$_flt" >/dev/null; echo $?)" +[ "$_rc" = 0 ] || _p="$_p;a filter that removes every line exits non-zero" + +# ...AND NOT THROUGH A SYMLINK INTO IT. Excluding the home was not enough: +# /lib/modules/<ver>/build is a SYMLINK to the source tree, the scan follows +# it, and the same quarter-million files come back as +# /lib/modules/<ver>/build/... -- outside the home, so the home filter never +# saw them and the "includes its own home" check stayed silent about 266623 +# entries. +printf '%s' "$_wl2" | grep -q 'NOT THROUGH A SYMLINK INTO IT' \ + || _p="$_p;the manifest still follows the build symlink into the sources" +# the home is matched ANYWHERE in the line: a prefix match assumed every +# entry is an absolute path starting exactly at the home, and the manifest +# stayed at 268k with the filter in place +_flt2="awk -v h=/usr/src/pkgusr/p_linux/ -v m=/var/mail/ -v s=/var/spool/mail/ 'index(\$0,h) { next } index(\$0,m)==1 { next } index(\$0,s)==1 { next } /\/lib\/modules\/[^\/]+\/build\// { next } /\/lib\/modules\/[^\/]+\/source\// { next } { print }'" +_kept2="$(printf '%s\n' /usr/bin/foo \ + ./usr/src/pkgusr/p_linux/src/relative \ + /lib/modules/7.2.4/modules.dep \ + /lib/modules/7.2.4/build \ + /lib/modules/7.2.4/build/Makefile \ + /lib/modules/7.2.4/source/Makefile \ + /lib/modules/7.2.4/kernel/fs/ext4/ext4.ko \ + | sh -c "$_flt2")" +case "$_kept2" in + *"build/Makefile"*) _p="$_p;files behind the build symlink are still recorded" ;; +esac +case "$_kept2" in + *"relative"*) _p="$_p;a home path that is not at the start of the line slips through" ;; +esac +case "$_kept2" in + *"source/Makefile"*) _p="$_p;files behind the source symlink are still recorded" ;; +esac +# the symlink ITSELF is an installed file and must stay +case "$_kept2" in + *"/lib/modules/7.2.4/build"*) ;; + *) _p="$_p;the build symlink itself was filtered out" ;; +esac +case "$_kept2" in + *"ext4.ko"*) ;; + *) _p="$_p;a real module was filtered out" ;; +esac +# and verify reports a manifest that was scanned through it +# (own path: $_vb belongs to another block, and a block must not depend on +# one another block happened to set -- 1.14.106 again) +grep -q 'scanned THROUGH' "$(dirname "$LFS_TOOL")/lfs-kernel" \ + || _p="$_p;verify does not notice a manifest scanned through the symlink" +_mm="$T/manifests"; mkdir -p "$_mm" +printf '/lib/modules/7.2.4/build/Makefile\n/usr/bin/x\n' > "$_mm/bad" +printf '/lib/modules/7.2.4/build\n/usr/bin/x\n' > "$_mm/good" +grep -qE "^/lib/modules/[^/]+/(build|source)/." "$_mm/bad" \ + || _p="$_p;a manifest full of source paths is not detected" +grep -qE "^/lib/modules/[^/]+/(build|source)/." "$_mm/good" \ + && _p="$_p;a manifest with just the symlink is wrongly flagged" + +# THE REPOSITORY IS THE PACKAGE USER'S FROM THE FIRST COMMAND. git pull ran +# as root and the chown came after, so the next command -- as the package +# user -- met a repository owned by somebody else: +# fatal: detected dubious ownership in repository at '.../v4l2loopback' +_k2="$(dirname "$LFS_TOOL")/lfs-kernel" +grep -q "THE REPOSITORY IS THE PACKAGE USER'S" "$_k2" \ + || _p="$_p;the module repository is not owned before it is used" +_v4="$(sed -n '/v4l2loopback (optional/,/depmod -a/p' "$_k2")" +printf '%s' "$_v4" | grep -q 'su - "$acct" -c "git clone' \ + || _p="$_p;the clone still runs as root" +printf '%s' "$_v4" | grep -q 'su - "$acct" -c "cd .\$d. && git pull' \ + || _p="$_p;the pull still runs as root" +# (matched on CODE, not on the comment that mentions why safe.directory is +# the wrong answer -- a grep over source hits prose too) +printf '%s' "$_v4" | grep -vE '^\s*#' | grep -q 'safe.directory' \ + && _p="$_p;the ownership problem is papered over with safe.directory" +# the chown must come BEFORE any git command, not after +_v4c="$(printf '%s' "$_v4" | grep -vE '^\s*#')" +_l_own="$(printf '%s' "$_v4c" | grep -n 'chown -R' | head -1 | cut -d: -f1)" +_l_git="$(printf '%s' "$_v4c" | grep -n 'git pull' | head -1 | cut -d: -f1)" +{ [ -n "$_l_own" ] && [ -n "$_l_git" ] && [ "$_l_own" -lt "$_l_git" ]; } \ + || _p="$_p;ownership is still fixed after the first git command" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a manifest lists installed files, and a repository is owned before use" +fi + +# ---- say it before it takes a while (1.14.141) ------------------------------ # +# "after that output nothing happens and i do not know whats going on,.. +# please always provide output before we get a long loading process." The +# Documentation copy is ~10k small files and the manifest scan walks the +# filesystem; both ran in complete silence for minutes. A tool that goes +# quiet is indistinguishable from one that has hung. +_p="" +_k3="$(dirname "$LFS_TOOL")/lfs-kernel" +grep -q 'SAY IT BEFORE IT TAKES A WHILE' "$_k3" \ + || _p="$_p;the long steps still run silently" +# every slow step announces itself FIRST +# ONLY A KERNEL THAT HAS BOOTED IS KNOWN TO BOOT. The old fallback wrote +# the version just built into last-good whenever the file was empty, so +# verify reported "last kernel known to boot: 7.2.4" about a kernel that had +# never run -- while 7.2.0 was carrying the machine. A record that holds a +# guess is worse than an empty one: the file exists to name something you +# can go back to. +grep -q 'ONLY A KERNEL THAT HAS BOOTED IS KNOWN TO BOOT' "$_k3" \ + || _p="$_p;an unbooted kernel can still be recorded as known-good" +_lgb="$(sed -n '/ONLY A KERNEL THAT HAS BOOTED/,/^fi$/p' "$_k3")" +printf '%s' "$_lgb" | grep -q '\[ "$_running" = "$VERSION" \]' \ + || _p="$_p;the running version is not compared with the built one" +printf '%s' "$_lgb" | grep -q 'nothing is recorded as known-good' \ + || _p="$_p;the case where nothing has booted is not handled" +printf '%s' "$_lgb" | grep -qE 'elif \[ ! -s "\$_lastgood" \]' \ + && _p="$_p;the guessing fallback is still there" +# the three cases +_lg3="$( for c in "7.2.4 7.2.4 no" "7.2.0 7.2.4 yes" "7.2.0 7.2.4 no"; do + set -- $c + if [ "$1" = "$2" ]; then printf 'built ' + elif [ "$3" = yes ]; then printf 'running ' + else printf 'none '; fi + done )" +[ "$_lg3" = "built running none " ] \ + || _p="$_p;the known-good decision is wrong: [$_lg3]" +# and verify explains an empty record rather than omitting the line +grep -q 'no kernel confirmed to boot yet' "$_k3" \ + || _p="$_p;verify is silent when nothing is known to boot" +grep -q 'it is running now' "$_k3" \ + || _p="$_p;verify does not say when the known-good kernel is the live one" + +# A CONFIG FROM ONE KERNEL IS NEVER COMPLETE FOR ANOTHER. "do we do an +# 'make old_config' when installing a new version?" It did -- olddefconfig, +# undocumented and not selectable, so every new symbol in a newer kernel was +# answered by upstream's default without a word. +# (own variable: $_kc3 is assigned further down this block, and a check +# must not depend on an assignment that follows it) +_kcR="$(dirname "$LFS_TOOL")/kernel.conf" +grep -q 'CONFIG_REFRESH:=olddefconfig' "$_k3" \ + || _p="$_p;the config refresh cannot be chosen" +grep -q '^CONFIG_REFRESH=olddefconfig$' "$_kcR" \ + || _p="$_p;the config does not document the refresh" +for _m in olddefconfig oldconfig menuconfig none; do + grep -q -- "$_m" "$_kcR" || _p="$_p;$_m is not documented as an option" +done +grep -q 'not refreshing the config' "$_k3" \ + || _p="$_p;CONFIG_REFRESH=none is not handled" +grep -q 'is not understood' "$_k3" \ + || _p="$_p;an unknown CONFIG_REFRESH is accepted silently" +# it must announce the refresh BEFORE running it, and report what changed +_l_ann="$(grep -n 'refreshing the config for' "$_k3" | head -1 | cut -d: -f1)" +_l_run="$(grep -n '_as_pkg "make \$CONFIG_REFRESH"' "$_k3" | head -1 | cut -d: -f1)" +{ [ -n "$_l_ann" ] && [ -n "$_l_run" ] && [ "$_l_ann" -lt "$_l_run" ]; } \ + || _p="$_p;the config refresh is not announced before it runs" +grep -q 'differ from the config you started from' "$_k3" \ + || _p="$_p;nothing says how many symbols the refresh decided" +# the count itself +_ca="$T/cfga"; _cb="$T/cfgb" +printf 'CONFIG_A=y\nCONFIG_B=y\n' > "$_ca" +printf 'CONFIG_A=y\nCONFIG_B=y\nCONFIG_NEW1=y\nCONFIG_NEW2=m\n' > "$_cb" +_dn="$(diff <(sort "$_ca") <(sort "$_cb") | grep -c '^>')" +[ "$_dn" = 2 ] || _p="$_p;the changed-line count is wrong ($_dn)" + +# THE COMPILE STEP MUST EXIST AT ALL. Two edits to this file collided and +# deleted `make $JOBS` outright -- the tool would have installed modules from +# a tree it never built. A build script's test should assert the build. +grep -q '_as_pkg "make \$JOBS"' "$_k3" \ + || _p="$_p;the compile step is missing" +_l_mk="$(grep -n '_as_pkg "make \$JOBS"' "$_k3" | head -1 | cut -d: -f1)" +_l_mi="$(grep -n '_as_pkg "make modules_install"' "$_k3" | head -1 | cut -d: -f1)" +{ [ -n "$_l_mk" ] && [ -n "$_l_mi" ] && [ "$_l_mk" -lt "$_l_mi" ]; } \ + || _p="$_p;modules are installed before the kernel is compiled" +for _pair in 'copying Documentation:cp -r Documentation' \ + 'compiling:_as_pkg "make \$JOBS"' \ + 'installing modules into:_as_pkg "make modules_install"' \ + 'installing headers into:su - .\$_hacct. -c "cp -r'; do + _say="${_pair%%:*}"; _cmd="${_pair#*:}" + _l_say="$(grep -n "echo \"# $_say" "$_k3" | head -1 | cut -d: -f1)" + _l_cmd="$(grep -n -- "$_cmd" "$_k3" | grep -vE '^[0-9]+:\s*#' | head -1 | cut -d: -f1)" + if [ -z "$_l_say" ]; then + _p="$_p;nothing is printed before '$_say'" + elif [ -n "$_l_cmd" ] && [ "$_l_say" -gt "$_l_cmd" ]; then + _p="$_p;'$_say' is announced after it has already run" + fi +done +# the manifest scan announces itself inside _record, where it happens +_rec="$(sed -n '/^_record() {/,/^}/p' "$_k3")" +printf '%s' "$_rec" | grep -q 'echo "# recording what' \ + || _p="$_p;the manifest scan does not announce itself" +_l_e="$(printf '%s' "$_rec" | grep -n 'echo "# recording' | head -1 | cut -d: -f1)" +_l_r="$(printf '%s' "$_rec" | grep -n 'reload-pkg-list' | head -1 | cut -d: -f1)" +{ [ -n "$_l_e" ] && [ -n "$_l_r" ] && [ "$_l_e" -lt "$_l_r" ]; } \ + || _p="$_p;the scan is announced after it runs" + +# ROOT COPIES THE HEADERS, THEN HANDS THEM OVER. Running the copy as the +# headers account cannot work: the source is inside p_linux's home, 0750, so +# it cannot even stat it -- and opening one account's home to another would +# undo the point of having accounts. +grep -q 'ROOT COPIES, THEN HANDS THE FILES OVER' "$_k3" \ + || _p="$_p;the header copy still runs as the wrong account" +grep -q 'su - "$_hacct" -c "cp -r' "$_k3" \ + && _p="$_p;the headers are still copied across accounts through a 0750 home" +grep -q 'chown -R "$_hacct:$_hacct" /usr/include' "$_k3" \ + || _p="$_p;the installed headers are not given to the headers package" +# verify checks what the CONFIG asked to install, not only what the image +# embeds -- the wireless firmware is loaded at run time and nothing checked it +grep -q 'for _d in ${FIRMWARE_ONLY' "$_k3" \ + || _p="$_p;verify ignores the firmware the config asked for" +grep -q 'intel/iwlwifi' "$_k3" \ + || _p="$_p;verify says nothing about the Intel wireless firmware" +# an implausible manifest shows its own evidence instead of guessing +grep -q 'far too many for a kernel' "$_k3" \ + || _p="$_p;an implausible manifest is only a note, not a failure" +grep -q 'head -3 "$_h/pkg.lst"' "$_k3" \ + || _p="$_p;the manifest failure does not show which paths got in" + +# THE HEADERS BELONG TO THE HEADERS PACKAGE. /usr/include/asm/*.h is owned +# by p_linux-headers, and /usr/include is sticky, so p_linux could not +# replace a single file: +# cp: cannot create regular file '/usr/include/asm/types.h': Permission denied +grep -q 'THE HEADERS BELONG TO THE HEADERS PACKAGE' "$_k3" \ + || _p="$_p;the kernel still tries to install headers as its own package" +grep -q 'HEADERS_PKG:=linux-headers' "$_k3" \ + || _p="$_p;the headers package cannot be named" +# (1.14.142: root copies and hands over, because the source lives inside a +# 0750 home that the headers account cannot read) +grep -q 'chown -R "$_hacct:$_hacct" /usr/include' "$_k3" \ + || _p="$_p;the headers are not given to the account that owns them" +grep -q 'reload-pkg-list "$HEADERS_PKG"' "$_k3" \ + || _p="$_p;the headers package manifest is not refreshed" +grep -q 'headers: skipped' "$_k3" \ + || _p="$_p;the headers cannot be left alone" +_kc3="$(dirname "$LFS_TOOL")/kernel.conf" +grep -q '^HEADERS=yes$' "$_kc3" || _p="$_p;the config does not offer HEADERS" +grep -q '^HEADERS_PKG=linux-headers$' "$_kc3" \ + || _p="$_p;the config does not name the headers package" +# the tree must be readable by that other account, or the copy cannot work +grep -q 'cp -r "$tree/usr/include" /usr' "$_k3" \ + || _p="$_p;the headers are not copied from the tree at all" +# and a genuine ownership clash points at reclaim rather than at nothing +grep -q 'packagemanager reclaim $HEADERS_PKG' "$_k3" \ + || _p="$_p;a header owned by a third package suggests no way out" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "slow steps announce themselves, and headers install as their own package" fi echo