Packagemanager-LFS-PackageUser-System git · main
Package-user based package manager for LFS/BLFS
Shell 63.7% Markdown 35.8%git clone https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System.gitwget https://git.christianimmanuel.de/linux-from-scratch/Packagemanager-LFS-PackageUser-System/archive/Packagemanager-LFS-PackageUser-System.tar.gzAdd Stacks And setup stuff
BUILD.md | 148 +- HANDOFF.md | 3727 +++++++++++++++++++++++++++++++- Makefile | 47 +- NEXT-TASK.md | 142 ++ README.md | 78 +- backup.conf | 132 ++ blfs | 313 ++- kernel.conf | 40 +- lfs | 52 +- lfs-backup | 694 ++++++ lfs-helper | 1777 +++++++++++++++- lfs-kernel | 869 +++++++- lfs-phases | 129 +- packagemanager | 5370 +++++++++++++++++++++++++++++++++++++++++++---- stacks/accounts.conf | 29 + stacks/accounts.sh | 117 ++ stacks/app-users.conf | 115 + stacks/app-users.sh | 655 ++++++ stacks/apps.stack | 94 + stacks/avoid.stack | 170 ++ stacks/bootscripts.conf | 44 + stacks/bootscripts.sh | 199 ++ stacks/dev.stack | 43 + stacks/diag.stack | 112 + stacks/fonts.sh | 149 ++ stacks/iptables-nft.sh | 86 + stacks/kernel-sway.sh | 23 + stacks/machine.conf | 246 ++- stacks/media.stack | 258 +++ stacks/net-wifi.sh | 213 ++ stacks/network.stack | 51 +- stacks/power.sh | 195 ++ stacks/services.stack | 130 +- stacks/setid.conf | 68 + stacks/shadow-pam.sh | 354 ++++ stacks/shell-env.sh | 155 +- stacks/sway-session.sh | 16 +- stacks/sway.stack | 426 ++-- stacks/tools.stack | 54 + stacks/vision.stack | 88 + stacks/wayland.stack | 204 ++ stacks/webrtc.sh | 152 ++ test_lfs_crosschain.sh | 5096 +++++++++++++++++++++++++++++++++++++++++++- 43 files changed, 22083 insertions(+), 977 deletions(-) diff --git a/BUILD.md b/BUILD.md index 0f8dcbf..5234eb8 100644 --- a/BUILD.md +++ b/BUILD.md @@ -80,7 +80,7 @@ Worth setting before `run`: | `locale`, `charmap`, `paper_size`, `timezone` | `en_US.UTF-8`, `UTF-8`, `A4`, – | baked into chapter 9 | | `hostname`, `network` | `lfs`, `dhcp` | `static` writes interface files; `dhcp` expects dhcpcd from BLFS | | `main_user` | – | your login. `init-accounts` creates it (group `users`) and asks for root's and its password. Blank = root only, no password | -| `makeflags` | `-j$(nproc)` | | +| `makeflags` | `-j$(nproc)` — the LFS build's job count | | | `remove_tools` | no | delete `/tools` at the end; it is your way back if chapter 8 needs redoing | | `bootloader`, `esp` | `none`, – | `refind` + `/boot/efi` generates a rEFInd step that only *adds* an entry | | `collector_import_file` | – | groups export from a previous build; skips every sharing question | @@ -169,7 +169,7 @@ offline. The rest needs the network `resolv` just gave you. |---|---| | `/etc/pkgusr/stacks/machine.conf` | this machine's build options (`[mesa] gallium-drivers = radeonsi`, `[iptables] configure_args = --enable-nftables`, `[dbus] systemd = disabled`). Merged into generated scripts when they are born, so set it **before** the stack runs | | `/etc/pkgusr/stacks/package-fixes.conf` | workarounds the tools ship. Replaced on every `lfs run`; put yours in `machine.conf`, which wins | -| `/etc/pkgusr/stacks/*.stack` | which packages, in order; `avoid` lines; pinned git refs | +| `/etc/pkgusr/stacks/*.stack` | which packages, in order; `avoid` lines; pinned git refs; `expect=` feature probes | | `/etc/pkgusr/desktop-user` | the login that owns the desktop. Set once with `LFS_DESKTOP_USER=<name>`; `sway-session` and `mytools` both read the file | Files under `/etc/pkgusr/` are yours once edited. `make install` updates a @@ -178,21 +178,100 @@ copy you never touched, otherwise it lands as `.new` ### The stacks, in order +**Updating part of a system.** A stack run builds what is MISSING; it never +updates an installed package. To move packages to what the book now carries: + + packagemanager update # everything it has moved past + packagemanager update --stack media # just that chain + packagemanager update --stack media --select --run # pick from them + packagemanager update webkitgtk --run # one, deliberately + +`update <pkg>` also reports which of that package's requirements are older +here than the version its page was written against -- not a refusal, just +the numbers. + +The desktop is three stacks that build as one: `wayland` (session, mesa, +wayland, GTK4, wlroots), `media` (the codec / gstreamer / WebRTC / webkitgtk +chain) and `sway` (the compositor, its tools, your own programs). sway.stack +starts with `include wayland` and `include media`, so a full desktop build is +still one command -- and when a codec or webkitgtk needs redoing you rebuild just that +part: + + packagemanager stack media --run + packagemanager probe + +`vision` is separate and optional: OpenCV 5, hours of compiling, nobody's +dependency. `packagemanager stack vision --run` when you want it. + +`diag` is optional too: glxinfo, eglinfo, vulkaninfo, vainfo, nvtop. +GPU vendor (for nvtop) is set at the top of `diag.stack`. +`packagemanager stack diag --run`. + + ```sh usermod -aG audio,video,input <name> # the main_user from the interview export LFS_DESKTOP_USER=<name> packagemanager stack sway --check-refs # every git ref still exists? -packagemanager stack sway --run # desktop: wlroots, sway, foot, webkit, pipewire, mesa +packagemanager stack sway --run # desktop: includes wayland + media, then sway, foot, pipewire +packagemanager stack apps --run # zathura, imv -- programs, not the desktop packagemanager stack network --check-refs packagemanager stack network --run # wpa_supplicant, iptables-nft, net-tools -packagemanager stack services --run # bootscripts, fcron, acpid, mytools, shell-env +packagemanager stack services --run # bootscripts, fcron, acpid, mytools, shell-env, app-users +packagemanager stack dev --run # gdb -- nothing needs it, so it goes last +packagemanager stack tools --run # trans, jq -- same: nothing needs them ``` `sway` includes `check kernel-sway`, which reads the running kernel's config and fails naming the missing option. In the chroot that is the host kernel; section 6 builds the real one. +**How many jobs.** Four build systems, and only one of them reads +`MAKEFLAGS`: + +| what | reads | set it in | +|---|---|---| +| make | `MAKEFLAGS` | `/etc/pkgusr/build.env`: `MAKEFLAGS=-j13` | +| cmake | `CMAKE_BUILD_PARALLEL_LEVEL` | derived from the above, or `--jobs` | +| cargo | `CARGO_BUILD_JOBS` | same | +| ninja | nothing at all | `machine.conf` `[all] jobs = 13` | +| rust (`./x.py`) | `CARGO_BUILD_JOBS` **and** `--jobs N` | same `[all] jobs` — it patches `x.py` and `cargo` lines too | + +A `jobs` value is patched into the script **when the script is generated**, +so a package built before you set it needs `--regenerate` to pick it up. Any +line that already names `-j` or `--jobs` is left alone. + +`lfs-helper build <pkg> --jobs 13` overrides all of them for one build. A +named `machine.conf` section beats `[all]`, which is how webkitgtk keeps its +lower count for RAM. + +**Order matters for autodetected features.** A package built inside another +package's dependency plan is built in the *book's* order, not the stack's — +so an optional library like `alsa-lib` must be listed before anything that +autodetects it, or ffmpeg and SDL3 come out with no sound and no error. An +entry can prove its own features afterwards: + + book ffmpeg expect="ffmpeg -hide_banner -devices | grep -qw alsa" + +A failing probe leaves the entry unrecorded, so the next run rebuilds it. +Probe the artifact that proves the feature: for a GStreamer plugin that is a +registered element (`gst-inspect-1.0 webrtcbin`), not a file — the library is +built either way. + +Two reports cover the chains where this bites: + + packagemanager stack network --only net-wifi --run # wifi + packagemanager stack sway --only webrtc --run # camera, mic, WebRTC + +**After a stack finishes, do the second pass.** Optional features are decided +when a package configures, so anything built before a dependency it can use +was built without it — and some of those cannot be reordered at all (cairo +requires glib2, so it cannot precede it): + + packagemanager probe # do the features exist? + packagemanager late-deps # orderable / circular / cosmetic + packagemanager late-deps --rebuild --run # in dependency order + Useful while a stack runs: ```sh @@ -215,7 +294,9 @@ generated, so it can be edited before the first build. | `ESP`, `KERNEL_NAME` | *which* kernel this is. `vmlinuz_new` lands beside the working one; `vmlinuz` replaces it. Previous file is kept as `_old` | | `JOBS` | leave a core free | | `CONFIG_FROM`, `CONFIG_REFRESH` | `running` + `olddefconfig` for unattended; `oldconfig`/`menuconfig` to decide yourself | -| `FIRMWARE_FROM`, `FIRMWARE_ONLY` | `git` and a short list (`amdgpu amd-ucode amd intel`). Empty means 10 GB | +| `CONFIG_SET` | options this machine always wants (`LEGACY_TIOCSTI=n`), applied after the refresh and read back | +| `FIRMWARE_FROM`, `FIRMWARE_ONLY` | `git` and a short list. A directory name covers a directory; a *driver* asks the loader for a flat path, so Intel wireless needs `iwlwifi-*.ucode` as well as `intel`. Empty means 10 GB | +| `REGDB` | yes. cfg80211's regulatory database is wireless-regdb, not linux-firmware; do **not** name it in `FIRMWARE_ONLY` | | `HEADERS`, `V4L2LOOPBACK`, `BOOTLOGO` | as you like | ```sh @@ -227,6 +308,15 @@ The kernel is a package: `packagemanager remove linux` takes it away. The tree of the last kernel known to boot is never deleted, and nothing is recorded as known-good until it has actually booted. +Firmware is its own package and its own command, so a missing blob does not +cost a kernel build: + +```sh +lfs-kernel firmware # install what FIRMWARE_ONLY names +lfs-kernel firmware --missing # what the running kernel asked for and + # did not get, and where it actually lives +``` + --- ## 7. Bootloader @@ -254,12 +344,49 @@ dmesg | grep -i firmware # the definitive FIRMWARE_ONLY list packagemanager verify ``` +If something that needs root privileges fails (`su -` reporting `setgid: +Operation not permitted`, `passwd` refusing to write), the setuid bit was +refused at install time — a package user may not set one, and a bit it *can* +set makes the program run as the package user rather than as root: + +```sh +lfs-helper setid # what is outstanding, and what is on disk +lfs-helper setid --run # chown root, set the bit, read it back +``` + +If a device is missing, ask what firmware it wants. This works in the chroot +too: the list comes from the tree's own modules and this machine's hardware, +not only from the running boot, and each line says which source named it. + +```sh +lfs-kernel firmware --missing # boot / modules / config, per line +lfs-kernel firmware --missing --fix +``` + +`boot` lines are things a driver actually asked for and did not get — the +reliable list. For each of those it also says whether the same file is +already on disk under another path (a `FIRMWARE_ONLY` entry keeps the +repository's path; the loader wants its own) or whether an older firmware +API of it is installed, which means the checkout needs updating rather than +a different path. `modules` lines are candidates: a module declares firmware for +every chip it supports, and yours loads only a few, so they are summarised +per module rather than reported as failures. `--fix --all` installs them +anyway. + Things a first boot commonly still wants: -- **wifi:** `/etc/sysconfig/ifconfig.wifi0` with `SERVICE=wpa`, and one - file per SSID in `/etc/sysconfig/wpa_supplicant/`. +- **wifi:** `/etc/sysconfig/ifconfig.<iface>` (the real interface name — + `wlp3s0`, not `wifi0`) with `SERVICE=wpa`, and one file per SSID in + `/etc/sysconfig/wpa_supplicant/`, chmod 600. The network stack installs + `dhcpcd`; `packagemanager stack network --only net-wifi --run` reports + whatever is still missing, including which hand-off variable the installed + `wpa` service reads. - **certificates:** `make-ca -g` weekly, as a cron entry in `/etc/cron.weekly/` (the systemd book used a timer). +- **power:** `halt`, `poweroff`, `reboot` and suspend belong to the + `<collector_prefix>_power` group; `cfg power` in the services stack sets + that up, and members must log in again before it counts. Check it with + `/etc/rc.d/init.d/power-perms status`. - **desktop:** log in on the active tty as the desktop user, run `sway`. The seat comes from `/usr/bin/seatd-<user>`, a setuid wrapper `mytools` built for that one name. If it fails, the launcher checks the wrapper, @@ -294,5 +421,12 @@ start again from section 2 with `collector_import_file` set. | a chroot step fails | `/usr/src/lfs-pkgusr/logs/<step>.log`; `lfs-helper check-toolchain` | | a stack entry fails | `packagemanager errors`; `--only <entry> --run` to retry one | | files owned by nobody | `lfs-helper fix-orphans --run`, `packagemanager files_with_broken_id` | +| a feature is missing from a package that built fine | `packagemanager late-deps` — it was built before that dependency existed; `--rebuild --run` fixes it | +| a package cannot install into its own directory | check who owns the parent; `packagemanager reclaim <pkg> --run` | | ownership drifted | `packagemanager verify --fix`; `lfs-helper fix-ownership --run` | +| `su`/`passwd`/`mount` say "Operation not permitted" | `lfs-helper setid` — a refused setuid bit | +| a device has no driver | `lfs-kernel firmware --missing` | | tools and scripts disagree | `lfs build-system gen-chroot-scripts --run --overwrite` (host) | + +The regression suite runs in pieces: `bash test_lfs_crosschain.sh --list`, +then `--from N --to M` or `--only <text>`. diff --git a/HANDOFF.md b/HANDOFF.md index beafb07..360f94c 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -7,23 +7,25 @@ account per package, so every file on the system has an owner, and Written for a **SysV** system (no systemd) built from the **systemd** BLFS book, because BLFS stopped publishing a SysV edition after 12.4. - all five tools at 1.14.146 - lfs c0858da - lfs-helper bf6756c - packagemanager 367a690 - blfs 877e4cb - lfs-phases ba72201 - lfs-kernel 1aa3396 (no version line of its own) + all six tools at 1.14.321 (lfs-backup has its own: 1.3.0) + lfs cb920be + lfs-helper 3422951 + packagemanager 3517a41 + blfs f8ce360 + lfs-phases 2c5f6a3 + lfs-kernel 219215c (no version line of its own) + lfs-backup c73acaa (1.3.0; replaces lfs-config) lfs-sysvbook 62ebe64 (1.0.0; untouched since it was written) **`BUILD.md` is the one document for building a whole system**, host to desktop, with what to configure before each step. README.md points at it. Keep it true: when a command or a config key changes, it changes there too. -`test_lfs_crosschain.sh` is the regression suite. It no longer finishes inside -one command slot in the authoring sandbox; run it whole on a real machine, or -one block at a time (each starts `# ---- <title>`). One check fails by -environment only: **rc.site** needs a SysV LFS book on disk. +`test_lfs_crosschain.sh` is the regression suite. It runs in pieces now: +`--list` numbers the blocks, `--from N --to M` or `--only N,M|<text>` runs +some, and a selection re-executes a generated subset (preamble + blocks + +summary) whose path it prints. rc.site now SKIPs when the book cannot have +it, so a clean run reads clean. The full per-release history — 144 entries, every bug and what it taught — is `HANDOFF-history.md`. It is worth grepping when something looks familiar; it @@ -36,46 +38,69 @@ is too long to read. A ThinkPad-class AMD Renoir laptop, SysV init, 93 package accounts, collector prefix `nimgnu`, package prefix `p`. -**Built and verified: 319 packages**, `packagemanager verify` clean. That -includes the whole sway stack (wlroots, sway, swaybg, foot, seatd, elogind, -webkitgtk, pipewire, mesa, gtk3/4, gstreamer), the network stack (wpa_supplicant, -libnl, iptables-nft with libnftnl, net-tools), services (dbus, alsa, bluetooth, -acpid, fcron, blfs-bootscripts) and bash-completion. +**Built and verified: 319 packages**, `packagemanager verify` clean — the +whole sway stack (wlroots, sway, swaybg, foot, seatd, elogind, webkitgtk, +pipewire, mesa, gtk3/4, gstreamer), the network stack (wpa_supplicant, libnl, +iptables-nft with libnftnl, net-tools, dhcpcd), services (dbus, alsa, +bluetooth, acpid, fcron, blfs-bootscripts, openssh) and bash-completion. -**Kernel 7.2.4 built, installed and verified** by `lfs-kernel`: image on the -ESP, modules with a depmod index, headers, v4l2loopback, and 15 embedded -firmware blobs. `lfs-kernel verify` reports 32 checks green. +**Kernel 7.2.4 built, installed, verified and BOOTED.** `su -` works, the +setuid bits are back, and the desktop runs: sway with foot, and the +application-account launchers in it. -**The machine is still running 7.2.0.** 7.2.4 is on the ESP as -`vmlinuz_new`, beside the working `vmlinuz`, and has never been booted. -Nothing is recorded as known-good yet, which is correct and deliberate. +**The desktop is in daily use.** 14 application accounts (browser, claude, +chatgpt, gemini, nrwgpt, grok, mercury, deepseek, github, codeberg, reddit, +youtube, dokus, mpv) with their launchers, menu entries and icons; `~/Shared` +links for the ten that asked for one; passwordless `su` to each through +/etc/pam.d/su_u_user. -**Nothing in the desktop has ever run.** The last sway attempt failed in the -chroot for want of `/run/udev`, which is the expected result there. +**Wired networking works and now starts itself.** The interfaces are +`enp2s0f0` and `enp5s0` (not the `eth0` of any example), and `cfg bootscripts` +wrote an `/etc/sysconfig/ifconfig.<iface>` for each: ONBOOT, SERVICE=dhcpcd. +sshd's init script and its rc?.d links are installed. +**Wireless is the one piece still missing.** The interface is `wlan0` on this +kernel — NOT the `wlp3s0` this file said for months, and not `wifi0`. It needs +two files nothing may write for you, because one of them holds a key: +/etc/sysconfig/ifconfig.wlan0 (SERVICE=wpa, handing off to dhcpcd) and a +per-SSID /etc/sysconfig/wpa_supplicant/wpa_supplicant-wlan0.conf, chmod 600. +`packagemanager stack network --only net-wifi --run` prints both, with the +hand-off variable read off this machine's own `/lib/services/wpa`. + +**Not verified since the tools changed under it:** whether the locale step has +run (see below), whether a .desktop entry launches now that the launcher +allocates a pty, and `packagemanager service`, which has never run on this +machine. --- ## What to do next, in order -1. **Point rEFInd at `/boot/efi/EFI/NimGnu/vmlinuz_new`** as a second entry. - Keep the existing `vmlinuz` entry: it is the way back. -2. **Reboot into 7.2.4.** Then `uname -r`, `lfs-kernel verify` (last-good - should read "it is running now"), and `dmesg | grep -i firmware` — that - last one is the definitive `FIRMWARE_ONLY` list for this hardware. -3. **Start sway** as the desktop user, on a tty: `sway` (an alias for their - own `sway_start`). The seat comes from `/usr/bin/seatd-<user>`, a setuid - wrapper compiled for one username; there is no seatd daemon and no `seat` - group. If it fails, `stacks/sway-session.sh`'s launcher checks the three - things that were each a separate bug: the wrapper, `/run/udev`, and - whether this tty is the active console. -4. **Bring up wifi.** `wifi` is in `/bin`, `wpa_supplicant`/`wpa_cli` in - `/usr/sbin`. Still missing: `/etc/sysconfig/ifconfig.wifi0` (with - `SERVICE=wpa`) and the per-SSID files under - `/etc/sysconfig/wpa_supplicant/`. Note their `wifi` script calls - `ifup $IFNAME wifi0` with `IFNAME` unset while `INTERFACE` is set and - unused — one of the two should go. -5. **make-ca's weekly job.** The systemd book enables a timer, which the shim - skips. fcron exists now, so add the 12.4 book's cron entry by hand: +Everything here is on the machine, not in the tools. The tools are at +1.14.302 and clean. + +1. **The locale.** Nothing sets `$LANG`: lfs-helper wrote an i18n.sh with the + export commented out (1.14.298) and there is no /etc/locale.conf. Every + program is in the C locale and foot says so at every start. + + lfs config locale de_DE.UTF-8 # if it is not that already + packagemanager stack services --only shell-env --run + # then log in again, or: . /etc/profile.d/i18n.sh + +2. **The launchers written before 1.14.295.** A menu entry does nothing when + the launcher does not allocate a pty for `su`, and a launcher these tools + did not write is left alone. `~/bin/claude` is the last hand-made one: + + rm ~/bin/claude + packagemanager stack services --only app-users --run + + Then click a menu entry. If one still does nothing, run it from a + terminal first -- that path prints everything `su` has to say. + +3. **Wireless.** `packagemanager stack network --only net-wifi --run`, then + write the two files it prints for `wlan0`. + +4. **make-ca's weekly job.** The systemd book enables a timer, which the shim + skips; fcron is installed, so the 12.4 book's cron entry goes in by hand: cat > /etc/cron.weekly/update-pki.sh << "EOF" #!/bin/bash @@ -83,12 +108,15 @@ chroot for want of `/run/udev`, which is the expected result there. EOF chmod 754 /etc/cron.weekly/update-pki.sh +5. **Try the new commands once**, so their first use is not in an emergency: + `packagemanager service` (list, then `service sshd start`), and + `packagemanager stack dev --run` for gdb. + Smaller, none blocking: `dbus` was built before `[dbus] systemd = disabled` went into machine.conf (rebuild or accept); a stray `.service` from -at-spi2-core sits in `/usr/lib/systemd/user`; root cannot log in after boot -and the cause is unknown — wants `ls -l /etc/shadow`, `getent shadow root`, -and whether `/bin/bash` is in `/etc/shells`. - +at-spi2-core sits in `/usr/lib/systemd/user`; `/etc/bashrc` on this machine +has a mangled line (`# End /etc/bashrcCK=$(tput setaf 0)`) where an append +ran without a leading newline -- harmless, `BLACK` is simply unset. --- ## The tools @@ -99,6 +127,9 @@ and whether `/bin/bash` is in `/etc/shells`. blfs turns book pages into install scripts lfs-phases the phase runner every generated script sources lfs-kernel build/install/verify a kernel and its firmware + lfs-backup back up backup.conf's files; restore creates what is + missing and writes .new beside everything else + (replaced lfs-config in 1.14.291) lfs-sysvbook graft the 12.4 SysV flavour onto a systemd-only LFS book -> 13.0-sysv (LFS only; BLFS has no counterpart yet, see deferred). Its own notes: @@ -132,6 +163,15 @@ Makefile's `TOOLS` line, so a new tool needs no second edit. These are the ones that cost releases. They are worth applying to new code without waiting for the failure that proves them again. +**A feature is not proved by a file existing.** ffmpeg and SDL3 installed, +passed `have=`, and had no ALSA, because the library was built after them. +An autodetected feature needs its dependency placed by entry order and a +probe (`expect=`) that reads the built artifact. + +**A repair must not be able to fail the thing it is repairing.** It runs +after something already went wrong, so every input it reads may be missing +and every variable it names must be one the caller actually sets. + **Confirm your own effects.** A zero exit is not proof: a phase whose log says "make: *** Error 1" failed whatever it returned; a write must be read back (`usermod -d` returned 0 and changed nothing, three times); a step that @@ -187,26 +227,3584 @@ booted, because an empty file looked like something to fill in. --- -## This session (1.14.146) +## 1.14.239 -- shadow stops at configure: --disable-logind -- `BUILD.md` written; README tightened (kernel caveat was stale: it said - nothing builds a kernel, `lfs-kernel` does). -- `lfs-helper`: the end-of-build message pointed at book 10.3 by hand; - now points at `kernel.conf` + `lfs-kernel`. -- `stacks/sway-session.sh` hardcoded one username as the desktop user while - `mytools.sh` read `/etc/pkgusr/desktop-user` -- two settings, one - question. Both now take `LFS_DESKTOP_USER` or that file, and both write - it. Installed copies of the stack file are unedited, so `make install` - updates them in place. -- Not run: the regression suite (sandbox slot). `bash -n` / `py_compile` - and `--version` only. + checking for sd_session_get_remote_host in -lsystemd... no + configure: error: libsystemd not found. If logind integration is not + required, disable it using the --disable-logind option. + +shadow 4.20 auto-detects logind, and the book's page -- written for a +systemd host where libsystemd is simply present -- passes no switch either +way. On this machine it stops. configure names the flag itself, so: + + [shadow] + configure_args = --disable-logind + +Verified: the flag lands on the book's own `./configure` line. + +This IS the case `book =` was built for -- a page whose COMMANDS do not +suit this machine -- but the 12.4 page would also mean shadow 4.18.0, and +one switch is cheaper than four versions back. Recorded next to the setting. + +**shadow was interrupted mid-build**, so on the machine: + + packagemanager update shadow --reinstall --regenerate --run --yes + +`--regenerate` matters: machine.conf is applied when the script is written, +and the script on disk was written before that section existed. + +## 1.14.240 -- `book` is for the tool, and it reached the build line + + meson setup -Dbook=/usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html .. + ERROR: Unknown option: "book". + +`[linux-pam] book = ...` says WHICH BOOK to read that package from. It is +addressed to the tool, and every other key in a machine.conf section is a +BUILD option, so it went straight onto the meson line. `jobs` and +`configure_args` were each popped out by hand; `book` was new and nobody +popped it. + +A reserved SET now, so the next such key cannot leak the same way. +`_warn_unknown_meson_opts` could not have saved this: at script-generation +time the source is not unpacked, so there is no meson.options to check +against. + +Asserted with a fixture carrying BOTH kinds in one section -- `book` must +not reach the build line, `jobs` must still become `ninja -j4`. + +Two things visible in that script worth noting: + + * the override WORKS -- `required=('Shadow-4.18.0')`, + `name_version='Linux-PAM-1.7.1'`, `pkgusr_book_mirror=.../12.4`: the + whole page came from the older book, which is the design. + * 1.14.238 said linux-pam does not need the override (13.1's page needs + nothing from systemd to build, and `install` drops avoided packages by + itself). With this fixed, either choice now works; the newer page + still means 1.7.2 instead of 1.7.1. + +## 1.14.241 -- zathura and imv, and a package that was built and avoided + +Neither is in the BLFS book, so they are `git` entries in **sway.stack** +(with the other applications, not media.stack -- media is the codec chain, +and a PDF viewer is not a codec). + +CHECKED with git ls-remote, all four at once: + + * **girara, zathura, zathura-pdf-poppler share the tag 2026.07.18** -- + they are one project released together, and mixing tags is the usual way + to end up with a zathura that cannot open a PDF. The suite asserts one + tag across the three. + * **imv v4.3.0**; its repo answers on both exec64/imv and eXeC64/imv. + +**zathura is three packages.** girara is its widget library, the core opens +no formats at all, and the backend is a separate plugin -- a zathura with no +plugin starts, shows a grey window and says nothing. So the probe looks for +`/usr/lib/zathura/libpdf-poppler.so`, not for the binary. + +**poppler left avoid.stack.** It was there as libcupsfilters' child; +zathura's backend wants it, so it is a `book poppler` entry now. mupdf +stays avoided -- it is the other backend and one is enough. + +**And the check for that class found a real contradiction:** `avoid openssh` +was in avoid.stack while network.stack BUILT openssh with the install-sshd +bootscript. Two files disagreeing about one package, and whichever a plan +saw first won. Removed, and every stack is now checked for it. + +One test bug fixed on the way: the accounts-before-openssh assertion +compared an AVOID entry (index 50) against a build entry (73) and called +the order wrong. It filters to build entries now -- the cost of every +stack including the exclusions is that an index means nothing unless you +say which kind you mean. + +## 1.14.242 -- cmake will not overwrite a symlink, and journald came back + +**tiff.** Round 1 of the repair worked -- `/usr/lib/pkgconfig/libtiff-4.pc +belongs to p_libtiff -- handing it to p_tiff` -- and the retry then died on + + file INSTALL cannot duplicate symlink ... at /usr/lib/libtiff.so.6 + because: File exists + +Ownership was never the problem: cmake refuses to REPLACE a symlink at all, +so handing it over changes nothing. It has to go. The repair removes a +symlink owned by ANOTHER PACKAGE USER (never root's, and it says whose it +was) and lets the install create its own. + +The extractor also had to grow: in the symlink message the paths are NOT +quoted, unlike `cannot copy file "A" to "B"`, which is why it saw nothing +and the round did nothing. + +**p_libtiff AND p_tiff are the same upstream package** -- 12.4 titles the +page libtiff, 13.1 titles it tiff. Two accounts, two copies of one +library, and whichever installed last owns every file. Reported here, NOT +fixed: `packagemanager which-package /usr/lib/libtiff.so.6` says who holds +it today, and one of the two accounts should be retired deliberately. + +**webkitgtk: `fatal error: systemd/sd-journal.h`** -- my regression. The +machine's own conf had `ENABLE_JOURNALD_LOG = OFF`; the [webkitgtk] section +I shipped at 1.14.203 did not, so taking the shipped version dropped it and +webkit autodetected journald. Nine thousand files in. Restored, with +ENABLE_BUBBLEWRAP_SANDBOX = OFF beside it (also in the machine's conf and +also missing from mine). + +When a shipped section replaces a hand-made one, every switch that is +missing from it is a switch silently turned back to its default. `diff +machine.conf machine.conf.new` before taking one. + +## 1.14.243 -- poppler's soft-mandatory deps, and bubblewrap confirmed + +**ENABLE_BUBBLEWRAP_SANDBOX = OFF is in the shipped [webkitgtk] section** +(1.14.242, beside ENABLE_JOURNALD_LOG). Both were restored in the same +edit; `grep -A9 '^\[webkitgtk\]' machine.conf` shows the pair. + +**poppler stops at configure for nss, and would then stop twice more.** +cmake calls these "soft mandatory" -- absent, they fail the configure and +name the switch themselves: + + Could not find the 3.98 version of NSS3. If you're not interested in + the features it provides set the cmake ENABLE_NSS3 option to OFF + +The point nobody had noticed: **nss, boost and gpgmepp are in avoid.stack +AS POPPLER'S CHILDREN** -- they were avoided because poppler was. Promoting +poppler out of that list (1.14.241, for zathura's backend) brought all three +back into question, and each would have failed in turn, one long configure +apart. They serve PDF signature verification and the splash backend; +zathura's poppler-glib backend needs none of it: + + [poppler] + ENABLE_NSS3 = OFF # OBSERVED -- the failure above + ENABLE_GPGME = OFF # ANTICIPATED, same avoid list, same mechanism + ENABLE_BOOST = OFF # ANTICIPATED + +The comment marks which was seen and which was inferred, and the three +children STAY avoided -- turning the features off is the whole point. +Verified the switches reach the book's `cmake -D CMAKE_BUILD_TYPE=...` line +(it sits after `&&`, which 1.14.222 taught the merge to handle). + +**The general rule worth carrying:** a package promoted out of avoid.stack +brings its avoided children back into question. The suite now checks this +particular case; the pattern will recur the next time something is +promoted. + +## 1.14.244 -- two invented values in one entry + +Both mine, both from guessing instead of reading, and both caught by +machinery that was working correctly: + + * **`-Dsqlite=enabled`** stopped meson dead: `Unknown option: "sqlite"`. + zathura has no such option. Its meson.build takes sqlite + unconditionally -- `dependency('sqlite3', version: '>=3.35.0')` -- and + its only options are synctex, seccomp, landlock, manpages, tests-x11, + tests-wayland and convert-icon. Dropped; sqlite3 is already installed + (p_sqlite). + * **`have=/usr/lib/pkgconfig/girara-gtk3.pc`** made the stack refuse to + record a girara that had installed perfectly: + girara finished, but /usr/lib/pkgconfig/girara-gtk3.pc is still + missing -- NOT recording it as done + girara's meson.build says `pkg.generate(name: 'girara')`, so the file + is **girara.pc**. That message is the have= check doing its job on a + wrong have=, which is the better failure of the two. + +The pattern, three sessions running now (librice's four wrong guesses, the +zathura option, the girara filename): an entry's option names and installed +filenames must come from the source, and the source is one `git clone +--depth 1` away. Asserted both, by the shape that was wrong. + +(The run also reported `misplaced home: p_zathura is recorded at +/usr/src/p_zathura` and moved it to /usr/src/pkgusr/p_zathura by itself -- +that check earning its keep.) + +## 1.14.245 -- zathura also wants libxxhash + + Run-time dependency libxxhash found: NO (tried pkgconfig and cmake) + meson.build:57:9: ERROR: Dependency "libxxhash" not found + +The BLFS book mentions xxhash eight times and gives it **no page**, so the +entry is ours. Checked in the checkout rather than guessed, for once +without a correction afterwards: + + * newest tag **v0.8.3**; + * it ships `libxxhash.pc.in`, so the pkg-config module really is + `libxxhash` -- the name meson asks for -- and `make install` runs + `install_libxxhash.pc` among its targets; + * a plain Makefile, no meson or cmake: `PREFIX` and `LIBDIR` are what it + takes. + +Placed at 143, between girara (142) and zathura (144); the suite fails if +either dependency lands after zathura. + +Everything else in that run was right: girara `installed` with the +corrected have=, poppler `done` with the three ENABLE_*=OFF switches, and +poppler and openssh both gone from the "never built by this stack" line. + +## 1.14.246 -- a book entry was recorded done without being checked + + pkg-config --list-all | grep -i poppler -> nothing + ...while the stack said: book poppler done (stack cache) + +poppler was never installed, the stack believed it was, and every later run +skipped it -- so zathura-pdf-poppler kept failing on +`Dependency "poppler-glib" not found` for a dependency the plan was certain +it had. + +The "is it actually there afterwards?" guard existed and covered **git and +tar only**. A `book` entry went straight to `_stack_mark_done`. Now every +kind answers the same question: git/tar by have=/prog=, book by the package +state that `verify` reads -- and an empty manifest is not an install. + +Also asked for and added: + + [poppler] + ENABLE_GLIB = ON + +cmake autodetects the glib frontend, and that frontend is the entire reason +poppler is in this stack. The thing a chain exists for should be asked +for, not inferred -- the gst-plugins-ugly `gpl` rule again. + +### Why the dependency was not "pulled in" + +It cannot be. girara, xxhash, zathura and the backend are `git` entries, +and dependency resolution only works for `book` entries -- required= and +recommended= come from the book page. A git entry has no page, so nothing +can know zathura needs girara. That ordering is by hand in the stack file, +which is why the suite asserts it. + +## 1.14.247 -- the fourth soft-mandatory switch, and how to stop guessing + + Could not find the 6.4 version of Qt6Core. If you're not interested in + the features it provides set the cmake ENABLE_QT6 option to OFF + +`ENABLE_QT6 = OFF`. qt6 is in avoid.stack and poppler's page passes only +`ENABLE_QT5=OFF`, because a BLFS system is expected to have qt6. + +That is **four** of these found one configure at a time -- NSS3, then GPGME +and BOOST (guessed from the avoid list), now QT6 -- each a full cmake run +apart. The list was never a guessing game: poppler's own CMakeLists names +every one, and on that machine the file is right there: + + grep -n find_soft_mandatory_package \ + /usr/src/pkgusr/p_poppler/src/poppler-*/CMakeLists.txt + +That is in machine.conf beside the switches now, to be read once when +poppler moves version rather than rediscovered a configure at a time. The +suite checks the four switches and that the grep is written down. + +The general shape, for the next package like this: when a build names a +switch in its own error message, that switch has SIBLINGS, and the +mechanism that produced the message (here `find_soft_mandatory_package`) is +greppable in the source that is already unpacked. + +## 1.14.248 -- `update` is a compound phase too + +Asked whether there is a general problem with rebuilding the whole package. +There was: + + !! phase 'install' FAILED <- the inner phase that failed + /usr/share/gir-1.0 -> group nimgnu_gir (owner p_glib2, + p_poppler) + # retrying poppler (phase update), round 1 ... + unpacked into /usr/src/pkgusr/p_poppler/src/poppler-26.08.0 + -- The C compiler identification is GNU 15.2.0 <- from scratch + +145 files were already installed and the grant had fixed the one thing that +was wrong. The retry re-unpacked the tarball and re-ran cmake anyway. + +`_retry_phase` mapped the inner failure for `all` and for nothing else, so +a failure inside `update` retried `update` -- and `update` re-unpacks. Both +compound phases use the inner phase now; a single-phase run keeps its own +(1.14.209). + +That is the third time this exact shape has cost a rebuild: 1.14.70 (foot, +seven rounds), 1.14.209 (the second round re-cloning), and now the compound +phase that was not in the list. The rule: **whenever a retry picks a +phase, it picks the one that FAILED, and the log says which that was.** + +The grant itself worked exactly as intended, and is worth noting because it +is the good half of that output: `/usr/share/gir-1.0 is owned by 'p_glib2' +-- asking rather than opening it to every package`, then a collector group +for the two of them. + +## OPEN BUG -- a skipped step leaves its privileged half live (poppler) + + make: *** No rule to make target 'install'. Stop. + !! phase 'install' FAILED (exit 2) ...after 218 files installed fine + +**Diagnosed, NOT fixed.** Generated from the real book, poppler's install +phase comes out as: + + ninja install + install -v -m755 -d /usr/share/doc/poppler-26.08.0 && + cp -vr ../glib/reference/html /usr/share/doc/poppler-26.08.0 + ## SKIPPED -- the book says: If you downloaded the additional encoding data package... + # tar -xf ../../poppler-data-0.4.12.tar.gz && + # cd poppler-data-0.4.12 + make prefix=/usr install <- LIVE + +The book writes an optional step as a PAIR: an unprivileged block that +unpacks and `cd`s into a sub-package, then a `<pre class="root">` block that +installs it. The guard comments out the first half and leaves the second +running -- so `make` runs in poppler's own ninja build directory, which has +no Makefile. The tell is the `cd`: whatever follows a skipped block that +ends by changing directory runs in that directory or not at all. + +I wrote that rule into both emitters (`extract_commands` and +`extract_commands_phased`) and **neither fired** -- the regenerated script +was byte-identical. **1.14.262 found why: there is no third path.** The +index cache was keyed on the book file and a hand-bumped CACHE_VERSION, so +the script was rendered from data the OLD code had extracted; the patches +never ran. The cache key now includes blfs's build id. The reverted rule +is probably right as written -- re-apply it and regenerate. Both patches are REVERTED rather than shipped: code that +fails the one case it was written for is worse than an admitted bug, and +the next person needs the diagnosis, not my dead code. + +Still reproducible from the rendered book (poppler-26.09.0, 1.14.262). +Where to start: re-apply the `cd` rule, then regenerate poppler -- + + blfs --book-file <the 13.1 book> script poppler -o /tmp/x + +-- and find which emitter actually appends `make prefix=/usr install`. + +**Meanwhile, on the machine:** poppler IS installed (218 files, and +`ninja install` plus the doc copy both ran). Delete that one line from the +script and finish: + + sed -i '/^make prefix=\/usr install$/d' /usr/src/pkgusr/p_poppler/install_Poppler-26.08.0 + lfs-helper build poppler --phase install --force + pkg-config --modversion poppler-glib + +Do NOT `--regenerate` poppler until this is fixed: it will put the line +back. + +## 1.14.250 -- `home` was not in scope, so the marker was never written + + # tracked 218 file(s) for 'poppler' + /usr/bin/lfs-helper: line 7345: home: unbound variable + # poppler: done (all phases complete) + +poppler installed. The line between those two is mine: the +`.pkgusr-installed` marker from 1.14.236 reads `$home`, which is a local of +the INSTALL function and not of the one that prints "done (all phases +complete)". Under `set -u` that aborted the block -- so the marker, the +entire point of 1.14.236, was never written for any package. + +It resolves the home the way every other caller here does, +`pkgusr_home_for "$name"`, and only writes when install_last is really +there. Asserted three ways: the old `$home` read must be gone, the +resolver must be used, and the marker must still sit within a few lines of +the success line. + +Worth naming the shape: 1.14.236 was tested by calling the PREDICATE +(`gather_user().version` with and without a marker) and never by running +the code that writes it. A test that exercises the reader and not the +writer proves half a feature. + +## 1.14.251 -- application homes in /usr/src, and imv's missing DSO + + ls /usr/src/ + cfg lfs-pkgusr linux p_mytools pkgusr u_browser-big u_browser-mini u_claude u_test + +**An application user is not a package user.** `add_package_user` creates +the account -- it handles the uid range and the matching group -- and it +puts the home where PACKAGE homes go. A package user's home is a build +tree; an application user's home holds a browser profile, a .config, a +Downloads directory. It belongs in /home, which is also where the shared +links and the XDG skeleton expect it. + +`_relocate_app_home` moves it with `usermod -m` (contents included) and +chmods 0750 -- but ONLY from the package tree, and only when the target +does not exist. A home someone has already placed elsewhere is left alone +and said so. + +For the four accounts already made: + + usermod -d /home/u_claude -m u_claude && chmod 0750 /home/u_claude + usermod -d /home/u_browser-mini -m u_browser-mini && chmod 0750 /home/u_browser-mini + usermod -d /home/u_browser-big -m u_browser-big && chmod 0750 /home/u_browser-big + usermod -d /home/u_test -m u_test && chmod 0750 /home/u_test + +**imv: `undefined reference to symbol 'ubrk_close_78' ... DSO missing from +command line`.** icu splits its API across libicuuc (common), libicui18n +and libicuio; imv's meson names libicuio, calls into libicuuc, and links +with -Wl,--no-undefined -- so the indirect dependency is an error, not a +warning. `-D c_link_args=-licuuc` names it. Nothing is patched. + +### (FIXED in 1.14.252 -- see above) shadow's install strips su's setuid bit + +Named, not fixed, and it needs its own session: the LFS Shadow page sets +4711 on /usr/bin/su and the BLFS page does not, so a BLFS-driven reinstall +of shadow leaves su non-setuid -- and the same applies to passwd, chage, +gpasswd and newgrp. `lfs-helper setid` records bits a PACKAGE USER was +refused; these are bits the SCRIPT never asks for, so there is nothing to +record. What it wants is a declared list of required setuid/setgid files +(the accounts.conf shape), enforced after any install that touches them. +Until then, after any shadow install: + + ls -l /usr/bin/su /usr/bin/passwd /usr/bin/chage /usr/bin/gpasswd /usr/bin/newgrp + chmod 4711 /usr/bin/su + +## 1.14.252 -- pam then shadow, and the modes are FIXED not reported + +Asked for both, and both were right. + +**`cfg shadow-pam`, immediately after `book linux-pam` in services.stack.** +Installing PAM changes nothing on its own: until shadow is rebuilt against +it nothing reads /etc/pam.d, so `su - u_claude` kept asking for a password +however correct su_u_user was. The book's Linux-PAM page ends with "You +should now reinstall the Shadow package"; that is now a step rather than a +sentence. It checks its own effect with `ldd /usr/bin/su | grep libpam`, +and with no PAM installed it says so and does nothing. + +**And the setuid bits are applied, not warned about.** The rebuild strips +them -- the BLFS Shadow page never sets them, the LFS one does -- so the +very rebuild that makes PAM usable broke su. Declared in +**stacks/setid.conf**: + + /usr/bin/su | 4711 | root | - + /usr/bin/passwd | 4711 | root | - + /usr/bin/chage | 4711 | root | - + /usr/bin/gpasswd | 4711 | root | - + /usr/bin/newgrp | 4711 | root | - + /usr/sbin/unix_chkpwd | 4755 | root | - + +`-` in the group column KEEPS what is there, which is what was asked for: +the group stays p_shadow (ownership follows the installer in this system; +only the USER must be root for a setuid bit to mean root). Result on a +test file: `4711 root:p_shadow`. + +Two details that decide whether this works at all: + + * **chown CLEARS the setuid bit**, so chmod runs last. The suite checks + the line order, not just that both calls exist. + * it reads the mode BACK afterwards -- a setuid bit that did not take is + the entire failure this step exists to end. + +`lfs-helper setid` is still the right tool for bits a build ASKED for and +was refused; these are bits no script ever asks for, which is why they need +declaring. + +## 1.14.253 -- `pam_start: error 26` -- linked, with nothing to read + + su: pam_start: error 26 + +PAM_ABORT: there is no /etc/pam.d/su. So the rebuild in 1.14.252 worked -- +su links libpam now -- and left it worse than before, because a PAM-linked +su with no service file cannot fall back to anything. + +`update` runs install. It does NOT run configure, and those files are +written by the CONFIGURE phase: linux-pam's writes /etc/pam.d/system-auth, +system-account, system-session, system-password; shadow's writes +/etc/pam.d/su, login, passwd, chage and the rest. Both were skipped by +every `update --reinstall` of either package. + +`cfg shadow-pam` now checks for /etc/pam.d/su and /etc/pam.d/system-auth, +runs `packagemanager script configure linux-pam` and `... shadow` when they +are missing, and reads back that both files exist -- naming the error they +produce if they do not. + +The general shape, worth keeping: **install and configure are different +phases, and a package whose configure writes system files is not finished +by an install.** That is the third face of the same coin as +"/etc/pam.d/su_u_user is inert until shadow links pam" and "a rule on disk +is not a rule in force". + +## 1.14.254 -- config BEFORE the rebuild (the window where nothing builds) + +1.14.252/253 rebuilt shadow and THEN wrote /etc/pam.d. Between those two +there is a state where su links libpam and has no service file -- and every +build in this system runs as its package user through `su - p_<pkg>`, so +that window is one where **nothing can be built**, including the configure +phases that would write the missing file: + + # packagemanager script configure linux-pam + su: pam_start: error 26 + +Root's own su survives it because pam_rootok is sufficient -- which is why +`su - p_bash` still worked and reported the pam failure as `(Ignored)`. +That is the only reason this was recoverable. + +So the step is reordered: **write the config, then rebuild.** A run that +stops midway now leaves pam config and a non-pam su (harmless) rather than +a pam su and no config (nothing builds). Asserted by LINE ORDER, not by +the presence of both calls. + +One more bug in the same file: step 1 called `_pam_installed` forty lines +above where it was defined. Under `set -u` that is a command that does not +exist yet, so the config would silently never have been written. Both +helpers are defined before either step, and the suite checks that too. + +### If it ever happens again + +Root can write those files directly -- no su involved. The contents are on +the book's Linux-PAM and Shadow pages; /etc/pam.d/su wants the `pam_exec.so +quiet /etc/pam.d/su_u_user` line FIRST, then pam_rootok, then pam_unix. + +## 1.14.255 -- the noise on every build: package users have no shadow entry + + su: Authentication service cannot retrieve authentication info + (Ignored) + +Not cosmetic, because **every build in this system is a `su - p_<pkg>`**: +that line will now appear in every package build and every log. pam_unix +reads /etc/shadow; `add_package_user` writes /etc/passwd and leaves no +shadow entry, so pam_unix cannot find the account. It is "(Ignored)" only +because pam_rootok already succeeded for root -- for a non-root caller it is +a real failure. + +`cfg shadow-pam` step 4 lists the accounts with no /etc/shadow entry, backs +up /etc/shadow, runs **pwconv** (shadow's own tool: it adds a locked entry +for every passwd entry that lacks one), and reads back whether they +appeared -- naming any that did not. + +**UNVERIFIED:** detection, backup and read-back were exercised against a +fixture; `pwconv` itself was not, because it edits the real /etc/shadow and +there is no safe way to run that here. The script says so where it runs it. +If pwconv turns out not to cover these accounts, the read-back names them +and the backup is beside the file. + +### State after this session + +pam config in place, `/usr/bin/su` links libpam, 6 setid files checked and +0 corrected (the bits were already right). What remains is one test: + + su - u_browser-mini -c id + +and, if that works, the home moves (`usermod -d /home/u_<n> -m u_<n>`) plus +`packagemanager stack services --only app-users --run` to recreate the +~/Shared links and the SHARED_* fragment against the new homes. + +## 1.14.256 -- the shared directory was built under /usr/src too + + $ mkdir -p /usr/src/u_claude/Shared + /home/n76310/Shared/Claude -> /usr/src/u_claude/Shared + +The homes had been moved to /home with usermod, and the next +`cfg app-users` run rebuilt Shared in the package tree anyway. Cause: +`appuser_home()` returned **BASE_DIR** -- so an application user's home, +its shared directory and its XDG profile link all defaulted to /usr/src. +1.14.251 moved the ACCOUNT's home and left three other places reading the +old default. + +Two changes: + + * `appuser_home()` defaults to **/home** (still configurable as + `appuser_home`). + * `_app_home(u)` asks **passwd first**, then falls back -- so an account + whose home has already been moved keeps it, which is exactly the case + that failed. The shared directory and the .bash_profile link both use + it. + +### STILL OPEN from the same output + + * **step 4 (pwconv) did not run.** `shadow-pam` printed neither "every + account has an /etc/shadow entry" nor the missing list, so the block was + not reached -- and `su - p_bash` still says "cannot retrieve + authentication info". The installed script is the thing to check: + grep -n pwconv /etc/pkgusr/stacks/shadow-pam.sh + If that is empty, `make install` kept an older copy (it keeps a file you + have edited, and says so). If it is present, the guard above it + (`[ -r /etc/passwd ] && [ -r /etc/shadow ]`) is what to instrument. + * **/etc/pkgusr/skel-u_xdg/.bash_profile does not exist** on that + machine, so the profile link fails with ENOENT for every account. The + file is in the zip under skel-u_xdg/; `make install` reported "kept your + existing .bash_profile" for it, which cannot be true if the file is + missing -- that message is worth distrusting. + +### Corrective commands for the machine + + for u in u_claude u_browser-mini u_browser-big; do + mkdir -p /home/$u/Shared && chown $u:$u /home/$u/Shared && chmod 2770 /home/$u/Shared + cp -a /usr/src/$u/Shared/. /home/$u/Shared/ 2>/dev/null || true + rm -f "/home/n76310/Shared/$(echo ${u#u_} | sed 's/^./\U&/')" + done + packagemanager stack services --only app-users --run # relinks to /home + +## 1.14.257 -- there is no /etc/shadow on this machine + + # ls -l /etc/shadow + ls: cannot access '/etc/shadow': No such file or directory + +That is the whole explanation for `su: Authentication service cannot +retrieve authentication info` on every package build: pam_unix reads +/etc/shadow and there is none, so it can look up NO account. The passwords +live in /etc/passwd, the traditional way, which is what LFS leaves when +Shadow is built without PAM. + +And my guard was `[ -r /etc/passwd ] && [ -r /etc/shadow ]` -- so the step +skipped in silence on precisely the system that needs it. That is why the +earlier run printed nothing either way. + +`pwconv` CREATES /etc/shadow from /etc/passwd, moving each hash out and +leaving `x` behind. That changes how the machine authenticates, so the new +branch: + + * backs up **/etc/passwd** (the file that changes), not /etc/shadow; + * requires `^root:` in BOTH files afterwards before calling it done; + * chmods /etc/shadow 600 and reports the entry count; + * on any failure restores passwd, removes the half-made shadow, and says + nothing changed; + * prints the undo by hand: `cp -p <backup> /etc/passwd && rm -f /etc/shadow`. + +**CONFIRMED ON THE MACHINE.** The rollback path was exercised here +(shadow absent, passwd restored); the SUCCESS path was then run for real: +pwconv created /etc/shadow, root survived, and + + # su - p_bash -c id + uid=10006(p_bash) gid=10006(p_bash) groups=10006(p_bash),9999(install) + +-- silent, no "cannot retrieve authentication info". That line is gone +from every future build log. The remaining `cannot set terminal process +group` / `no job control` pair on an interactive `su -` is unrelated to +pam -- and CONFIRMED on the machine: shadow 4.20's su calls setsid() for +the target shell when the CALLER IS ROOT and the kernel still allows legacy +TIOCSTI (`/proc/sys/dev/tty/legacy_tiocsti` not 0), so the user cannot push +keystrokes into root's terminal. The shell gets its own session (SID = its +PID, TTY `?`), and bash has no job control. Fix is the kernel's: +`dev.tty.legacy_tiocsti = 0` (sysctl) or CONFIG_LEGACY_TIOCSTI=n -- the +latter is in kernel.conf's CONFIG_SET since 1.14.277. + +It is also worth asking whether this is wanted at all: the system works +today, and the only symptom is a noisy line on every build. Converting to +shadow passwords is the conventional answer and PAM expects it -- but +"leave it alone" is a legitimate choice, and then the line is cosmetic. + +### And the Shared layout is right now + + Claude -> /home/u_claude/Shared + Browser-mini -> /home/u_browser-mini/Shared + Browser-big -> /home/u_browser-big/Shared + +with `.bash_profile already points at the XDG profile` for all three. One +thing still open there: `/tmp/xdg-n76310 is group 'n76310', which is not a +shared group` -- the session-group half of --shared, which needs the sway +session up with a proper runtime group. + +## 1.14.258 -- apply the setid bits, and four that were missing + +Two things from shadow's install output. + +**It printed nine chmod lines and walked away.** +`report_setid_requests` runs AS ROOT, immediately after the install -- the +one moment when the thing that is missing can simply be done. It now calls +`cmd_setid --run` (which applies each recorded request and reads the mode +back) and reports what it DID. Non-root callers still get the old advice, +because then it really is advice. Same lesson as the PAM file's mode and +/etc/pam.d/su: warning about a one-line fix you are already privileged to +make is not help. + +**And setid.conf declared five of shadow's nine.** chfn, chsh, newgidmap +and newuidmap were missing -- read off that request list, not guessed. Ten +lines now. + +### On the other question: do shadow and pam each get the other's features? + +**Shadow has to be built TWICE, and that is unavoidable.** Their pages +require each other: Linux-PAM's page lists Shadow as required, and shadow +only gains PAM support if libpam exists when it configures. So the order is + + 1. shadow (no PAM -- this is what the LFS build leaves) + 2. linux-pam (needs shadow) + 3. shadow again (now with PAM) <- `cfg shadow-pam` does this + +PAM itself does NOT need a second pass: nothing in its configure depends on +a feature shadow gains later. So one rebuild, of shadow, in that order -- +which is exactly what services.stack does, with `cfg shadow-pam` +immediately after `book linux-pam`. The step checks `ldd /usr/bin/su | +grep libpam` rather than assuming, so a shadow that somehow already has it +is not rebuilt. + +### Asked for and NOT built: a general post-install root file + +The idea is right and the mechanism half exists (`.pkgusr-setid.requests` +is exactly this, for one kind of request). A general +`$home/.pkgusr-root.post` run by root after a successful install would cover +setid bits, udev reloads, ldconfig, bootscript symlinks -- everything that +currently needs a named repair. + +It is NOT implemented here because of one hard requirement: that file must +be **root-owned and unwritable by the package user**, and the runner must +verify that before executing a line of it. A file inside the package +user's own home, written by root, that root later executes -- if the +package user can touch it, every package on this system has a root +escalation. The verification (ownership, mode, no symlink, parent +directory not writable) is the feature, and it wants writing deliberately +rather than at the end of a long session. + +## 1.14.259 -- root:p_shadow, and a backtick that ran a command + +Both mine, both visible in that output. + +**`root:root` threw away the package group.** Only the USER has to be root +for a setuid bit to mean root; the group is how this system records which +package owns a file, which is the whole point of package users. +setid.conf already used `-` for exactly this -- the REQUEST path +(`cmd_setid --run`, which handles the bits a build asked for and was +refused) forced root. It keeps the group the file already has now: + + /usr/bin/su is now 4755 root:p_shadow + +**And the report printed twice, mid-sentence:** + + applied. Setuid/setgid bits refused during the build ... + ... every setid bit this system needs is in place lists anything still outstanding. + +My success line said `` `lfs-helper setid` `` inside double quotes, so the +shell RAN it and pasted a whole second report into the middle of the +sentence. Single quotes. A backtick in a double-quoted string is a +command, not punctuation -- and this is the fourth time this session that +a quoting mistake produced output that looked like a tool bug. + +Test-harness note in the same spirit: the first version of the group check +created its fixture as root, so root:root came out either way and the +assertion compared nothing. It chgrps to a real non-root group first now. + +## 1.14.260 -- two tables disagreeing, seventeen changes to the wrong answer + +1.14.259 made the REQUEST path keep the group. The output then read: + + /usr/bin/su is now 4755 root:p_shadow <- request table + ... + /usr/bin/su is now 4755 root:root <- "upstream" table, one line later + Applied 17 change(s). + +`cmd_setid` has TWO tables: the bits a build asked for and was refused, and +`_setid_known` -- what upstream installs, for packages whose request was +never recorded. The second hard-coded `root` in its group column, so it +ran immediately after the first and undid it. Nine files changed twice. + +`-` in that column now means KEEP WHAT IS THERE, and the shadow rows use +it. A file already at `4755 root:p_shadow` is recognised as correct and +skipped, so the two tables agree instead of fighting. + +**A setGID entry keeps its NAMED group**, because there the group IS the +purpose: dbus-daemon-launch-helper must be group messagebus, fcrontab group +fcron. That row is unchanged and asserted unchanged. + +The general rule, now written in the table's own header: **for setuid, only +the owner matters and the group belongs to whoever installed the file; for +setgid, the group is the feature.** + +## 1.14.261 -- the setid pass locked linux-pam out of its own file + + rm: cannot remove '/etc/pam.d/other': Permission denied + /etc/pam.d -> already group nimgnu_bash-completion, and 'p_linux-pam' is a member + # retrying linux-pam (phase install only -- the build is done), round 1 ... + PermissionError: [Errno 1] Operation not permitted: '/usr/sbin/unix_chkpwd' + +Round 1 was right and worked. The second failure is one I created two +releases ago: **/usr/sbin is sticky**, so only a file's owner may unlink it +there. linux-pam installs /usr/sbin/unix_chkpwd 4755, the post-install +setid pass chowns it to root -- which is exactly what a setuid bit needs -- +and the next reinstall can no longer replace its own file. EPERM, not +EACCES, which is why no permission repair recognised it. + +Three parts to the fix, all in `_take_cmake_dest`: + + * **meson's wording**: `Operation not permitted: '<path>'` extracts the + path. cmake says `cannot copy file "A" to "B"`; python says this. + * **the GATE learned it too** -- it only accepted cmake's phrasing, so the + function returned before looking. The same two-places-one-fact shape as + 1.14.222. + * **root's files are still off limits, with one exception**: a path the + package's OWN manifest claims. That is the file it installed and we + then gave to root; handing it back for the install is safe, and the + setid pass re-applies the bit afterwards -- so the loop closes instead + of alternating. A root-owned file no manifest claims is left alone, + which the suite checks with two files and one manifest. + +The lesson is about the shape, not the file: **a repair that changes +ownership creates work for the next install of the same package.** The +setid pass and this take-back are two halves of one cycle, and either alone +is a bug. + +## 1.14.320 -- the stacks, reorganised: wayland / sway / apps + +Asked to recheck every stack for naming and fit, and to take sway and its +tooling out of the Wayland base. sway.stack was 477 lines answering four +questions at once. Now: + + wayland.stack kernel check, dbus/elogind, the audio floor (alsa-lib, + ffmpeg), llvm/glvnd/mesa, wayland, GTK4 + libsoup3, git, + seatd, libdisplay-info, wlroots. Nothing sway-specific. + media.stack unchanged (codecs, gstreamer, WebRTC, webkitgtk, mpv, + yt-dlp). + sway.stack `include wayland`, `include media`, then the audio + server (after media, so pipewire finds gstreamer), SDL3, + scdoc, sway, swaybg, grim/slurp/swayidle, wl-clipboard, + wf-recorder, light, foot (+tllist, fcft), fonts, and the + user's own programs (minibrowser, swbr, swov, swas) and + sway-session. `stack sway --run` is still the whole + desktop. + apps.stack poppler, girara, xxhash, zathura, zathura-pdf-poppler, + imv -- programs, not the desktop. + tools.stack translate-shell, and jq (moved from sway.stack). + +Every entry was carried over by slicing the file at its comment markers, so +the comments travel with their entries; a script compared the entry lists +before and after (none lost, none doubled). One order change, on purpose: +`book git` now comes before media.stack's git entries (libnice, libsrtp, +...) instead of after them. + +On a built machine nothing is rebuilt: `stack sway`'s progress file is keyed +by the stack you run, so its cached entries stay cached; `stack apps` and +`stack tools` have no progress file yet and re-judge their entries, which +are installed and skipped. + +Tests: every check that looked for an ENTRY in sway.stack now reads all five +desktop files in build order ($_DESKTOP_STACK, built in the preamble); a new +block checks the includes, the placement of key entries, and that no entry +is written in two stacks. The expect= "thin probe" heuristic now accepts +`pkg-config --exists` (it flagged librice and xxhash once it saw them). +Full suite: the same failures as before the split, minus two. + +Reviewed and left alone: network (fits), services (see NEXT-TASK -- its name +undersells it), dev, diag, vision, avoid. + +## 1.14.319 -- tools.stack, with translate-shell + +A new stack for "the small programs you type": nothing boots with them and +nothing depends on them, so none of the other stacks is the right answer. +First entry: translate-shell (`trans`) v0.9.7.1, the newest tag. Built and +installed here with the entry's own two commands (`make` is gawk writing one +script; `make PREFIX=/usr install` copies it and the man page). needs=curl: +it fetches with curl and returns nothing without it. expect= only tests the +file -- `trans -V` went to the network and hung in the sandbox. + +## 1.14.318 -- scdoc + +grim stopped on `Program 'scdoc' not found`. 1.14.316 asked grim, slurp +and swayidle for man pages on the assumption that scdoc was present because +sway's own line asks for them -- it was not on the machine. scdoc 1.11.5 +(SLFS's version, from git.sr.ht) is now a git entry in sway.stack, before +sway, so a replay meets it in order. Not built here: sr.ht is not reachable +from this sandbox and no mirror carries the tag. + +## 1.14.317 -- needs= by the package's name, not only the book's anchor + +grim's `needs=libjpeg-turbo` stopped the stack: the book files +libjpeg-turbo-3.2.0 under the anchor `libjpeg`, so the name found no page +and the stack tried to INSTALL a package that was installed. The stack +now says `libjpeg`, and `_resolve_need` maps a name that is not an anchor +but matches exactly one page's own package name (version string minus the +version) to that page -- and says so. Ambiguous or unknown names are used +as given and fail as before. + +## 1.14.316 -- grim, slurp, swayidle + +Three git entries in sway.stack after swaybg, refs checked: grim v1.5.0 from +gitlab.freedesktop.org (the SLFS page's source; the GitHub mirror stopped at +v1.4.0 in 2022), slurp v1.5.0 and swayidle v1.9.0 from GitHub. Options +asked for rather than autodetected: grim jpeg + man pages, slurp man pages. +swayidle is built -Dlogind=disabled: sway takes its seat from seatd-<user>, +so there is no logind session for that half to talk to; the timeouts work +without it. + +## 1.14.315 -- the shared DATA lives in /home/Shared, and app homes are 0700 + +1.14.312 moved only the LINKS to a hub; the data stayed in +<account home>/Shared. Reaching it went through the account's home, which +therefore had to be group-readable (0750) -- and the main user is in every +account's group, so it could read each account's browser profile and +cookies. Pointed out from the machine, with the fix: put the real +directory in /home/Shared and link from the home. + + * /home/Shared/<Name> is the directory (2770 u_<name>:u_<name>); the + hub is root:<main> 0775. `shared_hub=` still moves it. + * The account's own ~/Shared is a link to it, so `--download-dir + ~/Shared` in app-users.conf means the same thing. The main user's + ~/Shared links to /home/Shared. + * Application homes are 0700 now. + * Data is never moved by the tools: a ~/Shared with files in it is left, + and the one-line mv is printed. The machine was migrated by hand, with + commands given in the conversation, as asked. + * `user delete`: the account's hub directory is removed when empty and + kept (with a note) when not -- it is no longer inside the archived home. + * 1.14.312's argument against /home ("a directory there looks like a + home") was about a hub of links; a hub holding the one shared thing is + what lets every home be private, which outweighs it. + +## 1.14.314 -- `book hatchling` was not "make sure it is there" + +1.14.312 put `book hatchling` in front of yt-dlp, as its build backend. On +the machine hatchling 1.28.0 was already installed, and the entry still +planned a chain from the book -- vcs_versioning first, which stopped on +`packaging>=26.2` (installed: older) -- so the stack ended at "0 of 2 done" +and yt-dlp never started. A dependency that every python build on the +machine already needed is not a stack entry; if it is ever missing the +build says `BackendUnavailable: Cannot import 'hatchling.build'`, which +lfs-helper already maps to its page. Removed. + +Not fixed here, noted: a plan that includes a package whose INSTALLED +dependency is too old (vcs_versioning wants packaging >= 26.2) does not put +that dependency's update in front of it. + +## 1.14.313 -- the machine's own files, adopted + +The machine sent its current app-users.conf, machine.conf, sway-session.sh +and install_mytools. The last two were identical to what ships. The first +two differed only by what the machine had added -- a `git` and a `discord` +account, and `USE_GTK4 = ON` under [webkitgtk] -- so those are now in the +shipped copies, byte for byte. Why it matters: `make install` updates a +stack file only while the installed copy matches the last one it shipped; +an edited file is kept and the new one parked as `.new`. With the two +identical, the next install records them as ours again and later fixes +arrive without a manual merge. + +Also: block 121 had failed since 1.14.307 -- its checks still looked for +the "left alone" wording that release replaced with a real answer (branch +moved? newer tag?). They check the new wording now. + +## 1.14.312 -- the shared hub is a system path; yt-dlp + +**The hub.** The links to every shared directory lived in the main user's +~/Shared. Asked to move them "where a package could reach them too", maybe +/home/Shared. They are now in ONE hub, `/home/shared_user/Shared` by +default -- beside the u_* homes they point into -- and ~/Shared is a link to +it, so nothing typed changes. `packagemanager config --shared-hub DIR` +moves it. Not /home/Shared: /home holds homes, and a directory there that +is not one looks like an account to anything that lists /home. + + * root owns the hub, your login's group may add a link by hand (0775). + * The move takes only links; a file of yours in ~/Shared keeps it a + directory, and says so. Re-running is a no-op. + * The SHARED_* variables come from the hub, so every account sees the + same set. Before, each login enumerated its OWN $HOME/Shared -- and in + an app account that is its data directory, so u_claude got one SHARED_* + variable per FILE it had downloaded. + * PKGUSR_SYS_SHARED names the hub. app-users.sh asks the tool for it + rather than reading that variable, which the running shell inherited + from before the move. + * cfg app-users' expect= now fails while ~/Shared is still a real + directory, so the step re-runs once on this machine and does the move. + +What the hub does NOT change: who may open a directory. Each is still 2770 +u_<name>:u_<name>, and a program runs as whoever STARTS it, never as the +package user that owns its file -- so your own programs already reach +everything your login can. Only a daemon running under its own account +needs a membership, and that is `usermod -aG u_<name> <account>`, or a +collector group (`sysgroup create` / `make-group-dir` / `sysgroup add`) for +a directory several accounts share. + +**yt-dlp** in media.stack, from upstream's release tarball (not a BLFS +page). The release asset, not the git tree: the wheel's shared-data names +the man page, README.txt and the completions, which are generated at release +time, and hatchling stops on the first one missing. Built here with the +book's wheel recipe; `yt-dlp --version` answers 2026.08.19. Pinned, with +the two lines to change when YouTube breaks it. (A `book hatchling` line +before it was removed again in 1.14.314.) + +## 1.14.311 -- --clean deleted current files, and its repair had not run since 1.14.16 + + packagemanager update webkitgtk --run --clean # all green + jsc/jsc.h:29:10: fatal error: jsc/JSCOptions.h: No such file or directory + +`packagemanager script install webkitgtk` fixed it, which was the clue: the +files were missing, not wrong. + + * --clean calls a file an orphan when it is in the old pkg.lst and older + than the install. **"Older" is not "no longer installed":** CMake's + install skips a file whose content has not changed ("-- Up-to-date:"), + so webkitgtk's unchanged jsc headers kept their old date. meson and + `install -p` do the same with timestamps. + * The code knew -- it re-ran the install step afterwards "to restore + current files". That step ran `bash ~/install_<account>`, the copy + 1.14.16 stopped writing, with stdout and stderr sent to /dev/null and + the exit status ignored. It failed on `No such file or directory` for + every --clean since, where nobody could see it. + +Now nothing is deleted on the mtime test's word alone. Candidates are +renamed aside in place (`<file>.pkgusr-clean`, one rename, nothing copied), +the install step runs through the same door as `script install` +(`lfs-helper build --phase install`, logged, exit checked), and then: what +it put back is current and kept; what it did not is removed; **if it fails, +every file is put back and the clean is reported as not done.** A ledger +survives an interruption and is undone at the next clean. + +Tested: an unchanged "header" the reinstall re-creates is kept, a real +orphan goes, no `.pkgusr-clean` litter; a failing reinstall puts both back +untouched and leaves no ledger; and the reapply is checked to log, to return +its status, and to use the versioned script. + +## 1.14.310 -- who INSTALLED it, not who holds it + +Reported from the machine a THIRD time, and this time with the evidence that +finally named the mechanism: + + find /usr/sbin -user p_webkitgtk + /usr/sbin/iptables -> xtables-nft-multi (six links, Sep 12 12:56) + grep iptables /usr/src/pkgusr/p_webkitgtk/pkg.lst + /usr/sbin/iptables + /usr/sbin/iptables-restore + /usr/sbin/iptables-save + +Those links were made by `cfg iptables-nft` on a different day. 1.14.308 +guarded the ownership sweep and the damage came back anyway, because the +sweep was not the one doing it. **The lease was.** + + * A build manifest is built BY TIME WINDOW: everything newer than the + build stamp. A webkitgtk build runs for hours, so whatever root wrote + in those hours -- `cfg shadow-pam` touching /usr/bin/su, `cfg + iptables-nft` making six links -- landed in `webkitgtk.files`. + * Before an install phase, `lease_files` reads that manifest and hands + the package everything in it that someone else owns. Its guard for + root's files had two doors, and **"the file carries a setid bit" was + one of them, with no lock on it**: any package whose manifest named + /usr/bin/su was given it. `chown` then cleared the setuid bit, which + is why the file came back as `p_webkitgtk 0711` every time. The user + was mid-rebuild each time they looked: the lease is held for the whole + phase. + * `lease_settle` decides whether the package keeps a leased file by + `cmp`. **For a symlink that test can never say "unchanged":** `cp -a` + copies a relative link as a relative link, so the copy in the lease + directory dangles, `cmp` cannot open it, and the failed compare reads + as "the phase rewrote it". Every leased symlink was kept. That is the + six iptables links, and pkg.lst then recorded them because pkg.lst is + an owner scan. + * And it was self-sustaining. The manifest fed the lease, the lease made + the package the owner, and the manifest filter -- which asks who owns + the file NOW -- saw the owner's own file and kept it. So the same + damage repeated on every rebuild. + +So both answers to the question asked ("did we do something wrong with the +pkg list, or is there an error in the scripts?") are yes, and they are the +same fault: **nothing anywhere asked who INSTALLS a file, only who holds it, +and that was the thing that had gone wrong.** Two facts that do not depend +on current ownership now decide it: + + * `setid.conf` gained a 5th column, the package that installs the file + (`/usr/bin/su | 4711 | root | - | shadow`). shadow may reclaim its own + su; nobody else may borrow it, and a line with no package is leased to + nobody. The old three-column readers are unaffected -- the group is + still field 4. + * **A symlink belongs to whoever owns its target.** Applied in + `lease_settle` (compared with `readlink`, as `lease_restore` already + did), in the ownership sweep, and in the manifest filter. + +### Repairing a machine that already went through those doors + + packagemanager disown --all # what it would hand back + packagemanager disown --all --run # hand it back + lfs-helper setid --run # put the setuid bits back + +`disown` is `reclaim` in the other direction: `reclaim` pushes ownership +towards pkg.lst, which is right when a package lost files and wrong when the +RECORD is what went bad. It hands back only what the two rules above prove +is not the package's, restores the mode from setid.conf (the chown that took +the file cleared it), and strikes the paths from pkg.lst **and** from the +build manifest -- because leaving them in pkg.lst means `update webkitgtk +--clean` would delete the iptables links, and leaving them in the manifest +re-arms the lease on the next build. + +Tested: the lease fixture gained a setuid file setid.conf gives to another +package (not leased), one with no package column (not leased), a relative +symlink the phase did not touch (given back) and one it repointed (kept); +the manifest filter dropping both kinds; and `disown` end to end on real +files and real accounts -- dry run changes nothing, the setuid file goes +back to root at 4711, the link goes to the owner of its target, the +package's own file is left alone, and both records come out clean. + +## 1.14.309 -- turning the machine off is a membership, and nothing re-checked it + +Asked what a good way is for the desktop user to poweroff and reboot, and at +which step. The answer was already built -- `cfg power` in services.stack -- +and it had the same weakness as everything else that sets a mode: + + * halt, poweroff and reboot become 4550 root:<collector>_power. Setuid root + so they can do the job, executable only by the group so not everyone may: + the privilege is a MEMBERSHIP, which is the same shape as seatd-<user> and + su_u_user. The desktop user is added to that group; a package user owns + files, never identity. + * /sys/power/state and /sys/power/disk are made fresh at every boot, so the + step also writes /etc/rc.d/init.d/power-perms and links it into rcS.d. + udev cannot do this: a sysfs attribute has no uevent, so no rule ever + fires for it. + * **The entry had no `expect=`.** Once cached it was never looked at again + -- and these are exactly the modes that vanish quietly: a sysvinit rebuild + reinstalls halt without the setuid bit, and a chown clears it as a side + effect the kernel never reports (1.14.308, on su). It now re-checks the + group, `4550 root:<group>` on halt, and the boot script, on every run. + * setid.conf says why those three are NOT in it: the group's name is + configured, so a `| root | -` line would keep whatever group the file + happens to carry -- p_sysvinit on a machine where `cfg power` never ran, + which would hand shutdown to whoever builds sysvinit. One place knows the + group, and it is the step. + +Answering the question directly: the step is `cfg power`, it belongs in +services.stack after the desktop user exists and after /etc/rc.d/init.d does, +and it is already there. Running it again is free: + + packagemanager stack services --only power --run + +Tested: the expect= read back out of the stack file and checked as valid shell, +and asserted to test the mode, the owner and the boot script; power.sh still +setting 4550 and still re-applying /sys/power/state at boot. + +## 1.14.308 -- chown destroys a setuid bit, and su was root's + + ls -ald /bin/su + -rwx--x--x 1 p_webkitgtk p_webkitgtk 169952 Sep 15 12:43 /bin/su + +Reported from the machine, for the SECOND time. One command explains both +halves: the kernel clears setuid and setgid on chown, so a single +`chown p_webkitgtk /usr/bin/su` changes the owner AND removes the bit that +makes su work. Nothing warns, and nothing in the file says what it used to +be. + +How the sweep reached su at all: ownership is attributed BY TIME WINDOW -- +every file newer than the build stamp -- and a webkitgtk build runs for hours. +Anything that touched su in that window (`lfs-helper setid`, a shadow +reinstall, `cfg shadow-pam`) made it newer than the stamp. It is root-owned, +and the loop's own comment says "files owned by root are fair game (the +temporary system's)". For /usr/src that is true. For /usr/bin/su it is the +opposite of true. + +Two defects, one symptom: + + * **The root-owned rule was applied in one of the two places that needed + it.** The manifest already drops root-owned paths when the phase ran AS + the package user -- a package user cannot have written a root-owned file, + so whatever is root's now was not its doing. The chown loop walked the + RAW lists again and took exactly the files the manifest had refused. + * **A setuid or setgid file is never the sweep's to take**, whoever ran the + phase and whatever the time window says. Those modes are the system's, + declared in setid.conf, and a chown silently destroys them. The sweep now + skips any file with a mode of the form [1-7][0-7][0-7][0-7] and says so. + +Also, because it turned every update into a keypress: `pm-install` asked +"check that p_imv is the right account" on every run. That is a FIRST-INSTALL +question -- a typo in the account name scatters a package across the wrong +home -- and it is worth nothing on an update, where the account exists and +already owns the package's files. It is now asked only when there is no +pkg.lst and no .pkgusr-installed for that account. A prompt people press +through has stopped protecting anything. + +ON THE MACHINE: `packagemanager stack services --only shadow-pam --run` is the +repair -- its expect= is exactly `test -u /usr/bin/su && stat -c %U = root`, +which is why it can see the damage. By hand it is +`chown root:root /usr/bin/su && chmod 4711 /usr/bin/su`. Worth a look for +others: `find / -xdev -perm /6000 ! -user root -ls`. + +Tested: the mode test against 4711, 2755, 6755 (skipped) and 0711, 0755 +(taken); both guards asserted in the source; setid.conf still declaring su as +4711 root; and the prompt's decision against an installed package (silent), an +account with nothing recorded (asks) and no account at all (asks). + +## 1.14.307 -- "updating from its repo" for a package with nothing to update to + + packagemanager update imv + imv: git package -- updating from its repo. + Dry run -- to proceed, re-run with --run + +Everything there is correct -- `--run` is the missing word, as with every +command here that changes the system -- and it still does not answer the +question the person is asking, which is "will this get me anything?". imv is +pinned to v4.3.0 and v4.3.0 is the newest tag exec64/imv has: the rebuild +produces the commit that is already installed. The command works, does +exactly what it says, and changes nothing. + +`update <git-pkg>` now says which of the three it is, before the dry-run line: + + imv: pinned to tag v4.3.0, which is the newest released -- this rebuilds + the same commit + zz: pinned to tag v4.1.0; v4.3.0 is out + a rebuild stays on v4.1.0. To move it: + packagemanager git-ref zz v4.3.0 --run + sway: master moved 4e2a1b3 -> 9c7f0aa (12 commit(s)) -- this brings it in + +`--no-git-check` skips the remote when there is no network. + +Tested against the real exec64/imv (newest tag -> "rebuilds the same commit") +and a copy pinned to v4.1.0 (-> names v4.3.0 and the command to take it). + +## 1.14.306 -- "phased" means it ends in the runner + + packagemanager update imv + imv: git script isn't phased -- it won't fetch/rebuild cleanly. + Regenerate it with: packagemanager template imv --git <repo-url> + +The script was perfectly current -- format v17, build_pkg, install_pkg, and +`. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"` at the end. The test asked +whether it defines `unpack_pkg`, which is the RARE OVERRIDE that replaces the +built-in unpack entirely. A correct script does not define it: the runner's +own unpack is what clones git. So every properly phased git package was told +its script was broken. + +The advice was worse than the warning. `template --git` rewrites the file, +and imv's build_pkg carries `-D c_link_args=-licuuc` -- a fix found once and +not written down anywhere else. Following the suggestion would have deleted +it and the next build would have failed at link time for a reason nobody +would connect to this message. + + * `_is_phased_script()` asks what actually matters: does it END IN THE + RUNNER (`pkgusr_run`, PKGUSR_LIB/lfs-phases, or the `### phases:` header)? + That is the thing that makes fetch, resume and single-phase re-runs work. + * When a script really is unphased, the message now says what a phased one + looks like, and warns that `template` does not keep the build commands -- + write it to a new file and move them across. + +Tested: the machine's own imv script (recognised as phased), an old-style +script that clones inline (not phased), and the whole path end to end against +the REAL repository -- `git-status` on imv reports "tag, nothing newer +released", which is true: v4.3.0 is the newest tag exec64/imv has. + +## 1.14.305 -- a branch is not a point in time + +Asked whether `update` could follow a git package that tracks master, and +show the releases for one pinned to a tag. The first half was not a missing +feature, it was a bug: + + * **`git checkout master` does not move the branch.** The unpack phase + fetches (`git fetch --all --tags --prune`) and then checks the ref out -- + and on an existing clone git answers "Already on 'master'" and stays + exactly where it was. So a package pinned to a branch fetched the new + commits, rebuilt the OLD ones, installed them and reported success, for + ever. It now asks the repository what kind of ref it has: a remote + branch is taken with `checkout -B <ref> origin/<ref>` and says what moved + (`master: b21bd64 -> 6429c4b (3 new commit(s))`); a tag or a commit is + checked out DETACHED, because that is what it is and it must not move; + with no ref at all it follows origin's own default branch. + * **`packagemanager git-status [pkg]`** -- for each git package: the ref, + what kind it is *according to the remote* (guessing from the string is + how a tag called `main` goes wrong), the local and remote commits, how + far behind, and for a tag-pinned package every newer tag. `--local-only` + skips the network. + * **`packagemanager git-ref <pkg> [<ref>]`** -- with no ref it lists the + remote's branches and tags with the current one marked; with one it + rewrites `git_ref=` in the install script (backup kept, dry run unless + `--run`), refuses a ref the remote does not have, and NAMES THE STACK + FILE that also carries it -- editing only the script is undone by the + next stack run. + * `update`'s "N packages follow GIT -- left alone" now answers the question + instead of restating it: which are behind their branch (with the command + to rebuild), which are pinned to a tag that has been superseded (with the + command to move), and `--no-git-check` for no network. + +Tested against two real repositories built in the fixture: one package +following a branch that gained a commit (reported as moved, old -> new sha), +one pinned to 2.0.0 with 2.1.0/2.2.0/2.3.0 released (all three offered, +newest suggested); `git-ref` listing, dry-run rewrite, and a refusal for a +ref the remote does not have (rc 1). The branch bug itself is reproduced in +the suite: plain `checkout` leaves the clone where it was, `checkout -B` +moves it. + +## 1.14.304 -- a library something still links against is not an orphan + +From the machine: `pkg-config --modversion webkitgtk-6.0` said 2.50.5 while +the package said 2.52.5. Not a failed install -- the dates tell the story: + + libwebkitgtk-6.0.so.4.13.8 Sep 13 the GTK4 port (2.50.5) + libwebkit2gtk-4.1.so.0.21.9 Sep 15 the GTK3 port (2.52.5) + +The rebuild switched PORTS. The 6.0 files are old-version files this package +still owns and the new version never rewrote -- which is exactly what +`update --clean` deletes, and anything built against libwebkitgtk-6.0.so.4 +would have broken on the spot, with no warning and no way back but a rebuild. + +A package-user system records who INSTALLED every file and nothing about who +USES one. `ldd` answers it for one file; nothing answered it for a library. + + * **`packagemanager used-by <package|soname|path>`** -- every ELF that + records that soname in DT_NEEDED, grouped by the package user that owns + it. grep first (a soname is a literal string in .dynstr, so grep over + /usr/lib is seconds), then readelf only on the hits, because the same + string also appears in scripts, .la files and other libraries' rpaths. + The SONAME is READ from the library (libwebkitgtk-6.0.so.4), not guessed + from the file name (libwebkitgtk-6.0.so.4.13.8). + * **`--clean` now asks that question before it deletes.** A library still + recorded by a file another package owns is KEPT, the packages are named, + and the rebuild command is printed; the symlinks that point at it are kept + with it, or the soname it is found by disappears anyway. A library + nothing links against is swept as before. Files of the package being + cleaned do not count -- they go with it. + * `_needed_by`'s scan directories were a DEFAULT ARGUMENT, bound at import + and therefore impossible to override -- which is how the first test of it + appeared to find nothing. Resolved at call time now. + +Tested for real, not by grep: gcc builds a library with a soname and a +consumer that records it, the consumer is chowned to another account, and +`_orphan_sweep` is run twice -- with the consumer present (library KEPT, no +files removed) and with it gone (both files swept). That fixture is now a +block in the suite, and skips itself where there is no gcc. + +## 1.14.303 -- the suite could not run, so nothing it guards was guarded + +Asked "anything that needs fixing?", so the suite was run instead of guessed +at. It could not run at all, and had not been able to for some time. Five +things, in the order they were found: + + * **`lfs` would not start on Python 3.11.** One f-string reused its own + quote character inside the braces (`f"...{f(lfs, "progress", ...)}"`), + which is legal only from 3.12. `lfs` is the ONE tool that runs on the + HOST, where Python is whatever the distribution ships -- Debian 12 is + 3.11 -- so it did not misbehave there, it refused to start, with a + SyntaxError pointing at a debug print. It also took the whole suite with + it: the preamble builds a real package through a generated script, so + every block downstream failed. That one line was 12 of the failures in + the first block run. + * **The cleanup trap ate the tally.** `trap cleanup EXIT` is inherited by + every subshell, and the suite runs blocks inside `( )` and `$( )` + constantly; the first one to exit removed $T and the counter directory + out from under the run. Every later `ok()` printed a shell error instead + of counting, and the final line read `PASS: FAIL: SKIP: (final)` -- + a suite that reports nothing, which is worse than one that reports a + wrong number. `$$` is the same in a subshell, `$BASHPID` is not; the + trap now returns unless it is the shell that set it. Counters also moved + to a name nothing globs for and are recreated on write. + * **The launcher test had drifted from the launcher.** It neutralised `su` + with `sed '/^exec su - /'`, which assumed one ending; since 1.14.295 the + `exec su` is indented inside an `if` and there is a second path through + `script`, so the sed matched neither and the REAL su ran ("user u_a does + not exist"). It now stubs `su` first in PATH, which tests whatever shape + the launcher has, and asserts the two contracts that were added without + tests: a pty when there is no terminal, and a `{url}` default page. + * **Every stack file said 1.14.292 while the tools said 1.14.302.** The + suite asserts they match ("they ship as one artifact, so they carry one + number") and had been unable to say so for ten releases. Stamped -- and + `make stamp` now does it from `lfs --version`, because a rule only a human + remembers is a rule that drifts. + * **avoid.stack carried debris from the split out of sway.stack**: a second + `# stack-version:` line under a `# sway.stack -- a working Sway desktop` + header halfway down the file, and two orphaned section markers ("# 9. the + nimbin2 family", "# 10. wire the session together") with nothing after + them. All comments, all inert, all misleading to read. + +After: **751 PASS / 80 FAIL / 1 SKIP**, with a final line that prints. The 80 +are this container, not the tools -- it is not a package-user system, so the +/usr/lib/pkgusr wrappers, the book and the chroot blocks have nothing to test +against. `make check` on the machine is the number that counts. + +STILL BROKEN, not fixed here: `--only` / `--from` generate a subset script +that can cut a here-document in half ("here-document at line 667 delimited by +end-of-file") and lose helpers a previous block defined (`detail: command not +found`). The documented limitation is that a block may need what an earlier +one computed; a truncated here-doc is a bug in the extractor. Until it is +fixed, a selection can report failures a full run does not. + +## 1.14.302 -- dev.stack: gdb + +Asked for plainly: "we need gdb in a stack". None of the six fitted. Every +stack here answers one question -- services = what starts at boot, network = +how the machine gets an address, sway = the desktop, diag = is the GPU doing +what I think, media = what plays, vision = computer vision -- and a debugger +is none of them. Putting it in the nearest one is how a stack stops meaning +anything. + + * **stacks/dev.stack**: `book gdb expect="test -x /usr/bin/gdb"`, and the + question it answers is "the program crashed and I want to know why". + Nothing depends on it, so it is safe to build last and safe not to build + at all -- the other reason it is not in a stack the desktop needs. + * gdb's only "required" dependency in the book is the python module `six`, + and that is for gdb's own scripting at runtime; the rest of the list + (guile, doxygen, valgrind, rustc, SystemTap) is for gdb's test suite. + The build line comes from the book page, so nothing is repeated in the + stack. + * The file says out loud that strace, ltrace and valgrind are the obvious + neighbours and are NOT there: an entry in a stack is a package built on + every replay of it, so they wait to be asked for. + * BUILD.md runs it after services. + +Tested: the file through _parse_stack_file -- `include avoid` pulls in all 78 +exclusions and the one real entry parses as book/gdb with its probe. NOT +tested: the build itself (this box has no BLFS book). + +## 1.14.301 -- `packagemanager service`, and an exit code that was thrown away + +Asked for after `cfg bootscripts` installed sshd: a way to see what services +there are and to start one without enabling it. SysV has always kept those +two apart -- the init script starts it, the rc<N>.d links decide whether boot +does -- and nothing here made that easy to use. + + packagemanager service # all of them, at boot, running + packagemanager service sshd start # now, enabled or not + packagemanager service sshd enable|disable # boot only + packagemanager service start sshd # either order reads naturally + + * list: `at boot` (yes / other runlevel / no), `running` (asked of each + script, LSB exit code), and which runlevels. `functions`, `rc` and + `template` are not services and are not listed -- listing them invites + starting them. `--quick` skips the running column. + * start says "not enabled at boot -- starting it anyway" rather than + refusing. That was the point of the request. + * **disable records the links it removed** in /var/lib/pkgusr/services, so + enable restores exactly those. The priority number (S30, S26) comes from + the blfs-bootscripts Makefile and is the one thing that cannot be worked + out again; with no record, the number comes from a surviving K link and + the runlevels from the script's own `Default-Start:` header. disable + removes only the S links and says it has not stopped anything. + * It also lists /lib/services/* separately -- run by ifup, never on their + own, named by SERVICE= in ifconfig.<iface>. Not saying so is how "dhcpcd + is installed and nothing runs it" happens. + * **main() threw away every command's return value**, so packagemanager + exited 0 whatever happened -- including `service <name> status` on a + stopped service, which is the one command people put in an `if`. A + non-zero int now becomes the exit status; anything else is unchanged. + +Tested against a built fake /etc/rc.d (three scripts with INIT INFO headers, +one running, one not, one unlinked, plus a `functions` file): list; start on a +disabled service; enable from nothing (header runlevels, priority 50, seven +links); disable then enable (record written, K links untouched, links restored +identically); reversed word order; an unknown name; and the exit codes 0 / 3 / +1 / 0 for running, stopped, missing and list. NOT tested: a real boot. + +## 1.14.300 -- the build tree is one level deeper than the home + +`cfg bootscripts` from 1.14.299 said + + !! no unpacked blfs-bootscripts tree under /usr/src + +on a machine where it was right there. lfs-phases builds in BUILD_ROOT, +which is `$HOME/src` whenever the package user has one, so the tree is +/usr/src/p_blfs-bootscripts/**src**/blfs-bootscripts-<date>. The step looked +in the home and in the base, never a level deeper, and then told the reader to +unpack something that already existed -- the worst kind of error message, the +one that sends you to redo work that is done. + + * Four patterns now, newest wins: `<base>/*/src/`, `<base>/*/`, `<base>/` + and the sources directory. Then a `find` bounded at depth 4 for a layout + nobody thought of. + * **The tarball is enough.** A build tree can be cleaned away and these + targets only copy files, so with no tree at all the step unpacks + blfs-bootscripts-*.tar.* from $LFS_SOURCES_DIR (/sources) into a + temporary directory, installs from there, and removes it on exit (trap). + It says that those files land owned by root rather than by the package + user, because that is the one thing this path does differently. + * When it really finds nothing it now prints WHERE IT LOOKED, so the next + failure is diagnosable instead of another guess. + +Tested: the real layout (<home>/src/blfs-bootscripts-*, found and installed +from), and the tarball fallback with the tree deleted (unpacked, installed, +temp directory gone afterwards). + +## 1.14.299 -- a package and the thing that starts it come from different tarballs + +"dhcpcd does not start at boot; did we forget its init script?" There is no +init script to forget: BLFS ships no /etc/rc.d/init.d/dhcpcd at all. The only +target is install-service-dhcpcd (/lib/services/dhcpcd), which services.stack +already installs, and that script is started by nothing on its own -- the LFS +`network` script reads /etc/sysconfig/ifconfig.<iface> at boot and hands each +interface to the service ITS file names. With no such file the machine comes +up with dhcpcd installed, its service script in place, and no address. +`lfs config network dhcp` writes no interface files on purpose, and nothing +else wrote one either. + + * **stacks/bootscripts.conf** -- the services.stack install= list as a + MAPPING: `package | proof | targets | creates`. proof and creates take + several comma-separated alternatives (a daemon that moved between /sbin + and /usr/sbin is still installed; /lib/services and /usr/lib/services are + the same service). + * **cfg bootscripts** installs the targets whose package IS here and whose + script is NOT, skipping packages that are absent (an init script for + something uninstalled is a symlink in rc?.d that fails at boot), and then + **checks the disk rather than make's exit status** -- a target that + prints nothing and installs nothing is the failure that matters. make + runs as the tree's owner, so the files keep saying which package user + installed them, with root as a fallback that says so. + * The same step writes /etc/sysconfig/ifconfig.<iface> for a WIRED + interface that has none: ONBOOT, IFACE, SERVICE=dhcpcd, DHCP_START/STOP. + That file holds no secret, which is exactly why it can be written here -- + a wireless one needs an SSID and a key, so those are still left to + `check net-wifi`, which prints the file to paste. + * The entry carries `expect=`, re-checked when cached, so a package + installed later is noticed on the next run instead of on the next boot. + +Tested: a fake bootscripts tree with three targets, one package installed, +one absent and one target that lies (prints success, installs nothing) -- the +absent package skipped, the two real ones installed, the liar caught by the +after-check and the step exiting 1; a second run changing nothing; the +interface file written for eth0, wlan0 left to net-wifi, and an existing file +naming another service left alone. NOT tested: a real blfs-bootscripts +Makefile, and a boot. + +## 1.14.298 -- we wrote the file that unsets the locale + +The i18n.sh on the machine was not BLFS's. It was OURS -- +`lfs-helper install_shell_startup_files`, which wrote: + + # Locale. Set LANG here (e.g. en_US.UTF-8); an unset LANG means the C + # locale, which is a choice, not an accident. + if [ -z "${LANG:-}" ] && command -v locale >/dev/null 2>&1; then + : # export LANG=en_US.UTF-8 + fi + +The export is commented out, and the comment argues that this is deliberate. +It is not a choice anyone makes: nothing else on the system sets LANG either, +so every machine built with these tools sits in the C locale until a terminal +complains, and the only repair is to hand-edit a generated file. The three +previous releases chased this through /etc/profile and BLFS's own placeholder +without ever looking at who wrote the file. + + * lfs-helper writes an i18n.sh that reads /etc/locale.conf -- the file LFS + uses and the one `cfg shell-env` now writes from `lfs config locale`. It + carries `# written by lfs-helper`, and with no locale.conf it does + nothing, exactly as before. `_write_if_absent` still keeps an existing + file, so this only helps new systems. + * shell-env.sh therefore recognises all THREE writers of that file as ours + to replace when it produces no LANG: its own marker, `# written by + lfs-helper`, the book's `Begin /etc/profile.d/i18n.sh`, and -- for + machines already built -- the commented-out line `: # export LANG=` + itself. A file nobody here wrote is still left alone and reported. + * The replacement keeps `LESSCHARSET=utf-8`, which lfs-helper's version set + and less(1) reads instead of the locale. + +Tested: shell-env against a byte-for-byte copy of the machine's i18n.sh +(recognised, backed up, replaced, LESSCHARSET kept, LANG then set from the +locale.conf it had just written), and lfs-helper's new fragment sourced in an +empty environment with a locale.conf present (LANG set, rc 0) and absent +(nothing, rc 0). + +## 1.14.297 -- i18n.sh was already there, and /etc/profile.d belongs to nobody + +1.14.296 would not have fixed the machine it was written for. `ls +/etc/profile.d` showed i18n.sh already present (root, Sep 4, from the BLFS +page) -- so "exists and is not ours, left alone" would have been the whole of +it, and $LANG would still have been empty. The file was never the problem: +it does `. /etc/locale.conf`, and /etc/locale.conf did not exist. + + * **i18n.sh is judged by what it produces, not by who wrote it.** It is + sourced in an empty environment (TERM=xterm, because both the LFS and + BLFS versions hard-code C.UTF-8 when TERM=linux) and the LANG that comes + out is compared with the one this machine should have. Right -> left + alone, whoever wrote it. Wrong, and it carries the book's header -> + backed up and replaced. Wrong and hand-written -> left alone and + reported. locale.conf is written FIRST, so the book's own file usually + starts working and is never touched. + * **Root writes it, root owns it.** /etc/profile.d/pkgusr-system.sh was + owned by p_gstreamer10: `env --system` opens the existing file for + writing, which keeps its owner, and it was first created inside a build + window. A package owning a file no package installed is wrong on a + package-user system, and every login shell sources that directory -- + whoever may write there runs code as everyone who logs in. `env + --system`, shell-env.sh (dircolors, bashrc, locale.conf, i18n.sh) and + app-users.sh (shared-dirs.sh, xdg-runtime-share.sh, /usr/bin/xdg-runtime-share) + now re-assert root:root on every run, as the PAM file already did. + * **/etc/profile.d joins SHARED_DIR_PATTERNS.** rust, bash-completion, + gettext and the book's own config pages each drop ONE file in it, so the + first package to create it owned it -- p_libnice here -- and every later + package had to be granted it by hand. + * Permissions there are right as they are: /etc/profile SOURCES these + files (`. $script`), it does not execute them, so 644 is correct and +x + would mean nothing. Contrast /etc/pam.d/su_u_user, which pam_exec really + does run and which app-users forces to 755. + +Tested: the locale block against a stock BLFS i18n.sh with no locale.conf +(writes locale.conf, leaves the script alone -- the machine's own case), an +i18n.sh that yields nothing with locale.conf present (backed up, replaced), +and a hand-written one (left alone, reported); is_shared_dir on +/etc/profile.d with and without a trailing slash. NOT tested: the chowns on +a system where the files are really owned by a package user. + +## 1.14.296 -- nobody set $LANG + + warning: foot: 'C' is not a UTF-8 locale, falling back to 'C.UTF-8' + +Nothing was built wrong. LFS's own /etc/profile reads /etc/locale.conf, but +BLFS's "The Bash Shell Startup Files" page REPLACES /etc/profile with one that +reads /etc/profile.d/*.sh instead -- and i18n.sh, the piece of that page which +exports LANG, is a separate block, easy to walk past. Take that page (this +machine did: its /etc/bashrc is the BLFS one) and the locale quietly falls back +to C. `lfs config locale` was set, and nothing after the chroot ever read it. + + * shell-env.sh -- the step that exists because that page is config and no + package stack runs it -- now also does the locale, and writes BOTH files, + so it works whichever /etc/profile is installed: /etc/locale.conf + (`LANG=`) and /etc/profile.d/i18n.sh (sources it, exports LANG, and + LC_ALL when set). + * The locale is taken, in order, from $LFS_LOCALE, /etc/locale.conf, + `locale` in /usr/share/lfs/config.json (what `lfs config locale` wrote), + then en_US.UTF-8, then C.UTF-8 -- and only if `locale -a` really has it. + **A name is not a locale until glibc has built it**: locale.conf may name + one nobody generated, which looks exactly like setting nothing. That is + reported with the localedef line to fix it, as glibc's package user -- + the locale archive is glibc's, and root would leave root's files in it. + * An existing /etc/locale.conf is never rewritten; a hand-written i18n.sh + is left alone. A non-UTF-8 choice is reported, not overridden. + * services.stack gives `cfg shell-env` an `expect=` -- /etc/bashrc, + /etc/locale.conf and i18n.sh. Without it a machine that ran this step + before the locale was part of it stays `done (stack cache)` for ever and + the terminal keeps complaining with nothing to say why. + +Tested: the locale block with a de_DE.UTF-8 asked for and not installed (it +says so, names the localedef line, falls back), run twice (second run changes +nothing), the generated i18n.sh sourced in a shell with LANG unset (sets it, +returns 0), and a hand-written i18n.sh left alone; the expect= expression run +as the stack runs it. NOT tested: a real login, and a machine that has +de_DE.UTF-8. + +## 1.14.295 -- a menu entry has no terminal, and su wants one + +Three from the machine after 1.14.294: the .desktop entries did nothing, the +/etc/bashrc hook refused to apply, and /root filled with tarballs of homes +that held nothing. + + * **The launcher now allocates a pty when there is no tty.** Run from a + terminal, `~/bin/reddit` worked; clicked in the menu it did nothing -- + `su` fails without a terminal and its message goes to a stderr nobody + reads. The old hand-made entry worked around it with + `Exec=script -c 'reddit' /dev/null`, which is the same fix in the wrong + place: every entry had to remember it. The launcher now branches on + `[ -t 0 ] && [ -t 1 ]` -- `exec su` as before with a terminal, otherwise + `script -qec "$(printf '%q ' su ...)" /dev/null`, which gives su a pty, + sends the typescript to /dev/null and returns the program's own exit + status. A non-zero status with no terminal to print to is passed to + `notify-send` when it exists. + * **Menu entries get `Categories=`** -- Network;WebBrowser; for the + browsers, AudioVideo;Video;Player; for mpv, Utility; otherwise. Without + one an entry is "Other" at best and invisible at worst. + * **/etc/bashrc is recognised by `# Begin /etc/bashrc`**, its first line in + both the BLFS file and the one shell-env.sh writes. 1.14.294 looked only + for shell-env.sh's own line, so a machine carrying the BLFS file got the + line printed and the variables stayed unreachable from a terminal. One + line is appended, after a backup, starting with a newline -- the file on + this machine ends `# End /etc/bashrcCK=$(tput setaf 0)`, which is what an + append without one leaves behind. + * **No archive for a home that was never used.** The tarball exists for a + browser profile -- logins, cookies, history, downloads, which + `userdel -r` would take with it. A home holding only the .bash_profile + symlink these tools made and an empty Shared has none of that, so + `_home_is_untouched` skips it: a purge of fourteen fresh accounts now + leaves nothing in /root. When archives ARE kept, purge ends by saying + how many, how to look inside one and how to drop them. + +Tested: the launcher with and without a terminal, su replaced by a printer +that reports whether it got a tty (no tty -> wrapped in script, su sees a +tty; tty -> exec su direct), arguments and the default page surviving the +extra quoting, and a failing su's exit status coming back out; the archive +skip against an untouched home and one with a .mozilla in it, dry and real; +the bashrc hook against a BLFS-style file, twice (appends once, backs up +once). NOT tested: a real su from a real menu. + +## 1.14.294 -- the variables were right and invisible; simpleicons dropped OpenAI + +From the machine after 1.14.293 ran clean: `~/Shared` held the ten new links, +`shared-dirs.sh` was correct, and `$SHARED_<TAB>` still offered only +BROWSER_BIG, BROWSER_MINI, CLAUDE -- the three from the login BEFORE the +change. + + * **/etc/profile.d is read by LOGIN shells.** A terminal opened inside + sway is not one: it reads ~/.bashrc and shows whatever the session + inherited at login. The step now also sources the fragment from + /etc/bashrc -- the file every interactive shell reads, and the one + shell-env.sh writes, recognised by its `Based on the BLFS "Bash Shell + Startup Files" page` line. Anything else is left alone with the line + printed. `APP_USERS_BASHRC` overrides the path for testing. + * **The fragment now takes variables away too.** An account removed after + login left `SHARED_X` in the environment pointing at nothing, inherited + by every shell the session started -- so a correct removal looked like it + had failed. It drops every `SHARED_*` whose directory is gone (`env` + + sed + `eval`, POSIX, tested under dash) before exporting what is there, + and is safe to source again at any time: `. /etc/profile.d/shared-dirs.sh` + refreshes a running shell. + * **simpleicons.org has dropped `openai` and `grok`** (confirmed: both 404 + in the upstream repo, `claude` still there). That is why chatgpt, nrwgpt + and grok got no icon. Those three lines now carry Wikimedia URLs in the + stable `Special:FilePath/<File.svg>` form. + * **`#RRGGBB` at the end of the icon field** paints a one-colour SVG after + it is fetched (`_recolor_svg`): fills in attributes and in `style=`, + `fill="none"` left alone, a root `fill` added when there is none. It + replaces simpleicons' `/slug/colour` service, which cannot help for a + brand it no longer carries -- nrwgpt is the ChatGPT mark in sky blue. + The `.from` stamp records URL+colour, so changing only the colour + refetches. + * A launcher that is not ours is still left alone, but now says + `rm <path> and run this again` -- launchers from an older version are not + recognised either, so "left alone" had quietly kept the OLD behaviour (no + default page, no --app-id) on chatgpt, nrwgpt, grok, mercury, deepseek + and youtube. + +Tested: the fragment under bash and dash with a dead variable, a moved one +and a dangling link (dead dropped, moved corrected, dangling skipped); the +/etc/bashrc hook against a shell-env.sh file and a foreign one; `_recolor_svg` +on a real simple-icons file and on one with attribute, `style=` and `none` +fills; the launcher suite re-run unchanged. NOT tested: the Wikimedia +downloads (this machine cannot reach them). + +## 1.14.293 -- undoing the application accounts, and a default page per launcher + +Three things the last release left half-done: removal stopped at the account, +a search launcher with nothing typed landed on an empty search page, and the +conf did not say which window is which. + + * `packagemanager user purge [--run] [--yes] [--keep-home] [--no-archive] + [--accounts a,b] [--conf F]`: the whole undo. Every account in + app-users.conf, plus **strays** -- accounts with the user prefix whose + home is under `appuser_home` but whose line is gone (exactly + u_browser-mini/u_browser-big after the layout change) -- each through + `user delete`, so each home is archived to /root first. Then what no + per-account command could remove because the next account still needed + it: dead ~/Shared links, the hub and the icon dir when empty, + /etc/profile.d/shared-dirs.sh, /etc/profile.d/xdg-runtime-share.sh, + /usr/bin/xdg-runtime-share (each only if it carries the app-users + marker), /etc/pam.d/su_u_user, the `pam_exec` line in /etc/pam.d/su + (backup first), appuser_home when empty, and the session group last. + Kept on purpose: the /root archives and app-users.conf. + * `user delete` now removes **every dangling link** in ~/Shared, not only + the ones pointing into this home. A home moved after the link was made + (usermod -d, or the move to /home/shared_user) left the link behind, and + shared-dirs.sh enumerates that directory -- which is why + `$SHARED_BROWSER_MINI` outlived the account. Both commands end by + naming the SHARED_* variables an already-open shell still has, and the + `unset` for them: that file is read at LOGIN. + * `cfg app-users` takes a shared directory away again: a line changed to + `dir=no` has its ~/Shared/<Name> link removed when the target is empty + or dangling, and is reported, not touched, when it still holds files. + * Launcher templates take a default: `...?q={https://www.reddit.com/}`. + With something typed it is the search; with nothing typed the word + becomes what is inside the braces, instead of the URL cut at its `?` + (`/search/` -- a search page for a search nobody made). Plain `{}` is + unchanged. The fallback cannot contain `|`: that is app-users.conf's + field separator. `_write_launcher` now fills a plain template with + .replace() rather than %-formatting bash full of `%02X` and `${w%%...}`. + * app-users.conf: `--app-id <name>` spelled out on every browser line + except the four generic ones (browser-mini, browser-big, google, + duckduckgo), which keep the automatic one. Shared directory (and + `--download-dir ~/Shared`) on the accounts that want it; **not** on + dokus, reddit and youtube -- what they download stays in the account. + Default pages for google, duckduckgo, github, codeberg, reddit, youtube. + Icons: ARD and arte from Wikimedia, nrwgpt as the OpenAI mark in sky + blue (`https://cdn.simpleicons.org/openai/38BDF8`). + +Tested: launchers generated and run with su replaced by a printer (default +page with no arguments, search with `&` and an umlaut, `{}` passthrough, `~/` +expansion); the conf through the step with a stubbed packagemanager (every +line parses, the four accounts aggregate as expected); the dir=no cleanup +against an empty, a dangling and a non-empty link; purge dry and real against +a built fake state -- two accounts (one a stray), dead links, the marker +files, the su line, the group -- verified afterwards by hand. NOT tested: a +real su, and the icon downloads (this machine cannot reach the CDN). + +## 1.14.292 -- application accounts: several launchers, {} arguments, icons, delete + +From the user's app-users.conf (already written as `user | name | app ...`, +which the step could not read -- hence `~/bin/browser-mini` still running +u_browser, an account never created): + + * app-users.conf: `account | name | app | share | dir | desktop | launcher | + groups | icon`, one line per LAUNCHER. Accounts are aggregated: the + strongest share, any dir, all groups; created/set up once. Then + `packagemanager user launcher <account> <name> --app ... [--icon]` per + line. The old 7-column format (header `name`) still reads. + * Launcher (packagemanager `_write_launcher`): the argv is built in the + script -- a word `{}` = the arguments; `{}` inside a word = URL-encoded + search term, or with no arguments the word cut before its `?`; no `{}` = + nothing passed (claude's "a second address:" came from an extra '' + argument). `~/` in the app = the ACCOUNT's home. browser-mini/big get + `--app-id <name>` when missing. `exec su - u -c 'exec "$0" "$@"' -- argv`. + * Menu entry: named after the launcher; Exec %U only with {}; Icon from + `_fetch_icon`: URL, simpleicons slug, empty -> the name, - -> none; fetched + once into ~/.local/share/icons/lfs-apps (.from records the URL). + * `user delete <account> [--run] [--keep-home] [--no-archive]`: application + accounts only (prefix, home under appuser_home or the package tree), + refused while it runs anything; home archived to /root/<u>-home-<date> + .tar.gz (0600) first; removes its launchers/menu entries/icons (only ones + that run as it), ~/Shared links into its home, its su_u_user line, + account, group, home. `user remove-launcher <name> [--run]` for one + launcher. Both remind that the conf line must go too. + * The shipped app-users.conf is the user's list, corrected: google/ddg/ + youtube/reddit/github/codeberg search URLs, chatgpt/gemini pointed at + their own sites (both were claude.ai), codeberg .org, download dirs + ~/Shared (were /usr/src/u_*), media accounts full, broken ard/zdf/mpv + lines completed, icon slugs where the name differs (openai, + googlegemini). + +Tested: generated launchers run with su replaced by a printer (search term +with umlaut and &, no args, fixed URL with an extra arg, ~/ expansion), the +conf parser against the shipped file with a stub packagemanager, delete dry +and real (archive, launchers, link, pam regex), remove-launcher leaving a +hand-made file. NOT tested: real su, icon downloads. + +On the machine: u_browser-mini and u_browser-big stay until deleted. + +## lfs-backup 1.3.0 -- merge + +`merge <archive> [--plan f] [--only/--skip]`: the archive's existing .new +files one at a time, `diff -u` (MERGE_LINES, default 60), then k (keep live, +remove .new), n (take the backup's CONTENT; live owner/mode stay; .new +removed), e (MERGE_TOOL or vimdiff live .new, else $EDITOR .new; asks again), +s, q. passwd/group/shadow/gshadow refuse n. Answers from /dev/tty, or stdin +without one. Tested with piped answers (k, n twice with a 600 live file, +refusal for passwd); suite block extended. + +## lfs-backup 1.2.2 -- plan lines keyed by kind; absolute targets + +The user's first plan: `~/.bashrc` etc. exist in [user] and [root]; the plan +named them identically and restore took the FIRST matching line, so root's +file followed the user's line. And `~/` targets read as /root to someone +checking as root (953 "same" ALE files were the main user's, correctly). +Section column is now `kind:section`, the lookup key is kind:name, and the +target column is the resolved path (~/ still accepted). Older plans are +refused with "write a new one". Tested: user line skipped, root line kept. + +## lfs-backup 1.2.1 -- secrets restored 0600-or-less + +Second real backup: excludes worked (1043 files, 9.3 MB; no .git, test, +*~, .netrwhist). The host modes are unchanged (gshadow 644, Wi-Fi keys +644/664), so restore now drops group/other bits for shadow, gshadow, +wpa_supplicant/*, ssh host keys, ~/.ssh/id_*, authorized_keys, and says so. +Tested with a 664 key and a 644 gshadow -> 600 each. + +## lfs-backup 1.2.0 -- restore plan, excludes, ownership by kind + + * `plan <archive> [file]` writes `action | state | section | name | target` + per file (identical -> skip); `restore/clean --plan file` follow it: skip + lines are left alone, the target column may be ~/... (home of that kind) + or absolute. Unknown actions / short lines stop with the line numbers. + * `!pattern` lines exclude matching names everywhere (bash [[ == ]], so * + crosses /). Shipped: plugin .git and test trees, *~, .netrwhist. + * Owner on restore by KIND: [root] root:root always; [user] the main user + (group kept only if it is another existing group); [system] recorded + names that exist, never the main user, else root. + +From the user's first real backup (manifest reviewed): root's .bashrc and +.bash_profile were links into ~/LFS/lfs-config and recorded as n76310's -- +by name that restores a root dotfile the main user can edit. Also on the +HOST, not the tool: /etc/passwd, /etc/group, /etc/hosts owned by uid 9999 +mode 664/774; /etc/gshadow 644 (world-readable hashes); most +wpa_supplicant PSK files 644/664; /etc/vimrc owned by n76310 664. +/usr/share/vim/vimfiles held only cmake's files -- dropped from the config. + +Tested: plan with a skip and a moved target, root file ownership, clean by +plan, a malformed plan line; suite block extended. + +## lfs-backup 1.1.0 -- named sections, --only/--skip, clean + + * `[user:vim]`, `[system:accounts]` ...: a section's NAME (default: its + kind) is recorded per file in the manifest (7th column; older archives + fall back to the kind). `--only a,b` / `--skip a,b` select by name or + kind for create, restore and clean. `sections [archive]` lists them. + * `clean <archive> [--run]`: removes <target>.new for the archive's files + only when its content is still what restore wrote; an edited .new is + kept and listed; .new files from anything else are never touched. + * The shipped backup.conf is the user's list, sectioned, plus vim + (~/.vimrc, ~/.vim for user and root; /etc/vimrc, /usr/share/vim/vimfiles) + and /etc/pkgusr/backup.conf itself. /etc/rc.d is NOT taken whole: its + init.d scripts and rc?.d links come from the bootscript packages. + +Tested in the sandbox (sections from config and archive, --skip, --only, +clean with one edited .new); suite block extended. + +## 1.14.291 -- lfs-backup replaces lfs-config + +lfs-config was a git-repo manager (init/pull/push/adopt/mirror/undo). The +user wanted a backup: one list of files, carried to a new system, put back +without replacing anything. + + * /etc/pkgusr/backup.conf: [user] (~ = PKGUSR_SYS_MAIN_USER's home, read + from the environment or /etc/profile.d/pkgusr-system.sh), [root], + [system]; globs; a directory takes its files. Ships the user's list plus + suggestions (passwd/group/gshadow with shadow, hostname, hosts, resolv, + inittab, clock, console, profile, inputrc, sysctl, modprobe.d, sshd + + host keys, app-users.conf, packagemanager.conf, ~/.gitconfig, ~/.ssh/config). + * `create [--run] [archive]` (root): tar.gz with a manifest of mode + owner + and group NAMES (ids resolved through the tree's own passwd/group); a + symlink is stored as its target file. Archive 0600 (secrets). + * `restore <archive> [--run]` (root): missing -> created with mode/owner + (old main user's name -> this system's main user; unknown names -> root, + reported); identical -> left, mode differences reported; different -> + <file>.new; passwd/group/shadow/gshadow ALWAYS .new. Missing parent dirs + under a home belong to that user. The .new list is printed at the end; + an identical existing .new is not rewritten. + * LFS_BACKUP_ROOT restores into (or backs up from) a mounted tree. + * Makefile installs lfs-backup + backup.conf (yours kept, .new beside), + removes a retired /usr/bin/lfs-config. + +Tested end to end in the sandbox (list, create, show, restore dry and real, +re-run); suite block replaced with a functional one. + +## 1.14.290 -- machine.conf [gstreamer10] gst_debug = true + +The user wants GST_DEBUG logging. The book configures the core with +`-D gst_debug=false`; the machine option REPLACES that switch (tested: +apply_machine_opts turns the script's `-D gst_debug=false` into +`-Dgst_debug=true`). `gstreamer:gst_debug` is the subproject form -- not +applicable to BLFS's separate core package, not added. Plugin packages must +be rebuilt after the core: gstconfig.h's GST_DISABLE_GST_DEBUG is compiled +into each of them. No tool changed; the version moves with the stack file. + +## 1.14.289 -- launchers: a stale account, and an empty argument + + ~ > browser-mini + No passwd entry for user 'u_browser' + +~/bin/browser-mini was written when the account was still u_browser; every +later setup said "already exists -- left alone". Launchers and menu entries +that carry this tool's header ("# Runs ... as the application user ..." / +"Comment=Runs as the application user ...") are now rewritten when they +differ from what would be written; files without it are left alone. + +The CURRENT writer was also wrong: `su - u -c "app $(printf '%q ' "$@")"`, +and printf '%q ' with no arguments prints '' -- every launch without a URL +passed one empty argument. Now `exec su - u -c 'app "$@"' -- <name> "$@"`: +shadow su passes what follows the user name to the shell (checked in +su.c 4.20.2), so bash gets $0=<name> and exactly the typed arguments (n=0 +with none). Tested the writer (stale rewritten, current kept, hand-made +kept) and the argument count. + +Pre-existing in the upload: "a launcher could point at a program that does +not exist" (its sed range no longer matches the call). + +## 1.14.288 -- `script install mesa` ran the OLD version's script + + $ build p_mesa --script /usr/src/pkgusr/p_mesa/install_Mesa-25.3.5 --phase install --force + (Mesa 26.1.7 is what is installed) + +`_versioned_script_in` returned the alphabetically first install_<name>-<ver> +in the home, and every version's script stays there. It now takes the one +install_last names, else the newest by mtime. Tested both paths; suite runs +it on a fake home. + +Also: `# user 'p_<pkg>' already exists` is no longer printed on every build +(the build calls add-user with _QUIET_EXISTING=1); `lfs-helper add-user` +still says it. + +## 1.14.287 -- three undefined names (pyflakes) + + could not work out what WebKitGTK-2.52.5 depends on -- NameError: name 'ignore' is not defined + + * cmd_install's one-package missing-dependency check used `ignore`, a + local of build_install_plan. Every non-recursive install printed the + NameError instead of its missing dependencies. It now builds the same + set from args.ignore. + * cmd_script_status: `versions.get(...)` in a block that cannot currently + run (its states are script states); now `assess(u, None, run_cmd=False)`. + * lfs `_accounts_root`: `read_config()` does not exist; the NameError was + swallowed, so a configured pkgusr_subdir/cfguser_subdir was ignored. + Now load_config(). + +pyflakes finds nothing else but STACK_DIR (globals()-guarded on purpose). +Suite: runs pyflakes on packagemanager, lfs, blfs when it is installed. + +## 1.14.286 -- the session group: <prefix>_xdg-runtime + + /tmp/xdg-n76310 is group 'n76310', which is not a shared group -- u_x was not added + Set up a runtime group for the session first, then re-run this. + +Nothing ever set one up. The user's decision: one named group, +nimgnu_xdg-runtime, on the main user's XDG runtime directory; every shared +u_* account and the main user in it. + + * packagemanager `xdg_runtime_group()` = collector prefix + "xdg-runtime"; + `_join_xdg_runtime_group` creates it (collector range), adds main + u, + and chgrp/chmod 0750 the runtime directory if it exists now. The old + "read the group off the directory" rule and its tests are replaced. + * The directory is recreated each boot, so the SESSION applies the group: + app-users writes /usr/bin/xdg-runtime-share (dir 0750 + g+rw on + wayland-N and pipewire-0 sockets, polling ~30 s as they appear) and + /etc/profile.d/xdg-runtime-share.sh (dir only, at login). Not the + session bus: dbus refuses other uids. The user's sway config is not + edited; the step prints the `exec xdg-runtime-share` line to add. + * app-users expect= requires the helper, so the cached step re-runs. + +Tested: the generated helper, run as a non-root member, grouped the dir +(0750) and the wayland socket (g+rw), left the .lock alone. Not tested in a +real session: whether pipewire lets another uid in beyond the socket mode. + +## 1.14.285 -- "done" is not "current", and media's entries were done by sway + + media 1/19 done (with 189 of 215 packages installed) + ... ~70 lines of "(book is newer)" in `stack media --status --full` + +By design a stack run builds what is MISSING and never updates what is +installed; `update --stack <name>` does that. Two displays hid it: + * done counts read only the stack's OWN cache. sway includes media, so + media's entries were recorded under sway. The list and --status now + count an entry done if ANY stack's cache has it. + * --status printed the per-package "(book is newer)" only with --full. It + now prints `behind : N installed older than the book (...)` and the + `update --stack` command. The list shows "N entries behind the book" + (top-level entries only -- the dependency tree is --status's job). +Tested: the list with stubbed book/user data. Status blocks' failures are +pre-existing in the uploaded zip. + +Noticed in that status (unverified): spirv-llvm-translator and libclc show +installed at 22.x, but both last ended in failures under the record bug -- +check `llvm-spirv --version`. libical 3.0.20 -> 4.0.5 is a major (soname) +bump: update it with -R. + +## 1.14.284 -- `packagemanager stack` lists the stacks + +`name` is optional now. Without it: every *.stack in /etc/pkgusr/stacks +(and stacks/ beside the tool if different), its description (the header's +text after " -- "), buildable entries done/total from the stack cache (the +included avoid list is not counted), and a `.new waiting` note. avoid.stack +shows as "never built". Suite: the sway line is listed. README shows it. + +## 1.14.283 -- app-users' probe judged accounts it does not manage + +The 1.14.282 probe failed after a correct run: it checked EVERY u_* account, +and u_test (made by hand, not in app-users.conf) is still /home/u_test. A +permanently failing probe also re-runs the cached step on every stack run. +It now checks only the names in app-users.conf. Tested with a stub getent: +a managed account in /home fails, an unmanaged one does not. + +The failure text for a cfg entry said "MISSING A FEATURE ... optional +dependency" and suggested installing one. A cfg entry now gets: the step ran, +what it keeps true is still false, read its output, run it again. + +## 1.14.282 -- application homes in one subdirectory, as configured + + # ls /home + n76310 tester u_browser-big u_browser-mini u_claude u_test + +The machine's layout is appuser_home=/home/shared_user (history, "the user's +system"). `_relocate_app_home` ignored it and moved every account to a +hard-coded /home/<user>; `_setup_application_user` chmod-ed +appuser_home()/<user> instead of the real home; `lfs` defaulted +appuser_home to /usr/src. + + * APPUSER_HOME_DEFAULT = /home/shared_user (packagemanager), the same in + lfs's layout defaults; `appuser_home` in the config still overrides. + * Relocation targets appuser_home()/<user>, from the package tree OR the + old /home/<user>; creates the subdirectory root:root 755; re-points the + main user's ~/Shared/<Name> links into the new home. + * `user setup` relocates too, so `cfg app-users` moves existing accounts; + its expect= fails while any u_* home sits directly in /home, so the + cached entry re-runs. + +Tested in the sandbox: home moved with contents, 0750, parent created, link +re-pointed. Tests pinning the old /home and /usr/src defaults updated. + +## 1.14.281 -- no version record was written on a normal install + + .pkgusr-installed: LLVM-21.1.8 install_last: LLVM-22.1.8 llvm-config: 22.1.8 + +The "why is LLVM in the plan again" answer. cmd_build's success has two +branches: `--phase all|update` (every packagemanager install) and a +single-phase run finishing the last phase. .pkgusr-installed was written +ONLY in the second. VERSION, written before both, parsed only +name_version="..." -- the generator writes '...' -- so it was never written +for BLFS scripts either. Readers fell back to install_last, staged before +the build (the webkitgtk/LLVM "failed build advanced the version" bug). +1.14.273's seeding then pinned the OLD version with nothing to update it. + +`_record_installed` writes both records, parsing either quote style, and +both success branches call it. Suite: both calls, and a functional run of +the function; fails on 1.14.280. + +On the machine: markers seeded since 1.14.273 are stale for every package +that installed successfully since -- listed by a one-off in chat, set by hand +(LLVM: 22.1.8). + +The manifest cleanup found the pollution in 11 manifests and 4 pkg.lst +files, not only LLVM's (gst10-plugins-bad, libclc, libnftnl, librice, +libsndfile, mesa, opencv, pkgusr, shadow, webkitgtk). + +## 1.14.280 -- the earlier-stages claim took root's files from LLVM's manifest + + # claiming files from earlier stages of p_llvm ... + # gave 'p_llvm' 13 path(s) from its earlier build stages + +1.14.278 stopped the lease lending LLVM's polluted root entries, but the +claim step (before every phase) CHOWNS root-owned manifest entries to the +package for good: su_u_user and the tools became p_llvm's. The claim exists +for chapter 7's root-built leftovers under the same step name. + +Now: once the package has installed as itself (.pkgusr-installed or VERSION +in its home), only its -tmp/-pass1/-pass2/libstdcpp stage manifests are +claimed from, never its own. Claim blocks 87/197/210/214 pass. + +On the machine: chown the 13 back to root and run the 1.14.278 manifest +cleanup. Why LLVM is in the plan as "update" again is NOT known yet -- +its record should say 22.1.8 after the run that reached libclc; asked for +.pkgusr-installed and install_last. The install phase re-runs cmake+ninja +(the compile is in install_pkg) but the build tree is kept, so ninja should +have little to do. + +Pre-existing (upload): block "a failed build makes no install claim" fails +on libdisplay-info and poppler after sway.stack's git marker. + +## 1.14.279 -- firmware: patterns at any depth, flat names from WHENCE + + !! FIRMWARE_ONLY asks for 'iwlwifi-*.ucode', which is not in .../linux-firmware + intel/iwlwifi/iwlwifi-100-5.ucode ... + !! this config builds firmware INTO the kernel, so the build cannot proceed + +linux-firmware moved the iwlwifi blobs into intel/iwlwifi/ and keeps the flat +names drivers ask for as `Link:` lines in WHENCE (its copy-firmware.sh makes +the links). The sparse checkout already fetched the files at their new depth +(no-cone patterns match anywhere); the install looked at the top level only. + + * A pattern entry now matches files at any depth (and counts WHENCE link + names); files keep their paths (cp --parents). + * After a selective install, every WHENCE `Link:` whose target was + installed is created, never over a real file. `intel` alone now yields + the flat iwlwifi names too. + * No-cone sparse checkouts include /WHENCE. + +Tested on a fake checkout: both blobs copied under intel/iwlwifi/, three +links made, a link to a missing target skipped. WHENCE's exact format is +from memory (gitlab unreachable). + +**Also fixed:** 1.14.270 broke two existing suite checks -- the mesa probe +must use `test -e a -o -e b`, and the llvmpipe comment must sit directly +above `book mesa`. libglvnd now comes before that comment. + +**Pre-existing suite failures** (present in the uploaded zip, not touched): +alsa option for gst-plugins, mesa software gallium/vulkan drivers in +machine.conf, webkitgtk USE_LIBRICE / USE_WOFF2 / woff2 probe, openh264 +out-of-book markings. + +## 1.14.278 -- LLVM's manifest swallowed root's files, and the lease lent them + + # leased /etc/pam.d/su_u_user (755 root:root) to p_llvm for this phase + # leased /usr/bin/lfs (755 root:root) to p_llvm for this phase ... 14 files + +LLVM's compile is generated into install_pkg (OPEN, 1.14.273), so its +install window lasted hours. Root ran `make install` of these tools and +`cfg app-users` inside it; the touched-file scan is by time window and keeps +root-owned files, so they entered LLVM's manifest. The lease then treated +each as LLVM's own root file and chowned su_u_user -- run by root on every +su -- to p_llvm, every install round. + + * **Scan:** when a phase ran as the package user (not as_root, not + chown-after), a root-owned file it "touched" is not recorded -- it cannot + have written one; a leased file it rewrote is its own after settle. + * **Lease:** a root-owned file is lent only if it is setid (bit or + setid.conf) or the package's configure commands write it. A manifest + claim alone is no longer enough. Suite block 111 case 7; fails on + 1.14.277. + * Existing polluted manifests: one-off cleanup in chat (tool files, + su_u_user, S99power-perms, completions). + +**libclc: "Producer LLVM22.1.8, Reader LLVM 21.1.8"** -- llvm-spirv is still +the 21 build. spirv-llvm-translator failed at 22.1.5 before the 1.14.273 +marker fix, so its record says 22.1.5 and the plan skipped it (same poisoning +as LLVM's). Fix on the machine: rebuild it explicitly, then libclc, mesa. +After an LLVM major bump, `packagemanager update llvm -R` rebuilds dependents. + +## 1.14.277 -- kernel.conf CONFIG_SET + +lfs-kernel had no way to say "this machine always wants option X": the config +came from the running kernel, a refresh answered new symbols, and anything +else meant menuconfig by hand every time. `CONFIG_SET="SYM=val ..."` (y, m, +n, number, string) is applied after the refresh with scripts/config, resolved +with olddefconfig, and READ BACK -- a value the kernel reverts (missing +dependency, unknown symbol) is named and stops before compiling. + +Shipped value: `LEGACY_TIOCSTI=n` (shadow 4.20 su detaches root's `su -` +shells while TIOCSTI is allowed). An edited kernel.conf is kept by +`make install` (.new beside it), so an existing machine adds the line by hand. + +Tested: the apply/read-back block on a fake tree (disable taken; an unknown +symbol reported, exit 1). Kernel blocks 155 and 401 pass. Not built. + +## 1.14.276 -- lfs-helper regroup-dir + + /etc/pam.d p_bash-completion:nimgnu_bash-completion drwxrwxr-x + +A directory shared through the wrong group. `lfs-helper regroup-dir <dir>`: + * installers = packages whose manifest or pkg.lst has a direct child of it, + package/config accounts owning an entry in it, and its owner; + * asks the new name (1 dir-named, 2 owner-named, typed, q) -- pam.d gives + nimgnu_pam-d; + * members of the old group keep it only if they own, or install into, + another directory of that group -- found on EVERY real filesystem + (a missed mount would have dropped memberships); + * prints the plan, applies only on y; updates collector-map and granted. + * Leaves the old group in place and says when it is on no directory. + +Tested in the sandbox: installers found, two members dropped, the owner and a +member of another directory kept, q cancels, new group at 90000. + +(The 1.14.275 open item on /etc/pam.d stays: it should be root's. This +command at least narrows who can write it.) + +## 1.14.275 -- su to application users: four faults, one symptom + + ~ > su - u_browser-mini + su: Authentication failure + su: Authentication token manipulation error + + 1. **/etc/pam.d/su had no pam_exec line.** Shadow's configure rewrote su + (1.14.263); `cfg app-users` was `done (stack cache)` and never re-ran. + **A cached cfg entry with expect= is now re-checked on every run** and + re-run when the check fails. shadow-pam and app-users got expect= + probes (su setuid root + pam.d files; su calls su_u_user, root owns it, + caller check present). Probe tested against good and bad files. + 2. **SECURITY: su_u_user compared the TARGET with a list containing it** + (`[[ "$USERS" =~ "$PAM_USER" ]]`, empty PAM_RUSER filled with PAM_USER) + -- always true: any account could su to every listed account without a + password once the pam line was there. Now `[[ -n $PAM_RUSER ]] || exit + 1` and a whole-word check on PAM_RUSER. Existing files are repaired in + place with a backup. Tested: main user passes, another u_ and an empty + caller are refused. + 3. **SECURITY: su_u_user was owned by p_librice** (attributed to librice's + build window) -- a package user could replace a script root runs on + every su. app-users forces root:root 755 on every run. + 4. **"token manipulation error"** = the account step demanding a new + password: /etc/shadow last-change 0. app-users locks an EMPTY password + field (Enter would have logged in) and sets last-change when it is 0. + Tested on a sandbox account. + +**Open:** /etc/pam.d itself is p_bash-completion:nimgnu_bash-completion 775 +-- every member (bash-completion, gstreamer10, linux-pam, shadow) can replace +any PAM file. It should be root-owned; collector grants on /etc/pam.d need a +rule. + +## 1.14.274 -- the slow scan, and CMake's wrapped permission error + +**"scanning the tree ... takes a moment" took minutes** after a two-second +install. Every phase end walked the whole tree five times (snapshot, +snapshot_dirs twice, two `find -newer`), and every walk used +`-not -path "$SRCROOT/*"` -- which filters AFTER descending, so each one +went through /usr/src: every package's unpacked sources and build trees. +Now `-prune` (same output: compared old vs new on a synthetic tree, both +snapshot lists and both touched lists identical), snapshot_dirs once, and one +walk for touched files+dirs. Scan-related suite blocks pass; block 240's +string check accepts the pruned form. + +**nvtop stopped after one round** on a plain permission error. CMake wraps +it: `...metainfo.xml": Permission` / `denied.` -- no line matched, round 2 +found nothing and the loop ended. `unwritable_dirs_from_log` joins each +`file INSTALL` block; the repair gate also accepts "file INSTALL cannot". +Tested on nvtop's log text. + +The retry loop itself already runs while each round fixes something (cap +20, stops when the same directories repeat) -- it was the parser. + +## 1.14.273 -- a failed LLVM update was taken as installed + + CMake Error ... Could not find a configuration file for package "LLVM" + compatible with requested version "22.1.0" ... version: 21.1.8 + +The mesa plan was 9 entries (LLVM failed on its patch, 1.14.272); the +re-run's was 6, without LLVM. `User.version` reads `.pkgusr-installed` +(written after success) and, for a package built before that marker +existed, falls back to `install_last` -- which lfs-helper replaces BEFORE +the build. So LLVM's failed 22.1.8 attempt made it "current". + +The staging step now seeds `.pkgusr-installed` from the OLD install_last +before replacing it (only when the marker is absent). A failure then leaves +the old version recorded. Tested: the version extraction. Suite: grep. + +On the machine the record is already wrong -- one-time fix: + echo LLVM-21.1.8 > /usr/src/pkgusr/p_llvm/.pkgusr-installed + +### OPEN -- LLVM's compile is generated into install_pkg + +The page creates /etc/clang/*.cfg as root BEFORE cmake (on purpose: SSP for +compiler-rt), and the phase split starts install at the first root block, +so cmake + ninja + `ninja install` all land in install_pkg. It works, but +"re-run only the install step (no recompile)" is false for LLVM, and a +repair round in install recompiles. Needs a rule for a root block the page +says must precede the build. + +## 1.14.272 -- extra downloads were saved under the tarball's name + + HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable + The file is already fully retrieved; nothing to do. + patch: **** Can't open patch file ../llvm-22.1.8-upstream_fix-1.patch + +`_pp_unpack` called `_pp_fetch "$li"` for each additional_links entry with no +output name, and `_pp_fetch` defaults the name to `$pkg` (the openh264 +rename). So every patch was `wget -c -O <the tarball>`: 416 when the tarball +is bigger (patch silently missing), and appended onto it when smaller. Now +`_pp_fetch "$li" "$f"`, and a missing file after the fetch is an error. + +Every BLFS page with a patch or extra tarball was affected since that rename. +Tested with a stub wget. Suite: grep in "one list, two entry points". + +The run was mesa's dependency plan updating LLVM to the 13.1 book's 22.1.8 +-- hours; there is no skip-dependencies switch. + +## 1.14.271 -- libglvnd taking Mesa's headers was called a bundled copy + + /usr/include/GLES/egl.h belongs to p_mesa -- handing it to p_libglvnd + ! libglvnd has now taken three files, one per round, from 'p_mesa'. + ! It is building a BUNDLED COPY of that package ... + +It was not. Mesa with glvnd=disabled owned GL/GLES/EGL/KHR headers and +libGL; libglvnd legitimately replaces them. meson stops at the first file +it may not chmod, so the log named one per round, and the 1.14.116 streak +rule (3 single handovers from one owner) fired. + +`meson_takeover_from_log` now, after a handover, reads +`meson-info/intro-installed.json` (found by walking up from the source path, +no further than the package user's home) and hands over EVERY planned +destination still owned by that same package -- one round. Not done for a +subprojects/ source, and only a subprojects/ source now counts toward the +bundled-copy streak, so libdisplay-info's case still stops. + +Tested in the sandbox with two users: 1 logged + 3 planned files moved, a +file outside the plan untouched; a subproject source still sets the streak. +Suite: grep checks; "the directory was never the problem" and "one file per +round" blocks still pass. + +**Open:** directories (GL/, EGL/, KHR/) owned by p_mesa still get one +collector-group question each. + +## 1.14.270 -- Mesa with glvnd; libglvnd in sway.stack + +The user asked for it. `[mesa] glvnd = enabled` in machine.conf, and a +`git libglvnd` entry directly before `book mesa` in sway.stack. + + * **libglvnd is NOT a BLFS page** (the book only links GLFS). An earlier + chat reply said it was -- wrong. URL is the canonical freedesktop + gitlab; tag v1.7.0 and option names CHECKED on NVIDIA's GitHub copy. + Default options. + * **One go, or GL breaks**: between libglvnd's install and Mesa's + rebuild, libGL.so.1 is glvnd's with no vendor. + packagemanager stack sway --only libglvnd,mesa --run + Existing binaries keep working (same sonames). + * Mesa's probe now also requires 50_mesa.json and libGLX_mesa.so.0, and + its `a; b` form (first test ignored) became one expression. + * **`--clean` never deletes a file another package owns.** Mesa's old + manifest lists libGL.so.1, which libglvnd installed before the Mesa + rebuild -- older than the install start, not rewritten: `_orphan_sweep` + would have deleted the GL library. It now skips files not owned by the + package user. Tested with a stub remover. + +Suite: glvnd-before-mesa order and the sweep guard, in "one list, two entry +points". Not built. + +## 1.14.269 -- mesa-demos: libdecor's demo wants opengl.pc + + subprojects/libdecor-0/demo/meson.build:8:9: ERROR: Dependency "opengl" not found + +Not a missing stack entry. opengl.pc is libglvnd's, and machine.conf has +`glvnd = disabled` for Mesa, so Mesa ships gl.pc/egl.pc/glesv2.pc and no +opengl.pc -- by design. Only libdecor's DEMO (a meson subproject pulled in +by mesa-demos' wayland demos) asks for it. diag.stack passes +`-Dlibdecor-0:demo=false`, and wipes `build/` before `meson setup` in both +meson entries, since a failed setup leaves it half-configured. + +**The failure box misleads here**: "packagemanager install opengl" -- there +is no such package, and installing libglvnd without rebuilding Mesa against +it would not help. Not changed; worth a pc-module note later. (1.14.270 +switched the machine to glvnd.) + +**machine.conf `[mesa] gbm = enabled`.** Mesa builds gbm by default on +Linux, and sway works, so it is there. Named so a Mesa that cannot build +it fails at configure rather than finishing and breaking wlroots. No +rebuild needed. + +`libdecor-0:demo` is from memory (gitlab unreachable) -- if meson says +"Unknown option", the name is in +`subprojects/libdecor-0/meson_options.txt` in the clone. + +## 1.14.268 -- diag.stack: GPU diagnostic tools + +New optional stack, none of it in the book: mesa-demos (glxinfo, eglinfo), +vulkan-tools (vulkaninfo), libva-utils (vainfo), nvtop. + + * **vulkan-tools and libva-utils pick their tag at build time** from the + installed version (`pkg-config --modversion vulkan` / `libva`). Both + must match what is installed, and a fixed pin breaks on the next book. + Tag patterns checked with git ls-remote. + * **nvtop's vendor is the only choice in the file**, in a marked block at + the top. Every backend defaults ON on x86, so all are named; ON is + AMD only (this machine). + * **mesa-demos is NOT CHECKED**: gitlab.freedesktop.org was unreachable. + Pinned mesa-demos-9.0.0; option names from memory -- first thing to + look at if it fails. + +Tested: parsed, the four generated scripts pass `bash -n`, libva-utils tag +selection on a real clone. The conventions and version suite blocks pass. +Not built. + +## 1.14.267 -- id ranges, and cfg power's three false failures + +**Ranges, one per kind of id.** + + package users + private groups 10000-20000 + collector groups (no account) 90000-98999 + application users (u_*) 30000-39999 + +Found on the machine: `nimgnu_python-modules`, `-xorg7-input-driver`, +`-perl-modules` were group AND login account at 10096/10147/10162, home +`/usr/src/pkgusr/p_nimgnu_...` -- `sysgroup create` ran the package-user +helper. It now calls `_create_collector_group`, group only. `u_*` were +at 10324-10349 among the package users; `user create` now uses +`APP_UID_MIN/MAX`. `nimgnu_power` was 997 (power.sh: system groupadd); +it now asks `packagemanager sysgroup create`. + +The machine's existing ids are repaired by a one-off script given in chat +(dry run by default): removes the collector accounts, renumbers collector +groups into 90000+, u_* into 30000+, re-owns files on every real and tmpfs +mount, and puts each file's mode back -- chown/chgrp clear setuid, which +is how 1.14.265 lost su. Tested in the sandbox on fake accounts: setuid +bit, group members and home ownership kept. + +**power.sh, three false `!!`:** + * poweroff/reboot are symlinks to halt; `stat` read the link (777). It + now resolves each path and handles every target once. + * it linked only into rcsysinit.d; LFS's sysinit runlevel is rcS.d. + * the two udev lines that never matched (KERNEL=="state", =="bootlog") + are now removed, not explained -- the user asked. No bootlog account + or group is needed. + +Suite: greps in "one list, two entry points"; the 1.14.266 files fail all +five. Tested: power.sh's udev cleanup on sample rules. + +## 1.14.266 -- cfg power could not find the collector prefix + + !! no collector_prefix in /etc/pkgusr/packagemanager.conf + +power.sh parsed packagemanager.conf with the lfs config's JSON pattern +(`"collector_prefix": "..."`); that file is `key=value`. It could never +match, and on a booted system with no lfs config it stopped the stack. Two +parsers, one fact, again. + +packagemanager now exports the RESOLVED `LFS_COLLECTOR_PREFIX` and +`LFS_MAIN_USER` to every cfg script: environment, lfs config, +packagemanager.conf, then the collector groups on the system -- never a +built-in default. power.sh reads the variable, and falls back to the +conf's real `collector_prefix=` line. + +Suite: grep checks in "one list, two entry points"; the old power.sh fails +them. Tested: power.sh's group step with and without the variable. + +**The stack cache hides the repair.** `stack services --run` printed +`cfg shadow-pam done (stack cache)` and did not run it, so the stripped +setid bits stayed stripped. `--only shadow-pam --run` runs it. Whether a +cfg entry should ever be cached is open. + +## 1.14.265 -- the earlier-stages claim stripped every setuid bit + + ~ > su - + su: Authentication service cannot retrieve authentication info + +On the booted machine su had no u+s, nor had the rest. The cause is in +the configure output of 1.14.263's run: + + # gave 'p_shadow' 16 path(s) from its earlier build stages + +`_claim_earlier_stages` runs before EVERY phase and chowns root-owned +manifest entries to the package user -- its own manifest included, not only +-tmp/-pass stages. chown clears setuid, so su, passwd, chage ... went +4711 root -> 755 p_shadow. The lease then saw files the installer already +owned and had nothing to restore. The claim fired before the lease could. + +`_claim_candidates` now skips any file with a setuid or setgid bit; those +are the lease's (copy aside, restore on failure, setid.conf on success). +Suite block 111 case 6; checked to FAIL against 1.14.264. + +**On the machine:** the bits are already gone, and shadow-pam's step 3 +applies setid.conf -- that is the stack's own repair. It needs root, and +su is what broke: log in as root on a tty, or run it from the chroot. + +**`update --clean` does not touch this.** It is opt-in, removes only +old-manifest files the new install did not rewrite, and never under /etc, +/var, /boot, /home, /root, /srv. + +## 1.14.264 -- a package reclaims its own files + +The rule, from the user: **a package owns what it installs, whoever holds +the file now.** The lease already settled a rewritten file to its +installer, but on INSTALL it only leased the manifest's ROOT-owned entries. +A manifest file another package had taken was not leased, so a bash +redirect or `cp` over it failed with a wording no repair parses -- the same +shape as shadow's /etc/pam.d/su. + +`_lease_candidates` now takes every manifest entry not owned by the +installer. The guards stay in `lease_files`: package accounts only, root +only when the manifest claims it, no symlinked directory. On success a +rewritten file stays with the installer; an unchanged one goes back. + +Suite block 111, case 5; checked to FAIL against the 1.14.263 lfs-helper. +Not run on the machine. + +## 1.14.263 -- the lease: hand files over before a phase, not after it fails + + /usr/src/pkgusr/p_shadow/install_Shadow-4.20.2: line 153: /etc/pam.d/su: Permission denied + +/etc/pam.d/su was p_linux-pam's, 644. The repair loop recognised nothing +in a bash redirect's wording, granted /etc/pam.d (never the problem) and +stopped. Every tool words that refusal differently, and each needed its own +parser -- and the setid pass (root takes the file) and the 1.14.261 +take-back (a repair hands it back) were two halves of one cycle. + +**The lease**, in `run_phase_as` -- the one runner -- around every +install/configure/all/update phase, as root: + + * **before:** each file the phase will write that another account owns + is COPIED to `$STATE/leases/<pkg>/` with owner and mode, and chowned + to the installer. Copied, never moved: the phase starts with + `su - p_<pkg>`, and su, its pam file and unix_chkpwd must exist. + * **failure** (including exit 0 with a failure in the log -- it settles + AFTER that check): every copy back, owner and mode, read back. + * **success:** a setid.conf path gets its policy; an unchanged file goes + back to its owner; a rewritten one belongs to the installer. + * a ledger left by a killed run is restored before the next lease. + +**Which files:** the manifest's ROOT-owned entries (install), and what +`configure_pkg` writes (configure) -- `lfs-helper script-targets`: redirects, +tee, install/cp/mv/ln destinations, sed -i, for-loops expanded, heredoc +bodies and comments skipped, /tmp /dev /proc /sys /run ignored. + +**Security:** a root-owned file is leased only when this package's manifest +claims it (the script is the package user's to edit; one line could name +/etc/shadow). Non-package accounts' files never. No path through a +symlinked directory. The first install over another package's file cannot +be predicted, so the repair loop remains the fallback -- minus its root +take-back, which is gone. + +**One setid table.** `_setid_known` had its own list saying 4755 while +setid.conf said 4711, so `cfg shadow-pam` and `setid --run` corrected each +other. It reads setid.conf now (the dbus helper line moved there), and a +requested bit on a path setid.conf names is left to setid.conf. + +**The stale script on the machine.** `packagemanager stack services` run +from ~/lfs read ~/lfs/services.stack, and `cfg` entries run +`<that dir>/<name>.sh` -- an old shadow-pam.sh. packagemanager now warns +when a stack or cfg script found there differs from /etc/pkgusr/stacks. + +shadow-pam asks `lfs-helper script-targets` for its pam.d set (one parser). +Stack stamps: all at 1.14.263 -- 1.14.262 bumped the tools and not the +stacks, which the suite reports. + +**Tested:** `run-script` end to end with real su, two throwaway package +accounts and sticky directories (failed configure restored; install over a +root 4711 file settled to 4711 root:<pkg>; successful configure owned by +the installer; interrupted lease restored). Suite block 111 does the same +without su; it FAILED until the fixture left /tmp (script-targets ignores +/tmp) and `_setid_conf` survived `set -u`; checked to FAIL against the +1.14.262 lfs-helper. Whole suite, all 429 blocks, run against both trees: +identical FAIL lines except the stack-version ones 1.14.263 fixes -- the +1.14.262 zip shipped stacks still stamped 1.14.261. + +**Not tested:** a real package build on the machine. + +### Suite: pre-existing, not from this release + + * Block "one quote character, three symptoms" deletes the suite's + counter directory; every PASS/FAIL total after it reads empty. Same + on the 1.14.262 tree. + * `--only` of a block that uses `_hlv` without block 114 ("the version + is what FINISHED") died on `_hlv: unbound variable`. Fixed for block + 121 ("one list, two entry points"), which now sets it itself; other + blocks may share the shape (grep `_hlv` before relying on --only). + +## 1.14.262 -- shadow's pam.d files, and the cache that hid three fixes + + passwd: pam_start() failed, error 26 + login: PAM Failure, aborting + +**The cache.** The first try at this fix did not fire either, which is how +the real bug turned up: blfs's index cache was keyed on the book's size, +mtime and a hand-bumped CACHE_VERSION -- not on the code. An edited +extractor rendered from the index the OLD code built. That is the whole +story of 1.14.212 and both poppler patches "not firing". `_book_stamp` now +includes `_build_id()`, so any edit to blfs re-indexes once (~20s). + +**The walk.** Shadow's commands are in `div.configuration > div.sect3 > +div.sect4 > pre.root`, and `extract_commands` dropped every div that was +not a note. It now descends into `div.sect4`/`div.sect5` with the same +rules (guards included) and keeps their titles as `##` comments. The top +level is unchanged byte for byte. + +**Proved against a rendered book, not a sample.** The BLFS source was +cloned and rendered (`make nochunks REV=systemd`). Before and after, for +all 146 pages with a configuration section: 141 scripts byte-identical, no +build or install phase changed anywhere, and exactly five configure phases +gained blocks -- the five pages the XML says have commands in sub-sections: +Shadow, MIT-Kerberos, NFS-Utils, SANE, Samba. + + * Shadow: login.defs, login, passwd, su, chpasswd+newusers, chage, the + `for PROGRAM` loop, access and limits renames. 16 pam.d files. + * NFS-Utils, SANE: the new blocks are all guarded (SKIPPED). + * Samba: `systemctl enable` lines (the shim's job); smb.conf examples + have no command markup and stay out. + * **MIT-Kerberos: see OPEN below.** + +**`cfg shadow-pam` checks the SET.** It read two files and said "in +place" while fourteen were missing. Now `_book_pam_set` reads each +package's `configure_pkg` (every `/etc/pam.d/<n>` on a live line, plus the +words of a `for VAR in ...` loop writing `/etc/pam.d/${VAR}`) -- linux-pam 5, +shadow 16. The script comes from `packagemanager script fetch`; if it names +no pam file (the stale pre-1.14.262 one), it is fetched again with +`--regenerate`, so a correct script is never regenerated just for a check. + +**The su trap.** Shadow's su block has no pam_exec line. app-users greps +for `su_u_user` and re-adds it, and runs after shadow-pam in services.stack; +the suite now asserts that order. shadow-pam says so when it happens. + +Suite: block 110 renders a shadow-shaped page through the generator (with a +fresh BLFS_STORE), and runs `_book_pam_set` on the output. Checked that it +FAILS five ways against 1.14.261's blfs. + +### OPEN -- MIT-Kerberos's configure phase is now a transcript + +It gained a KDC walkthrough: `kdb5_util create -r __EXAMPLE_ORG__ -s`, +`kadmin.local` followed by lines like `kadmin.local: addprinc ...` (typed +INTO the program, not shell), `kinit`. stdin is closed, so nothing hangs; +the phase fails loudly instead of silently doing nothing, as before. Not in +any stack. The fix is a "prose is not code" rule: a line `<prog>: ...` +after a bare `<prog>` line is a transcript -- comment the block. + +### Small: shadow's login.defs block is not idempotent + +A second configure run copies the ALREADY-edited login.defs over +`login.defs.orig`. Harmless for login; the pristine copy is lost. + +## GAP IN THIS DOCUMENT -- 1.14.180 to 1.14.238 + +The per-release notes for **1.14.180 through 1.14.238** are NOT in this +file. They were written each release and are not here now; the section +below jumps from 1.14.179 to 1.14.178. I do not know which edit dropped +them, and I am not going to reconstruct fifty-eight sessions from memory -- +invented notes would be worse than an admitted gap. + +Where the real record is: + + * every release's zip contains the HANDOFF.md of that release. The + newest zip that still has a given section is the place to read it. + * the CODE carries its own reasons: every fix in that range went in with + a comment naming the failure it came from, and + `test_lfs_crosschain.sh` carries an assertion per fix with the release + number in its comment. `grep -n '1\.14\.2' test_lfs_crosschain.sh` + lists them in order and is a usable index of what changed and why. + +What that range covered, from the assertions, in one line each: the +swallowed-failure dialects; the stale-destination and cmake-destination +repairs; job counts reaching make/cmake/cargo/ninja; the `[all]` section; +media.stack and `include`; avoid.stack and every stack including it; +app-users.conf with create-or-setup and the PAM script; `env --system`; +librice; the version marker; and the per-package `book =` override. + +## This session (1.14.179) -- lfsconf is now lfs-config, and it installs + +It was `lfsconf`, living inside the repo it managed: the repo carried a copy +of the tool, the tool had no version anyone could compare with the others, +and `make install` did not know it existed. It is `lfs-config` now, on the +Makefile's TOOLS line -- so install, uninstall, the stamp file and the chroot +refresh all pick it up with no second edit. + +**The point of it is the other machine**, and that half was missing: + +- **The repo records whose config it is** (`repo-user`). A directory's owner + answers that only on the machine that made it; cloned elsewhere it is + whoever ran git, while every `~` path and every owner field still means the + original user. +- **`lfs-config install [--as USER]`** maps the recorded user to this one + (`n76310:n76310` -> `chris:chris`), expands `~` to the target's home, + refuses a user that does not exist here, and **keeps what it replaces** as + `<path>.orig-<timestamp>`. +- It does NOT rewrite file contents. It lists the tracked files that still + mention the old name, because a username inside a config file may be a + path, a group, an SSID comment or a mistake, and only the reader can tell + which. The suite fails if a `sed -i` ever appears there. +- **`lfs-config init`** makes the repo. Creating one used to be a side + effect of every command -- `--help` in a directory with no repo wrote a + manifest into ~/LFS/lfs-config and printed two errors before the help. + +Tested end to end: collected as n76310, installed as chris, owner mapped, +original kept, the one file mentioning n76310 named. + +## Previous session (1.14.178) -- the gate decides, the loop acts + + # /usr/share/doc/libtiff-4.7.1/tiff is p_libtiff's own output ... + # removing it so this install can replace it + ! libtiff FAILED ... gate: recognised, rounds run: 0 + +The repair did exactly the right thing and the package failed anyway. I +called it from the GATE -- the code that decides whether a retry is worth +attempting -- so the directory was removed before the retry loop started, +and the loop counts only what IT fixed. Nothing re-ran the phase, and the +obstacle was already gone. + +The gate now only greps for the message; `_clear_stale_dest` runs inside the +loop beside the other repairs, and setting `_fixed_something=1` is what +causes the retry. A second round finds the path gone and returns 1, so the +loop breaks instead of spinning. + +Two things this cost, both now asserted: + +- a repair called from the gate is a bug in itself: block 375 fails if + `_clear_stale_dest` appears with `_perm_ish=1`. +- **`_slice_fn` stops at the first line starting with `}`**, and cmd_build is + 1700 lines with several of those -- so two assertions were matching against + an empty slice and passing. They grep the file now. A test that cannot fail + is worse than no test. + +Resume: `packagemanager late-deps --rebuild --run` -- 45 left, and libtiff +should clear itself and carry on this time. + +## Previous session (1.14.177) -- the repair failed the build it was repairing + + /usr/bin/lfs-helper: line 5914: user: unbound variable + update failed for git (exit 1) + +git's install had just finished -- every hardlink, every alias, 345 files +tracked. Then yesterday's repair hook, `_clear_stale_dest "$log" "$user"`, +read a variable that does not exist: `cmd_build` calls the package user +`owner`, and `set -u` turned a successful install into a failed one with a +line number and no explanation. + +Two fixes, and the second is the rule: + +- `${owner:-}`, which is what that function calls it. +- `_clear_stale_dest` returns 1 on an empty user instead of running. **A + repair must not be able to fail the thing it is repairing**: it runs after + something already went wrong, so anything it does must be safe when its + inputs are not what it expected. + +Block 375 asserts both, and that `cmd_build` never reads `$user` at all -- +checked by putting the bug back and watching two assertions fire. + +Resume: `packagemanager late-deps --rebuild --run`. git is installed +(the install completed); it will be re-checked and skipped or redone +quickly. + +## Previous session (1.14.176) -- a book command that works once + +The rebuild pass stopped on libtiff, 38 packages from the end: + + mv: cannot overwrite '/usr/share/doc/libtiff-4.7.1/tiff': Directory not empty + +Nothing was wrong with the system. libtiff's page moves its html docs into a +versioned directory; the first install creates that directory, and the +second cannot move onto it. The destination was **the package's own output +from last time**, and a reinstall means replacing it -- but the repair gate +did not recognise the message ("gate: did not recognise this failure, rounds +run: 0"), so a 38-package pass stopped on a directory the package itself had +put there. + +`_clear_stale_dest` reads `cannot overwrite '<path>'` out of the log and +removes that exact path -- **only when the package being installed owns +it**. A directory owned by anyone else is a conflict, not stale output, and +deleting it would be the worst repair available: that case warns and leaves +it. Proved on real text in block 375, all three outcomes (own output +cleared, other owner refused, unrelated failure untouched). + +Resume with: + + packagemanager late-deps --rebuild --run + +libtiff will be retried from its install phase; what already succeeded is +recorded and does not repeat. + +## Previous session (1.14.175) -- the help was two lists, both stale + +No, it was not up to date. + +- The grouped `packagemanager --help` -- the part a reader actually uses -- + had no `stack`, `probe`, `late-deps`, `reclaim`, `bootstrap` or `env`. + `stack` is the primary command of this tool. Two new groups now: "a whole + environment at once" and "features, not files". +- The module docstring kept a **Roadmap** marking check, search, + dependencies, install, remove and reload-pkg-list as `[TODO]` -- all six + implemented releases ago -- and none of the twenty commands added since + appeared at all. Replaced by a pointer to `--help` and the two design + questions that really are open. +- `lfs-helper --help` was missing `undone`, `sort-users` and `check-login`, + and said nothing about the chroot build's own steps (init-accounts, + establish-ownership, seal-install-dirs, install-profile, + repair-adoption). They are listed as what they are: dispatched by + build-all, and `lfs-helper list` names them. + +Block 375 now asks ARGPARSE for the command list -- it prints one when given +a bad command -- and requires every name to appear in the grouped help. +A list of commands maintained anywhere but beside the commands goes stale +silently; this one now cannot. + +## Previous session (1.14.174) -- audiornnoise needs no second git entry + +It is in gst-plugins-rs already: the element lives in **gst-plugin-audiofx**, +not in a plugin of its own, so it is one more `-p` on the entry added in +1.14.172 rather than a new checkout. Its noise model is the nnnoiseless +crate, vendored by cargo, so there is no system library to install first. + +The probe grew with it -- `webrtcsink && audiornnoise` -- and `check webrtc` +lists audiornnoise among the optional rust elements, with a line saying what +it is for (noise suppression for a microphone in a call). + +Three assertions in block 376 that came out of writing this: + +- an element named in a probe must be asked for by some `-p`, or the check + tests something nothing builds; +- `audiornnoise` must NOT appear as `-p gst-plugin-audiornnoise`, which is + the mistake that looks right and fails before anything builds; +- the build and install lines must ask for the SAME plugin set -- a plugin + compiled and not installed is the most confusing outcome available + (checked by removing audiofx from the install line and watching it fail). + +## Previous session (1.14.173) -- a plugin is a shared object + +The first real gst-plugins-rs run compiled everything (5m40s, four plugins) +and died installing a STATIC library: + + failed to copy `.../libgstrsrtp.a` to `/usr/lib/gstreamer-1.0/libgstrsrtp.a` + Permission denied (os error 13) + +Two separate faults, and neither was the permission: + +- **cargo-c installs a .a as well as a .so by default.** Nothing loads a + static GStreamer plugin. `--library-type cdylib` on both the build and the + install line, and the file that failed is never produced. +- **The repair ran ZERO rounds** ("gate: recognised, rounds run: 0") because + cargo names the destination in BACKTICKS, on the line above the reason -- + no quotes, and no "Permission denied" on the same line, so the path + extractor found nothing while the box still said "grant access and retry". + It now recognises `failed to copy ... to \`<path>\`` and offers the + destination's directory; asserted against that exact text. + +/usr/lib/gstreamer-1.0 is `p_gstreamer10:nimgnu_gst`, so the .so still needs +the group -- which the repair can now grant by itself, in one round. + + packagemanager stack sway --only gst-plugins-rs --run + +The compile is cached, so the retry is the install step only. + +## Previous session (1.14.172) -- gst-plugins-rs, and what it is honest about + +Added to sway.stack at entry 124: after gst10-plugins-bad and gst10-libav, +which it builds against, and before webkitgtk. cargo-c is already installed +(librsvg pulled it in), so nothing new is needed to build it. + +It is NOT a BLFS page, so three things that a book page would settle are +ours, and each is stated where the entry is -- the suite checks that the +statements are there: + +- **`ref=main` is not a pin.** Upstream tags 0.14.x for GStreamer 1.28, but + that pairing is unverified from here, so the entry follows main rather + than claiming a tag that may not match. Pin it once a build succeeds: the + ref that worked is the one to write down. +- **It downloads crates at build time.** There is no offline mode, so this + entry cannot be part of a build that has no network. +- **The plugin list is a choice.** Four are named -- webrtc (webrtcsink and + webrtcsrc), rtp, dav1d, gtk4 (gtk4paintablesink). All of gst-plugins-rs is + enormous and mostly irrelevant here. Add with another `-p`. + +`have=/usr/lib/gstreamer-1.0/libgstrswebrtc.so`, and +`expect="gst-inspect-1.0 webrtcsink"` -- an element again, not a file. + +`check webrtc` reports webrtcsink, webrtcsrc and gtk4paintablesink as +OPTIONAL (`->`, not `!!`): the plugin set is not a BLFS page and not +everyone builds it, so its absence must not fail a stack. Verified by +deleting the network warning from the stack and watching the suite fail. + +## Previous session (1.14.171) -- the advice was a loop + +`stack sway --only sdl3 --run` on an INSTALLED package runs `install`, which +answers "Nothing to do -- everything is installed and up to date". The expect +probe then failed and the tool recommended... the same command. Four bugs in +one screen: + +- **The suggested command could not work.** A stack entry does not rebuild an + installed package. The expect= failure now prints + `packagemanager update <pkg> --reinstall --regenerate --run --yes` + (`_rebuild_cmd`, the one place that knows this) and says plainly that + re-running the entry will not help. `probe` had the same mistake for the + installed case. +- **And it named a stack that does not exist**: `stack sway.stack --only ...` + -- `os.path.basename(path)` without dropping the suffix, while the probe + path next to it stripped it correctly. +- **"NOT recorded as done, so the next run rebuilds it" was false** for an + installed package. Removed; the truth is in its place. +- **The newer-stack notice argued against itself**: `(149 entries, this file + has 149)` for two files differing in every entry's options. It reports + added/removed lines now, and says when the entry count is unchanged so the + difference is inside the entries. Also `--take-new` said "149 entries now" + two lines above a header reading "(62 entries)" -- one counts file lines, + the other the buildable plan. One measure, one word. + +For sdl3 on the machine, after `make install` and taking the stack again +(the glob fix is in the shipped file): + + packagemanager probe sdl3 + packagemanager update sdl3 --reinstall --regenerate --run --yes + +## Previous session (1.14.170) -- four probes pass; both failures were misreported + +With the stacks taken, `probe` runs: **ffmpeg has ALSA, opusenc, v4l2src, +vp8enc, webrtcbin, dtlssrtpenc and srtpenc are all there** -- the WebRTC +chain and the earlier ALSA work came out right. The two failures were both +described wrongly: + +- **gst10-libav is not installed at all** (it was missing from the stack + until 1.14.167), and the report said "the feature is NOT in the installed + package" -- which sends the reader looking for a build option. A probe + failure now checks for `pkg.lst` and says NOT INSTALLED, with "build it" + rather than "rebuild it". +- **The sdl3 probe named one path.** SDL3 installs `libSDL3.so.0.<n>` and + the bare `libSDL3.so` symlink is not always there, so the probe could fail + for a reason that has nothing to do with ALSA. It globs now. A failing + probe also prints its own first line of output, because a probe can be + wrong about a path as easily as a package can be wrong about a feature. + +On the machine: + + packagemanager stack sway --only gst10-libav --run # not built yet + packagemanager probe sdl3 # after make install + # if it still fails, sdl3 really was built before alsa-lib: + packagemanager stack sway --only sdl3 --run + +Note the take-new also brought `cfg power` and `check net-wifi` into the +stacks as unbuilt entries -- both are read-only or root-only steps and can +be run with `--only`. + +## Previous session (1.14.169) -- "no probes" was the wrong half of the answer + +`packagemanager probe` on the machine: *no expect= probes in the stacks*. +True, and useless -- the probes are in the SHIPPED stack files, and an +edited stack is never overwritten, so they arrived as +`/etc/pkgusr/stacks/sway.stack.new`. The tool carried probes no installed +stack had, and the message read as "nothing to check". + +It now looks for `*.stack.new` carrying `expect=` and says so, with the +command to take each one. Failing that, it says probes live in stack files +as `expect="..."` and that an older stack has none -- rather than leaving the +reader to conclude everything is fine. + +**On the machine**, before probing anything: + + packagemanager stack sway --take-new # then network, services + packagemanager probe + +The whole `.new` mechanism remains on the deferred list -- three functions +move stack files and it still surprises people. This is the second time a +feature was invisible because the stack carrying it was sitting beside the +one in use. + +## Previous session (1.14.168) -- "+0 day(s)" was the whole bug, filed as fine + +`late-deps ffmpeg --why` came back clean on every encoder (x264, x265, +libvpx, aom, svt-av1, opus, lame, fdk-aac, dav1d -- all days earlier) and +showed one line that should have stopped the eye: + + recommended alsa-lib-1.2.15.3 -> alsa-lib, built +0 day(s) relative to it + +A day of slack was chosen because pkg.lst is rewritten by reload-pkg-list +and --clean, so packages from one run differ by minutes. But alsa-lib a few +HOURS after ffmpeg is the original bug exactly, and the slack filed it as +fine. An hour covers the rewrite noise; the report prints hours below a day, +and marks a gap under six as "same session: only a probe can settle it", +because two packages built in one session cannot be ordered by mtimes at +all. + +`packagemanager probe` is that answer: it runs every `expect=` in the +installed stacks against what is installed, and a failure names the rebuild. +Timestamps are evidence, a probe reads the artifact. + + packagemanager probe # all of them + packagemanager probe ffmpeg # one + +## Previous session (1.14.167) -- gst10-libav was in no stack + +Right: nothing built it. It carries the ffmpeg-backed decoders (avdec_h264, +avdec_aac and the rest), so without it H.264 -- most video on the web -- is +decoded by nothing, and the symptom is a black frame rather than an error. + +`book gst10-libav` now sits after ffmpeg (entry 1b) and gst10-plugins-base, +which it needs, named as an entry rather than left to a dependency plan so +that stays true. Resulting order: ffmpeg 90, base 116, bad 122, **libav +123**, webkitgtk 125, mpv 126. + +Its probe is an element, for the same reason as the others: the package +installs libgstlibav.so either way, and whether `avdec_h264` registers +depends on what ffmpeg was built with -- the thing that was wrong about +ffmpeg and ALSA. `check webrtc` reports it too, so a missing decoder shows +up beside the camera and the microphone. + +Block 376 asserts both the order and the probe, and names it as MISSING when +the entry is absent (checked by deleting the line). + +On the machine: `packagemanager stack sway --only gst10-libav --run` after +the reclaim and the rebuild pass. + +## Previous session (1.14.166) -- repair the chown, not one package at a time + +The rebuild pass stopped on glib2 with the SAME cause as gst-plugins-bad: +`/usr/include/glib-2.0` is `p_linux-headers`, from lfs-kernel's old +`chown -R "$_hacct" /usr/include`. The whole include tree is that package's, +so every package that ships a header fails to reinstall. The tool fix +(1.14.158) stops it happening again; the machine needs the ownership put +back. + +`packagemanager reclaim` was the right repair but took a named list, and +finding the victims means finding them first. It takes `--all` now: every +installed package, checked against its own pkg.lst, and a package with no +manifest is skipped quietly rather than reported as an error ninety times. + +**The repair, in order:** + + packagemanager reclaim --all --run # ownership back, from manifests + lfs-helper verify --fix # groups, modes, install dirs + packagemanager late-deps --rebuild --run # resume; glib2 first again + +Tested against the exact shape: a directory and a file under +/usr/include/glib-2.0 owned by p_linux-headers but recorded in p_glib2's +manifest -- both returned, directory included. + +Also: `_slice_def` (slicing a python def for the suite's assertions) was +defined inside one block and used by another, where it returned nothing and +made two assertions pass vacuously. It is in the preamble now, beside +`_slice_fn`. + +## Previous session (1.14.165) -- the optional edge is the whole point + +The clusters came out right (freetype2+harfbuzz before fontconfig and cairo, +libdrm+libxcb before mesa, base before bad), and one thing was still +backwards: `mpg123` was rebuilt BEFORE `pulseaudio`, and mpg123's missing +feature IS pulseaudio. The order graph was required+recommended, which are +the edges that break a BUILD -- while the edges this whole pass exists for +are the optional ones. + +The rebuild order now uses required + recommended + optional. That makes the +graph much more cyclic, which is honest rather than convenient, and the SCC +pass already handles it; inside a component the strong edges still decide, +because those are the ones that fail a build rather than a feature. + +`--why` earned its keep immediately: libxcb RECOMMENDS mesa and mesa +REQUIRES xorg7-lib and libdrm, so libxcb/mesa are one genuine cycle -- the +plan puts libdrm and libxcb before mesa, which is the best an ordering can +do there. It also showed the unresolved names are what they should be: +Doxygen, libunwind, lm-sensors, Valgrind -- packages that are not installed. + +## Previous session (1.14.164) -- order the clusters, not the nodes + +Still wrong: freetype2 and harfbuzz came out LAST, after fontconfig and +cairo, which both require freetype2. The cycle-break picked one node and then +went back to the ready scan, which strands the rest of the cluster at the +end -- picking fontconfig out of the cluster made everything that depended +on it ready, and freetype2 was left over. + +A cycle has to be ordered as a UNIT. Tarjan's strongly connected components, +computed iteratively (this graph has 400 nodes, so recursion is not an +option), condensed into a DAG that sorts by construction; inside a component +the rule is unchanged, least dependent and most depended on. The +freetype2/harfbuzz component now comes out before fontconfig and cairo, +asserted in block 375 on that exact shape. + +`late-deps --why <pkg>` prints the edges actually used for a package -- what +each dependency resolved to, how many days apart they were built, and which +names resolved to nothing. When a position still looks wrong, that is the +output to read: an edge that does not resolve is invisible to the sort. + +## Previous session (1.14.163) -- a cycle broken alphabetically is not an order + +Still wrong, and visibly: `gst10-plugins-bad` before `gst10-plugins-base`, +`cairo` and `fontconfig` before `freetype2`. Those are CLUSTERS with no entry +point -- base needs mesa, mesa needs libxcb, bad needs base -- so the +topological sort ran out of ready nodes and the fallback was +`sorted(left)[0]`. Alphabetical order produced exactly the mistake the plan +exists to prevent. + +The pick is now "least dependent, most depended on": fewest unmet +dependencies inside the cluster, then the most packages waiting for it. base +beats bad (bad needs base, nothing needs bad); libxcb beats mesa. And the +cycle is NAMED in the output, with the note that a cycle can need two passes +-- breaking one silently makes the second pass look like a failure. + +Block 375 asserts it on the failing shape itself: three packages in a real +cycle, and base must come out before bad. + +Also: "103 dependency names could not be matched" read as 103 problems. It +now says what it means -- a name that matches no INSTALLED package is +usually one that is not installed, and a package that is not there cannot +have been built after anything. A name you know IS installed appearing there +is the interesting case, and worth reporting. + +## Previous session (1.14.162) -- the order was wrong, and the 119 explain why + +"Are we sure about the ordering?" No, and the tool had already said why in +its own last line: *119 dependency names could not be matched to a package*. +An edge that does not resolve is an edge that does not exist, so the +topological sort ran on what was left -- and put `gst10-plugins-bad` before +`gst10-plugins-base`, which it REQUIRES, and `fontconfig` before +`freetype2`. + +The mapping was being guessed by normalising strings: a dep is written as +the page TITLE (`gst-plugins-base-1.28.1`, `FreeType-2.14.1`), a package +directory is the page ID (`gst10-plugins-base`, `freetype2`), and no amount +of lowercasing bridges that. Every script already records `name_version` -- +the tarball base, which IS the title. `_dep_index` builds the mapping from +that, and both cases now resolve (asserted in block 375 against a real +resolve, not a grep). + +Two more, from the same run: + +- **`--reinstall` is not a question about versions.** With no recorded + version, `update curl --reinstall` answered "nothing to update, 1 it + cannot judge" and did nothing -- for 28 packages whose records predate the + tool. `--reinstall` now plans the rebuild regardless of what is recorded. +- **The rebuild order uses required + recommended.** mesa RECOMMENDS libxcb, + so ordering by `required` alone rebuilds mesa against the old libxcb. + Cycles are expected on that graph and are broken by picking one. + +Re-run `late-deps` after installing this: the order will differ, the +unmatched count should drop a long way, and what is left in it is worth +reading. + +## Previous session (1.14.161) -- there is no host any more + +The machine boots on its own now, and packagemanager still printed, before +every single command: + + These tools differ from the copies on the host: blfs, lfs, ... + On the HOST, run: lfs build-system sync-tools --run + +`/usr/share/lfs/tool-stamps.json` is written by `lfs build-system +install-tools`, which runs on a host against a chroot. After the system +boots, `make install` on the machine itself replaces the tools and nobody +updates the stamp -- so all seven "differ", and the advice names a host that +no longer exists. + +- `make install` writes the stamp now: it records what is installed, so + whoever installs it owns it. +- The check returns early unless it is actually in a chroot + (`/proc/1/root` is not this root). No outside, nothing to be behind. + +On the machine: `make install` from the repo once, and the warning goes for +good. `sync-tools` remains correct for the case it was written for -- editing +the tools on a host and refreshing a chroot. + +## Previous session (1.14.160) -- the rebuild pass rebuilt nothing + +Two bugs in 1.14.159, both visible in one run on the machine: + +- **`install` does not rebuild.** The plan emitted + `packagemanager install <pkg> --run --regenerate`, which answered + "Nothing to do -- everything is installed and up to date" 28 times in a + row -- correctly, and in a way that read like a successful pass. The verb + that rebuilds at the same version is + `packagemanager update <pkg> --reinstall --regenerate --run --yes`, and + that is what `_rebuild_cmd` emits now, in the plan and in the per-package + hint. +- **A recommended edge is still a dependency.** The cycle test walked only + `required=`, so the report advised `put mesa before libxcb` -- and mesa + needs libxcb; BLFS lists that edge as recommended. Same for cairo -> glib2, + which is why glib2 showed up as orderable. Advice that cannot be taken is + worse than none, so the "does an order exist" walk uses + required + recommended. + +Also widened the build-tool list (mako, setuptools, wheel, hatchling, +pytest, gi-docgen, gperf, unifdef) -- those were showing as missing features. + +## Previous session (1.14.159) -- three answers, not one list + +`late-deps` on the real machine returned 40 packages in one list, and they +are not one problem: + +- **An order exists** (ffmpeg/alsa-lib, gst-plugins-bad/libnice, mpg123, + libheif, pipewire, sdl3, v4l-utils): the dependency does not need the + package, so it can be built first. Each line now says + `in the stack: put alsa-lib, sdl2 before ffmpeg` -- a stack edit, fixed + for the next machine. +- **No order exists** (glib2/cairo+dbus, libxcb/mesa, xwayland/mesa, + xorg7-lib/dbus): the dependency REQUIRES the package. cairo cannot precede + glib2. The only fix is a second pass, and saying "move it earlier" would + be advice that cannot be taken. Detected by walking each package's own + `required=` list. +- **Documentation and build tools** (graphviz against six packages, + docutils, ruby, git, fcron, cython): rebuilding gains documentation, not + features. Summarised in one line; `--include-docs` to see them. + +`late-deps --rebuild` prints the plan in DEPENDENCY order -- rebuilding a +library after the thing that links it means doing it twice -- and +`--rebuild --run` carries it out, stopping at the first failure with what is +left. A finished stack run now ends by naming the question, because a second +pass is part of the job rather than an afterthought. + +For the machine: `alsa-lib` before `ffmpeg`/`sdl3`/`v4l-utils`/`pipewire` is +already the stack's order since 1.14.156, so those entries only need the +rebuild. The circular ones stay circular -- they need the second pass on +every fresh build, which is what the reminder is for. + +## Previous session (1.14.158) -- one line owned every header on the system + +gst-plugins-bad could not install `mse-enumtypes.h` into +`/usr/include/gstreamer-1.0/gst/mse`, because that directory belonged to +**p_linux-headers**. The repair handed it one file per round; after three +rounds the bundled-copy heuristic concluded meson was vendoring +linux-headers. Every step of that reasoned correctly from a wrong fact, and +the wrong fact was one line in `lfs-kernel`: + + chown -R "$_hacct:$_hacct" /usr/include + +A package owns what it INSTALLED, not the directory it installed into. +`make headers` output is now listed with `find -printf '%P\n'` before the +copy and only those paths are chowned; `/usr/include` itself stays root's, +because every package that ships a header shares it. + +**Repair on the machine** (the ownership is already wrong there): + + packagemanager reclaim gstreamer10 --run + packagemanager reclaim gst10-plugins-base --run + packagemanager reclaim gst10-plugins-bad --run + packagemanager install gst10-plugins-bad --run --regenerate + +`lfs-helper fix-ownership --run` is the wider net if other packages' headers +were taken too -- everything under /usr/include that is not a kernel header +is suspect on that machine. + +### packagemanager late-deps + +The audit the last two sessions kept needing. Every generated script records +the page's own required/recommended/optional lists; every pkg.lst is written +when the install finished. So: which packages were built before something +they can use? + + packagemanager late-deps # all of them + packagemanager late-deps ffmpeg # one + packagemanager late-deps --kind recommended,optional,required + +It reports the gap in days and the rebuild command, ignores gaps under a day +(pkg.lst is rewritten by reload-pkg-list, so packages from one run differ by +minutes), and counts the dependency names it could NOT match to a package -- +a silent miss would under-report exactly what is being looked for. Tested +against a synthetic tree reproducing both the ALSA and the WebRTC cases. + +## Previous session (1.14.157) -- a library is not a plugin + +The browser: "GstWebRTC plugin not found." gst-plugins-bad *is* installed, +`verify` is clean, and `libgstwebrtc-1.0.so` is in its file list -- because +that library is always built. The **webrtcbin element**, which is what a +browser loads, is compiled only if libnice was there at configure time. Their +`p_gst10-plugins-bad/install_last` says `generated by blfs 1.14.43`: built +long before the WebRTC block was added to the stack, so libnice did not +exist yet. Same shape as the ALSA bug, one level further in -- the artifact +that proves the feature is an entry in the element registry, not a file. + +- `opus` and `libvpx` are stack entries now, before gst10-plugins-base and + -good: a call negotiates Opus and VP8, and both plugins are autodetected. +- `expect=` probes on all three gst entries, by element: + `gst-inspect-1.0 webrtcbin && dtlssrtpenc && srtpenc` for -bad, + `opusenc` for -base, `v4l2src && vp8enc` for -good. +- **`check webrtc`** (`stacks/webrtc.sh`): the whole chain in one report -- + eight elements with the library each needs and the command that rebuilds + it, the .pc files, /dev/video* and the video group, what + `p_webkitgtk/install_last` was built with, and a warning when a package's + script is OLDER than `nice.pc` (built before its dependency existed -- + exactly this bug, detectable from mtimes). +- machine.conf gained a commented `[gst10-plugins-bad]` block: setting + `webrtc = enabled` turns silent autodetect into a build failure. Commented + because the option NAMES are machine-checkable, not memorable -- it says + how to confirm them against the source on disk. + +Fix on the machine: `packagemanager stack sway --only gst10-plugins-bad --run` +(libnice, libsrtp and usrsctp are installed now, so it picks them up), then +`--only webrtc --run` to confirm. webkitgtk does not need rebuilding for the +element -- the plugin is loaded at run time. + +## Previous session (1.14.156) -- ffmpeg and SDL3 without ALSA + +Both autodetect ALSA, and both were built before alsa-lib existed. `book +alsa-lib` sat at entry 7, while ffmpeg is built inside **mesa's** plan +(entry 2) and SDL3 inside gstreamer's. They configured cleanly, installed +cleanly, passed `have=`, and had no sound. Nothing failed, so nothing said +so. + +The stack already stated the rule, for harfbuzz: an optional dependency of +something built inside a plan can only be placed by ENTRY ORDER, because a +plan builds its own tree in the book's order, not the stack's. It was simply +not applied to audio. + +- `book alsa-lib` is entry 1b now, before mesa. `ffmpeg` and `sdl3` are + named as entries there too, so they are built in this order rather than + wherever a plan happens to reach them. +- **`expect="<shell>"` on a stack entry**: run after the entry installs, and + it must succeed. A failing probe does NOT record the entry as done, so the + next run rebuilds it. Two are shipped: + `ffmpeg -hide_banner -devices | grep -qw alsa`, and for SDL3 (which + dlopens libasound rather than linking it, so ldd says nothing) + `strings /usr/lib/libSDL3.so | grep -q libasound`. +- Fix on the machine: `packagemanager stack sway --only alsa-lib --run` + (already built -- it will say so), then + `--only ffmpeg --run` and `--only sdl3 --run`. Anything else that wanted + ALSA and was built early -- mpv, pipewire, gstreamer -- is worth the same + treatment; `--only <name> --run` each. + +Block 375 of the suite asserts the order out of the stack file itself, and +fails when alsa-lib is moved back after ffmpeg (checked both ways). + +## Previous session (1.14.155) -- a package is not a service + +The network stack ran on a second machine and stopped at 9 of 9: + +- **`install-service-dhcpcd` was missing from blfs-bootscripts.** The dhcpcd + package installs no service script; `/lib/services/dhcpcd` comes from that + target, so `SERVICE=dhcpcd` and the wpa hand-off had nothing to run. + Adding `book dhcpcd` in 1.14.153 was half the job. Re-run + `packagemanager stack services --only blfs-bootscripts --run`. +- **`check net-wifi` failed the stack for the user's own files.** No + /etc/sysconfig/ifconfig.wlan0 is not something the stack was ever going to + write -- it holds an SSID. Two levels now: `!!` is the stack's and exits 1, + `->` is yours and does not. The step completes and still lists the work. +- **Two contradictory lines about one file.** It said + `wpa_supplicant-__wifi0__.conf has no psk` and, two lines later, that it + `holds a key and is world-readable`. The permission warning is now tied to + actually holding a key, and a `__placeholder__`/example/template name is + reported as a template to copy rather than a broken config. + +## Previous session (1.14.154) -- power as a group, and two dead udev rules + +The reference machine delegated halt/poweroff/reboot/pm-suspend by hand: +`4550 root:nimgnu_power`, plus a boot script re-applying modes on +/sys/power/state because /sys does not survive a reboot. That is the same +shape as `seatd-<user>` and it was in no stack. It is `cfg power` in +services.stack now (`stacks/power.sh`): the group (from `collector_prefix`, +or `POWER_GROUP`), the desktop user's membership read back afterwards, the +setuid bits with the mode confirmed, and **one** boot script in +`/etc/rc.d/init.d/power-perms` symlinked into `rcsysinit.d` -- the machine +had a second full copy under `rc3.d`, which is a second place to edit. + +Two pieces of that setup could never have worked, and both looked like +configuration that did: + +- `SUBSYSTEM=="power", KERNEL=="state", MODE="4550"` -- /sys/power/state is a + kernel attribute with no uevent, so udev is never asked about it. And the + mode is wrong twice: setuid is meaningless on a sysfs file, and 550 has no + write bit, so a group member could not write `mem` to it even if the rule + had applied. The boot script is what does the work, and it uses 0660. +- `ACTION=="add", KERNEL=="bootlog", OWNER="nimgnu_bootlog"` -- /run/bootlog + is an ordinary file the bootscripts write, not a device. No device, no + match, no owner ever applied: **the `nimgnu_bootlog` account is not needed + and never was.** Where the boot log goes and who may read it is `BOOTLOG` + in /etc/sysconfig/rc.site. + +`power.sh` reports both if it finds them in /etc/udev/rules.d, and does not +delete anything. + +## Also this session (1.14.153) -- dhcpcd was never in the stack + +`wlp3s0` exists, so 7.2.4 booted and the driver found its firmware. The link +would not go anywhere because **nothing in any stack installs a DHCP +client** -- the network stack had wpa_supplicant, libnl, libmnl, libnftnl, +iptables and net-tools, and `lfs config network dhcp` deliberately writes no +static interface files *because* a client from BLFS is supposed to follow. +Nothing followed. `book dhcpcd` is in `network.stack` now. + +Beside it, `check net-wifi` (`stacks/net-wifi.sh`): a read-only report of +the five separate things that each produce "wifi does not come up" -- +missing client, missing `ifconfig.<iface>`, a file named for the wrong +interface, missing per-SSID config, or a `SERVICE=` no installed script +implements. It reads the wireless interface names out of `/sys/class/net` +(a file called `ifconfig.wifi0` does nothing for `wlp3s0`) and the hand-off +variable out of `/lib/services/wpa` rather than remembering which release +uses which. It also flags a world-readable psk file. + +The new script needed no list edits: the Makefile and the chroot refresh +both walk `stacks/` (the "derive lists" rule, already applied there). + +## Also this session -- the regulatory database, and a latent failure + +The run came back healthy: 686 of 723 amdgpu declarations installed, 32 of +50 iwlwifi, the four embedded blobs present, `iwlwifi-cc-a0-77.ucode` flat +where the loader wants it. One item left, and it was homework rather than a +fix: + +- **wireless-regdb is now installed by the tool.** `_install_regdb` reads + kernel.org's listing, takes the newest release, unpacks it as + `p_linux-firmware` and installs `regulatory.db` + `.p7s` into + /lib/firmware -- one directory, one owner. `REGDB=yes` in kernel.conf + (default), and `--fix` does it. Tested against a local http mirror; the + sed that parses **kernel.org's real listing is unverified** (no network to + kernel.org from the authoring sandbox), and if it comes back empty the + command says so and prints the two manual commands. +- **A latent failure in the shipped config.** 1.14.147 put + `regulatory.db*` in FIRMWARE_ONLY. A pattern that matches nothing in the + repository is an error, so the next full `lfs-kernel firmware` would have + failed outright, blaming the pattern. Removed from the shipped value, and + `_install_firmware` now recognises an entry that lives elsewhere and + installs it from there instead of failing. +- **A message that contradicted the note above it.** The candidate section + said the boot list "is the reliable one" two lines under a note saying a + chroot's boot list is the host's. In a chroot it now says to boot this + kernel and run it again there. + +## Previous session (1.14.152) -- the chroot host is another machine + +`ls` settled the wifi question: `/lib/firmware/iwlwifi-cc-a0-77.ucode` is +there, flat, 1368100 bytes -- the file 7.2.4 asked for. It is in place, and +`intel/iwlwifi/` has the -50 to -77 series beside it. Nothing more to +install for the AX200; **boot 7.2.4 and look**. + +Three things the report was getting wrong about its own authority: + +- **A satisfied request was skipped in silence**, so a run with everything + in place looked identical to one that had not looked. Boot and config + lines now print `ok ... installed now`, which in a chroot is the only + reassuring thing on the page. +- **The candidate summary counted only what is absent.** It says "1 of 2 + declared file(s) not installed" now, so a module with most of its + declarations present reads as such. +- **The chroot note was too weak.** The host may be another distribution on + other hardware -- it is here -- in which case its boot log can neither + confirm nor deny anything about this tree. And no static check can + promise a driver will find its firmware: iwlwifi derives the name from + the RF and CRF ids it reads off the card at run time. The report says + both. + +Still open for the machine: `regulatory.db` (wireless-regdb, not +linux-firmware) and the boot itself. + +## Previous session (1.14.151) -- say what IS there + +The 7.2.0 boot logs no iwlwifi firmware failure, so that kernel's wifi +firmware loaded: a cc-a0 blob is on this disk somewhere, and 7.2.4's driver +would not take it. The report said nothing about either fact. It now checks, +for every proven miss, the two things that change what you do next: + +- **the same file under another path.** A `FIRMWARE_ONLY` entry keeps the + repository's path, so `intel` installs + `/lib/firmware/intel/iwlwifi/iwlwifi-cc-a0-77.ucode`, and the loader asks + for `/lib/firmware/iwlwifi-cc-a0-77.ucode` and nothing else. `--fix` + copies it into place, no network and no repository needed. +- **an older firmware API installed.** `iwlwifi-cc-a0-72.ucode` present + while the driver requires 77: the number is the API the driver demands and + it does not fall back, so the answer is a newer checkout, not a different + path. The line names the sibling that is there, and the newest three the + checkout has. + +`_fw_neighbours` returns the path in `_FW_LOCAL` rather than on stdout -- +the first version had the caller's `$(...)` swallow the whole explanation +along with it, so the report said only "installed". + +## Previous session (1.14.150) -- two bugs the chroot run exposed + +The modules scan worked and printed 56 MISSING lines. Both problems were in +how that list was produced and presented. + +- **`git ls-files` lies under a sparse index.** Git collapses every excluded + directory into a single entry, so the listing was `intel/` and every blob + under it looked absent: 56 files reported as "this checkout has no file by + that name" while they were all in the repository. `_fw_index` now uses + `git ls-tree -r --name-only HEAD`, which reads the commit and no sparse + setting touches. (Reproduced here: `ls-files --sparse` prints `intel/`.) +- **A declaration is not a failure.** `modinfo -F firmware amdgpu` lists + firmware for every chip amdgpu supports -- Vega, Sienna Cichlid, + Aldebaran, gc_12_1_0 -- and iwlwifi lists five RF families this laptop + does not have. All true, and the conclusion ("56 missing") was wrong: a + driver loads what its own chip needs. `modules` lines are now collected + per module and summarised in one line each as CANDIDATES; only `boot` and + `config` lines are treated as things to fix. `--fix --all` installs the + declared ones anyway. + +The `boot` list stays the reliable one, which is why booting 7.2.4 and +re-running is still worth doing: it named `iwlwifi-cc-a0-77.ucode` exactly. + +## Previous session (1.14.149) -- answer in the chroot + +"Why not make me able to run in chroot?" -- fair. Refusing to answer where +the work happens is not an answer, and dmesg was the wrong source anyway: it +describes the boot that is running, and firmware is requested at boot, so it +can only ever report the PREVIOUS kernel's experience. + +`--missing` now derives the list from three sources and labels every line +with the one that named it: + +- `boot` -- the failed loads in dmesg (in a chroot, the host's; said so) +- `modules` -- every modalias under /sys/devices resolved against + `/lib/modules/<ver>/modules.alias` (`modprobe -S <ver> -R`), then + `modinfo -k <ver> -F firmware`. This machine's hardware, this tree's + kernel, no boot required. +- `config` -- `CONFIG_EXTRA_FIRMWARE`, which a running kernel never + complains about because the image already carries it. + +Which kernel's modules: `VERSION` from kernel.conf first, then `uname -r`, +then the newest under /lib/modules -- and the report names it. `uname -r` in +a chroot is the host's. + +Two more things the output showed: + +- **`regulatory.db` is not in linux-firmware at all.** It is released as + wireless-regdb, so "update the checkout" was advice that could never + work. There is now a small table of firmware that lives elsewhere, with + the URL and the two commands. +- **A nested blob with a flat request cannot be expressed in + FIRMWARE_ONLY.** An entry is copied to the same relative path, so + suggesting `intel/iwlwifi` for a driver asking for + `iwlwifi-cc-a0-77.ucode` would install it where the loader never looks. + The suggestion is now only made when the paths match, and the mismatch is + named. Relatedly, `_install_firmware` copied a nested entry with + `cp -a $src/$d /lib/firmware/`, dropping the directory (the deferred item + from 1.14.147): it uses `cp -a --parents` from inside the checkout now. + +## Previous session (1.14.148) -- three bugs in 1.14.147's firmware command + +All three were found by the user's first real `firmware --missing --fix`, and +the first two are old rules re-broken: + +- **A swallowed error became a confident wrong answer.** `git -C "$src" + ls-files 2>/dev/null` returns nothing and rc 128 when the repository + belongs to `p_linux-firmware` and the command runs as root (`fatal: + detected dubious ownership`). The empty result was printed as + "linux-firmware does not have a file by that name at all", about + `regulatory.db`, which is in the repository. There is now one `_fw_index`, + it passes `-c safe.directory=$src`, and a failed lookup reports UNKNOWN + with git's message rather than "absent". +- **A check that did not know what it was measuring.** `dmesg` is a syscall + against the running kernel, so in a chroot it works and answers about the + host's boot. `--missing` now detects the chroot and says the list is the + host's. +- **`local a="$2" d="/lib/firmware/$a"`** reads `$a` before that same + `local` assigns it; `set -u` killed `_fw_place` on its first real call. + Split into two statements. + +Also: `--fix` no longer touches the sparse checkout at all. It read one blob +with `git show HEAD:<path>`, because the old path widened the checkout and +fell back to `sparse-checkout disable` -- which in cone mode would have +materialised all ten gigabytes to install one 2 MB file. And an unchanged +`FIRMWARE_ONLY` is no longer printed as though it were a fix. + +Tested here: all four `_fw_place` branches (on disk, index-only, +nested-repo-path to flat-request, absent), the ownership failure, and blocks +109, 374, 375. + +## Previous session (1.14.147) + +- `lfs-helper setid [--run]`: the setid ledger is no longer emptied by the + one notice that prints it. `setid` lists what was refused, checks a small + on-disk list (su, passwd, chfn, chsh, gpasswd, newgrp, newuidmap, + newgidmap, dbus-daemon-launch-helper) for the case where nothing was + recorded, and `--run` chowns to root, sets the mode, and reads it back. + `verify` now counts outstanding requests as a problem. +- The **install** wrapper records refused setid modes, against the file that + actually landed, and only after the install succeeded. The chmod wrapper + had done this since 1.14.82; the same rule, one of two places. +- `lfs-kernel firmware` (install firmware with no kernel build) and + `lfs-kernel firmware --missing [--fix]`: reads the failed loads out of + dmesg, finds each file in linux-firmware's git *index* (a sparse checkout + has removed the very file being looked for), installs it at the path the + driver asked for, widening the sparse checkout by that one path, and + confirms the copy. The firmware functions were hoisted above the build to + make this reachable without one. +- `lfs-kernel --help` derives its own slice from the header instead of a line + range that a new command pushes out of view. +- Suite: `--list/--from/--to/--only`, a `skip()` counter, rc.site skips with + a reason, and two new blocks (374, 375). Blocks 109, 293, 373, 374, 375, + 204 run green here; the rest of the suite was not run. + +## Previous session (1.14.146) + +- `BUILD.md` written; README tightened (kernel caveat was stale: it said + nothing builds a kernel, `lfs-kernel` does). +- `lfs-helper`: the end-of-build message pointed at book 10.3 by hand; + now points at `kernel.conf` + `lfs-kernel`. +- `stacks/sway-session.sh` hardcoded one username as the desktop user while + `mytools.sh` read `/etc/pkgusr/desktop-user` -- two settings, one + question. Both now take `LFS_DESKTOP_USER` or that file, and both write + it. Installed copies of the stack file are unedited, so `make install` + updates them in place. +- Not run: the regression suite (sandbox slot). `bash -n` / `py_compile` + and `--version` only. + +## 1.14.321 -- git reinstall, a stack on the svn book, @setup + +- `install <gitpkg> --reinstall [--clean]` builds from the package's own + script and never asks the book. It used to ask three times (map, + `blfs debug`, `blfs search`), take a minute and say "not in the BLFS + book" for imv. A stack git/tar entry that was never built is found too. + `book_version` remembers a miss, so no name is looked up twice. +- `from-book <selector>` in a stack file: its book entries read that book + (versions, deps, scripts) -- permanently, so `update` and `info` agree. + Deps outside the stack keep their own book; a page only the stack book + has is read from it. `blfs path` resolves a selector to the cached file + once per run; `stack`/`update --stack` refresh it (`blfs fetch`) once. + One version map per book (`book-versions-<md5>.json`). The default map + is the default book, no longer "newest file on disk" -- that would have + become svn the moment it was fetched. media.stack now says `from-book + svn`. NOT tested against a real svn book: check the selector exists + (`blfs books --discover`). +- `stack` prints the avoid COUNT; `stack <name> --avoided` lists them. +- app-users.conf `@setup <account|*> <cmd|URL|script>`: runs as the account, + once per line text (`/var/lib/pkgusr/app-setup/<account>`), + `APP_USERS_RERUN=1` repeats. +- Tested: py_compile, bash -n, stack parse (media entries only get svn + inside sway), `--avoided`, the git path with stubs. Suite not run. ## Deferred tool work None of this blocks the machine. -- The regression suite exceeds one execution slot; make it splittable - (`--from N`, or per-tool) before it grows further. - Three functions move stack files (Makefile, chroot refresh, `--take-new`); they should be one, and the whole `.new` mechanism deserves rethinking around a single source of truth. @@ -218,9 +3816,14 @@ None of this blocks the machine. the stack, not a package installed by hand. - `--check-refs` cannot see tar URLs. A `have=`-style check for tarballs would close the gap libnftnl exposed. +- The preamble of the suite (everything before block 1) always runs, and a + block that needs a variable an earlier block computed dies on `set -u` + rather than skipping. Only tool-path assignments are hoisted. +- The `modules` source needs `modprobe`/`modinfo` (kmod) and a mounted + /sys; without either it falls back to the boot list silently apart from a + header line. It was tested with stubs here, never against real kmod. - The "busy tree" test is flaky: it races a `sleep` in the process table and should use a private marker. -- The rc.site test should say it needs a SysV book rather than failing. - The whole-book invariants (954 scripts, 0 syntax errors, 0 empty install phases) have caught or confirmed nearly every filter change. Extend them to build phases, listing the known-empty pages by name. diff --git a/Makefile b/Makefile index 5d7850f..81ac384 100644 --- a/Makefile +++ b/Makefile @@ -23,7 +23,7 @@ SYSCONFDIR ?= $(DESTDIR)/etc # The four tools, the book merger, and the phase runner every generated # install script sources (it goes in bin so `. lfs-phases` finds it on PATH, # on the host, in the chroot and on the built system alike). -TOOLS := lfs lfs-helper packagemanager blfs lfs-sysvbook lfs-phases lfs-kernel +TOOLS := lfs lfs-helper packagemanager blfs lfs-sysvbook lfs-phases lfs-kernel lfs-backup COMPLETION := lfs-completion.bash TESTS := test_lfs_crosschain.sh @@ -39,7 +39,7 @@ INSTALL ?= install # mkdir -p leaves an existing directory exactly as it is. MKDIR ?= mkdir -p -.PHONY: all install uninstall check clean help +.PHONY: all install uninstall check stamp clean help all: help @@ -47,6 +47,7 @@ help: @echo "make install [PREFIX=...] [DESTDIR=...] install the tools" @echo "make uninstall [PREFIX=...] remove them" @echo "make check run the tests" + @echo "make stamp stamp the stack files with the tools' version" @echo "make clean remove scratch files" @echo "" @echo "installs into: $(PREFIX)/bin" @@ -60,6 +61,10 @@ install: rm -f "$(BINDIR)/packagemanager_install" \ && echo " removed retired packagemanager_install"; \ fi + @# lfs-config is retired (1.14.291): lfs-backup replaces it. + @if [ -e "$(BINDIR)/lfs-config" ]; then \ + rm -f "$(BINDIR)/lfs-config" && echo " removed retired lfs-config (now lfs-backup)"; \ + fi @for t in $(TOOLS); do \ if [ -f "$$t" ]; then \ $(INSTALL) -m 755 "$$t" "$(BINDIR)/$$t" && echo " $$t"; \ @@ -67,6 +72,16 @@ install: echo " !! missing: $$t" >&2; exit 1; \ fi; \ done + @# THE STAMP RECORDS WHAT IS INSTALLED, so whoever installs writes it. + @# It was written only by `lfs build-system install-tools`, which runs + @# on a host against a chroot -- so on a BOOTED system every later + @# `make install` left the stamp describing the old copies, and + @# packagemanager announced "these tools differ from the copies on the + @# host" and told the reader to run sync-tools on a host that no longer + @# exists. + @$(MKDIR) $(DESTDIR)/usr/share/lfs + @{ echo '{'; first=1; for t in $(TOOLS); do [ -f "$$t" ] || continue; m=$$(md5sum "$$t" | cut -d' ' -f1); [ $$first = 1 ] || echo ','; first=0; printf ' "%s": "%s"' "$$t" "$$m"; done; echo; echo '}'; } > $(DESTDIR)/usr/share/lfs/tool-stamps.json + @echo " recorded $(DESTDIR)/usr/share/lfs/tool-stamps.json" @echo "==> stacks -> $(DESTDIR)/etc/pkgusr/stacks" $(MKDIR) $(DESTDIR)/etc/pkgusr/stacks @# A STACK FILE IS THE USER'S. This overwrote it unconditionally, so @@ -106,6 +121,17 @@ install: @# kernel.conf is a file to EDIT BEFORE building, so it is installed @# rather than written on first run -- and, like a stack file, an @# edited one is kept and the new one lands beside it. + @# backup.conf: what lfs-backup carries. Yours is kept, like kernel.conf. + @if [ -f backup.conf ]; then \ + $(MKDIR) "$(DESTDIR)/etc/pkgusr"; \ + d="$(DESTDIR)/etc/pkgusr/backup.conf"; \ + if [ ! -e "$$d" ] || cmp -s backup.conf "$$d"; then \ + $(INSTALL) -m 644 backup.conf "$$d" && echo " backup.conf"; \ + else \ + $(INSTALL) -m 644 backup.conf "$$d.new" \ + && echo " backup.conf -> KEPT YOURS ($$d.new)"; \ + fi; \ + fi @if [ -f kernel.conf ]; then \ d="$(DESTDIR)/etc/pkgusr/kernel.conf"; \ if [ ! -e "$$d" ] || cmp -s kernel.conf "$$d"; then \ @@ -184,11 +210,26 @@ uninstall: @echo "Left alone (they are yours):" @echo " $(SYSCONFDIR)/pkgusr/skel-u_xdg/.bash_profile" @echo " /usr/share/lfs books, settings, snapshots" - @echo " /etc/pkgusr packagemanager settings" + @echo " /etc/pkgusr packagemanager settings, kernel.conf, backup.conf" check: @bash $(TESTS) ./lfs +# ONE ARTIFACT, ONE NUMBER -- and a rule a human has to remember is a rule +# that drifts. The stack files each carry `# stack-version:`, the suite +# asserts it equals the tools' version, and it was ten releases stale before +# anyone ran the suite. This stamps them from `lfs --version`, so cutting a +# release is `make stamp` rather than seven sed commands. +stamp: + @v=$$(./lfs --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1); \ + [ -n "$$v" ] || { echo "cannot read a version from ./lfs"; exit 1; }; \ + for f in stacks/*.stack; do \ + grep -q '^# stack-version:' "$$f" \ + || { echo "$$f has no stack-version line -- add one"; exit 1; }; \ + sed -i "s/^# stack-version: .*/# stack-version: $$v/" "$$f"; \ + done; \ + echo "stamped $$(ls stacks/*.stack | wc -l) stack file(s) with $$v" + # Nothing here is compiled, so clean removes only what running the tools here # leaves behind: Python bytecode and the test suite's scratch directories. # It never touches an installed tree -- that is what uninstall is for. diff --git a/NEXT-TASK.md b/NEXT-TASK.md new file mode 100644 index 0000000..205bfa3 --- /dev/null +++ b/NEXT-TASK.md @@ -0,0 +1,142 @@ +# First task for the next session + +The machine boots, `su` works, the desktop runs and the application accounts +are in daily use. Everything below is configuration on the machine, not a +build. The tools are at 1.14.320. + +## 0. Repair what the lease took (do this first) + +`/usr/bin/su` and six `/usr/sbin/iptables*` links are owned by `p_webkitgtk`, +and the links are recorded in its `pkg.lst` — so `update webkitgtk --clean` +would delete them. 1.14.310 shuts the doors; the machine still has to be put +back: + + packagemanager disown --all # what it would hand back + packagemanager disown --all --run # hand it back, and clean the records + lfs-helper setid --run # put the setuid bits back + su - # check + +Then the same rebuild that broke it is safe to repeat. + +## 0a. The stacks are reorganised (1.14.320) + + make install + ls /etc/pkgusr/stacks/*.stack.new # should print nothing + +wayland.stack is new (the base sway.stack used to carry), sway.stack +includes it, and zathura/imv moved to apps.stack, jq to tools.stack. +Nothing needs rebuilding; `packagemanager stack sway` still means the whole +desktop. See HANDOFF 1.14.320. + +## 0b. The shared hub and yt-dlp + +Shared data moved to /home/Shared by hand (commands from the 1.14.315 +session), then: + + packagemanager stack services --only app-users --run # links + 0700 homes + packagemanager stack media --only yt-dlp --run + +Log in again for the SHARED_* variables to name /home/Shared. + +## 1. The locale (one command, then log in again) + +Nothing sets `$LANG`. lfs-helper wrote `/etc/profile.d/i18n.sh` with the +export commented out, and there is no `/etc/locale.conf` — so every program +runs in the C locale and foot says so at every start. Both are fixed in the +tools (1.14.296–1.14.298); the machine still has to run the step: + + lfs config locale de_DE.UTF-8 # if it is not that already + packagemanager stack services --only shell-env --run + # log in again, or right now: . /etc/profile.d/i18n.sh + +It writes `/etc/locale.conf`, replaces that i18n.sh (backed up), and adds one +line to `/etc/bashrc` so a new terminal picks it up without a login. + +## 2. The launchers written before 1.14.295 + +A `.desktop` entry does nothing if its launcher does not give `su` a pty, and +a launcher these tools did not write is left alone — so the old ones survive. +`~/bin/claude` is the last hand-made one: + + rm ~/bin/claude + packagemanager stack services --only app-users --run + +Then click a menu entry. If one still does nothing, run the same name in a +terminal first: that path prints what `su` says. + +## 3. Wireless — the only piece of networking still missing + +Wired works and starts itself (`enp2s0f0`, `enp5s0`). The wireless interface +is **`wlan0`** on this kernel. + + packagemanager stack network --only net-wifi --run + +It prints both files to write: `/etc/sysconfig/ifconfig.wlan0` and a per-SSID +`/etc/sysconfig/wpa_supplicant/wpa_supplicant-wlan0.conf` (chmod 600). Nothing +writes the second one for you — it holds the key. + +## 4. make-ca's weekly job + +The systemd book enables a timer, which the shim skips. fcron is installed: + + cat > /etc/cron.weekly/update-pki.sh << "EOF" + #!/bin/bash + /usr/sbin/make-ca -g + EOF + chmod 754 /etc/cron.weekly/update-pki.sh + +## 5. Try the new commands once + +So their first use is not in an emergency: + + packagemanager service # list; then: service sshd start + packagemanager stack dev --run # gdb + +## 6. Run the test suite once, on the machine + + make check + +It has not been able to report a verdict for some time (1.14.303): `lfs` would +not load on Python < 3.12, and the cleanup trap fired in every subshell and +removed the counters, so the final line printed blank. Both are fixed. Here it +reads 761 PASS / 84 FAIL / 1 SKIP (1.14.320), and those 84 are all "this is not a +package-user system" -- the wrappers, the book and the chroot blocks have +nothing to test against. On the machine they should mostly turn into passes; +whatever is left is a real list worth reading. + +Known and NOT fixed: `--only N` / `--from N` can cut a here-document in half +and lose helpers an earlier block defined, so a selection can report failures +a full run does not. Trust the full run. + +## Still open, none blocking + + * services.stack is misnamed: it holds bash-completion, shell-env, + linux-pam, shadow-pam, app-users, power and mytools as well as the boot + services. `system.stack` would say what it is. Not renamed yet: the + machine's progress file (services.done), NEXT-TASK, BUILD.md and ~20 + test references use the name. Decide, then do it in one release. + * the scdoc, grim, slurp and swayidle entries have not been built yet -- + first real run is on the machine (sr.ht / freedesktop are unreachable + from the sandbox). + * a plan does not put an update of an installed-but-too-old dependency in + front of the package that needs it (vcs_versioning wanted packaging >= + 26.2; HANDOFF 1.14.314). + + * poppler: re-apply the reverted `cd` rule (the patches only "did not fire" + because of the stale index cache, fixed in 1.14.262); diff every generated + script before and after. Do NOT `--regenerate` poppler first. + * shadow's login.defs block overwrites `.orig` on a second run. + * MIT-Kerberos configure transcript blocks (not in any stack). + * `dbus` was built before `[dbus] systemd = disabled` went into + machine.conf — rebuild or accept. + * A stray `.service` from at-spi2-core in `/usr/lib/systemd/user`. + * `/etc/bashrc` has a mangled line (`# End /etc/bashrcCK=$(tput setaf 0)`) + from an append without a leading newline. Harmless: `BLACK` is unset and + nothing uses it. + +## Settled since this list was last written + + * "cfg entries are stack-cached, so a repair script is skipped once it has + run." Answered by convention rather than by changing the cache: every cfg + entry that must stay true now carries `expect=`, which is re-checked on + every run — shadow-pam, app-users, shell-env, bootscripts. diff --git a/README.md b/README.md index 4c79fc7..a81bf07 100644 --- a/README.md +++ b/README.md @@ -70,13 +70,19 @@ lfs build-system session # the interview: partition, prefixes, locale, logi lfs run # everything up to the chroot, then enter it lfs-helper build-all # inside: chapters 7-9 packagemanager bootstrap --run # wget, Python modules, BLFS book, certificates -packagemanager stack sway --run # then network, services +packagemanager stack # list the stacks +packagemanager stack sway --run # then network, services; apps and tools when wanted lfs-kernel # the kernel, from /etc/pkgusr/kernel.conf ``` `lfs run` and `build-all` continue from wherever they stopped. Run them again after a reboot, a failure, or a cancel. +A stack can read its packages from another book. `from-book svn` in a stack +file puts every book entry below it on the development book. Dependencies +outside the stack stay on the default book. `packagemanager stack <name> +--avoided` lists what the stack never builds; the plan shows only the count. + | Command | Does | |---|---| | `lfs build-system next` | What is done, what comes next | @@ -171,6 +177,7 @@ anything; it does only what is left. ```sh packagemanager install <pkg> --recursive --run packagemanager update <pkg> --run +packagemanager install <gitpkg> --reinstall --clean --run # git packages: no book lookup packagemanager which-package /usr/bin/foo blfs order <pkg> # build order, dependencies first blfs sources <pkg> # download URLs @@ -194,6 +201,55 @@ packagemanager user create firefox --shared --share-dir --launcher The session group is read from the runtime directory, not assumed. Set it up once for your own account; every shared user then joins whatever is there. +Many accounts at once come from `/etc/pkgusr/stacks/app-users.conf`, one line +per launcher, applied by `packagemanager stack services --only app-users +--run`. A line `@setup <account|*> <command | URL | script>` runs extra +setup as that account, once; a changed line runs again. Undoing them: + +| Command | Removes | +|---|---| +| `packagemanager user remove-launcher <name> --run` | one launcher, its menu entry and icon | +| `packagemanager user delete <account> --run` | the account; its home is archived to `/root` first | +| `packagemanager user purge --run` | all of them, plus the session group, the profile fragments and the PAM wiring | + +A home with a browser profile in it is archived to `/root/u_<name>-home-<date>.tar.gz` +(0600) before the account goes — logins, cookies and downloads that nothing +else can give back. A home that was never used is not archived. Drop the +archives when you no longer want them: `rm /root/u_*-home-*.tar.gz`. + +Each is a dry run without `--run`. `$SHARED_*` comes from +`/etc/profile.d/shared-dirs.sh`. `/etc/profile.d` is read by **login** shells +only, so a terminal opened inside your session shows what it inherited at +login. The step also sources the fragment from `/etc/bashrc`; any shell can +refresh itself with `. /etc/profile.d/shared-dirs.sh`, which also drops the +variables whose directory is gone. + +--- + +## Services + +SysV, so a service is an init script in `/etc/rc.d/init.d` plus S/K symlinks +in `/etc/rc.d/rc<N>.d`. **Starting is not enabling** — `start` runs the script +now, whatever the links say. + +```sh +packagemanager service # what exists, at boot, running +packagemanager service sshd start # now, enabled or not +packagemanager service sshd enable # at boot, without starting it +packagemanager service sshd disable # boot only; it keeps running +``` + +`disable` records the links it removed, so `enable` puts back exactly what was +there — the priority number lives in the blfs-bootscripts Makefile and is the +one thing that cannot be worked out again. With no record, the runlevels come +from the script's own `Default-Start`. `status` exits 0 when running, 3 when +not, so it works in an `if`. + +The scripts themselves come from blfs-bootscripts, one `make install-<target>` +per service. `packagemanager stack services --only bootscripts --run` installs +the ones whose package is here and whose script is not, from the mapping in +`/etc/pkgusr/stacks/bootscripts.conf`. + --- ## When ownership starts @@ -224,6 +280,13 @@ Everything is dry-run by default. `--run` applies. | `lfs-helper which-package <path>` | Which package installed this | | `lfs-helper manifests` | What each package installed | | `lfs-helper sort-users --run` | Sort passwd and group by id | +| `lfs-helper setid [--run]` | Setuid/setgid bits the build refused (`su`, `passwd`, ...) | +| `lfs-backup create --run` | Back up the files in `/etc/pkgusr/backup.conf` (modes, owners) | +| `lfs-backup restore <archive> [--skip accounts,ssh] [--run]` | Put them on a new system: creates missing files, writes `.new` for the rest | +| `lfs-backup plan <archive> [file]` | A restore plan to edit: skip files, or send them elsewhere | +| `lfs-backup restore <archive> --plan <file> [--run]` | Restore by that plan | +| `lfs-backup merge <archive>` | Go through the `.new` files: diff, keep live, take backup, or edit | +| `lfs-backup clean <archive> [--run]` | Remove those `.new` files after merging (edited ones are kept) | --- @@ -297,6 +360,7 @@ lfs-kernel list # what kernel.org offers lfs-kernel # build the version in /etc/pkgusr/kernel.conf lfs-kernel 6.12.40 # that version, once lfs-kernel verify # is everything installed? +lfs-kernel firmware --missing [--fix] # firmware this machine wants and lacks ``` Everything it does is configured in `/etc/pkgusr/kernel.conf`, which is @@ -324,15 +388,19 @@ enter it. ## Tests ```sh -bash test_lfs_crosschain.sh ./lfs +bash test_lfs_crosschain.sh ./lfs # all of it +bash test_lfs_crosschain.sh --list # numbered blocks +bash test_lfs_crosschain.sh --from 300 --to 340 ./lfs +bash test_lfs_crosschain.sh --only firmware ./lfs ``` Each test encodes a bug that actually happened, with a comment explaining what broke. -Run it from a directory holding all the tools — about 100 tests skip -without them. It no longer fits one sandbox slot; on a real machine run it -whole, or one block at a time (each starts `# ---- <title>`). +Run it from a directory holding all the tools — about 100 tests skip without +them. A selection re-executes a generated subset (preamble + chosen blocks + +summary); a block that needs something an earlier one computed says so +instead of passing quietly. --- diff --git a/backup.conf b/backup.conf new file mode 100644 index 0000000..cfa3e88 --- /dev/null +++ b/backup.conf @@ -0,0 +1,132 @@ +# backup.conf -- read by `lfs-backup`. What to carry to the next system. +# +# One path per line. Globs are fine. A directory takes every file below it. +# A symlink is backed up as the file it points to. +# [user:NAME] ~ is the main user's home (PKGUSR_SYS_MAIN_USER) +# [root:NAME] ~ is root's home +# [system:NAME] absolute paths +# NAME is what you pick with --only / --skip: +# lfs-backup sections the names, with file counts +# lfs-backup restore <archive> --skip accounts,ssh +# +# A line starting with ! leaves out whatever matches it, in every section +# (* crosses /). +# +# lfs-backup list what this resolves to, right now +# lfs-backup create --run the archive (in /root, mode 600) +# lfs-backup restore <archive> the plan on the new system; add --run +# lfs-backup merge <archive> go through the .new files one by one +# lfs-backup clean <archive> remove the .new files once merged +# lfs-backup plan <archive> [file] a restore plan to edit: skip files, or +# give them another target; then +# lfs-backup restore <archive> --plan <file> + +# ---- left out everywhere --------------------------------------------------- +# plugin git histories (ALE's pack alone is 20 MB) and their test trees: the +# plugin runs without them, and `git clone` gets them back +!~/.vim/pack/*/.git/* +!~/.vim/pack/*/test/* +# editor backups and history +!*~ +!~/.vim/.netrwhist + +[user:shell] +~/.bash_profile +~/.bashrc +~/.profile +~/.bash_aliases + +[user:desktop] +~/.config/sway/config +~/.config/foot/foot.ini +~/.config/swas/config +~/.config/swbr/config +~/.config/swov/config +~/.config/mpv/mpv.conf + +# ~/.vim holds the plugins (pack/, autoload/, plugin/ ...): all of it +[user:vim] +~/.vimrc +~/.vim + +[user:git] +~/.gitconfig + +# private keys: secrets, like everything in the archive +[user:ssh] +~/.ssh/config +~/.ssh/authorized_keys +~/.ssh/id_* + +[root:shell] +~/.profile +~/.bash_profile +~/.bashrc +~/.bash_aliases + +[root:vim] +~/.vimrc +~/.vim + +# --------------------------------------------------------------------------- +# BOOT SCRIPTS: NOT ALL OF /etc/rc.d. init.d/* comes from the bootscript +# packages and rc?.d/ are their start links -- a new system installs both, +# and backing them up only yields a .new for every changed upstream script. +# Carry what YOU wrote or edited. +[system:boot] +/etc/fstab +/etc/inittab +/etc/sysconfig/rc.site +/etc/sysconfig/clock +/etc/sysconfig/console +/etc/rc.d/rc.iptables +/etc/rc.d/rc.ip6tables +/boot/efi/EFI/refind/refind.conf +/etc/sysctl.conf +/etc/modprobe.d/*.conf +/etc/cron.weekly/update-pki.sh +# written by `cfg power`; carried so a hand edit is not lost +/etc/rc.d/init.d/power-perms + +[system:network] +/etc/hostname +/etc/hosts +/etc/resolv.conf +/etc/sysconfig/ifconfig.wlp3s0 +/etc/sysconfig/wpa_supplicant/* + +[system:udev] +/etc/udev/rules.d/*.rules + +[system:tools] +/etc/pkgusr/kernel.conf +/etc/pkgusr/backup.conf +/etc/pkgusr/desktop-user +/etc/pkgusr/stacks/machine.conf +/etc/pkgusr/stacks/package-fixes.conf +/etc/pkgusr/stacks/app-users.conf +/etc/pkgusr/packagemanager.conf + +[system:shell] +/etc/profile +/etc/inputrc + +# the global vimrc. /usr/share/vim/vimfiles is NOT taken: what is in it +# belongs to packages (cmake's syntax files were all it held) and comes back +# with them. +[system:vim] +/etc/vimrc + +# ALWAYS restored as .new, to merge by hand. shadow alone is useless +# without the passwd and group lines it belongs to. +[system:accounts] +/etc/passwd +/etc/group +/etc/shadow +/etc/gshadow + +# host keys keep this machine's SSH identity; they are secrets +[system:ssh] +/etc/ssh/sshd_config +/etc/ssh/ssh_host_*_key +/etc/ssh/ssh_host_*_key.pub diff --git a/blfs b/blfs index dbfacf1..88f08d7 100755 --- a/blfs +++ b/blfs @@ -17,6 +17,7 @@ Subcommands blfs books list known/cached books and the default blfs books --discover probe the live site for available versions blfs set-default <selector|url> choose the default book + blfs path [selector] the local file a book reads (for scripts) blfs search <name> find packages, including python modules blfs script <name|url> [-o dir] write install_<name-version> (install/update) blfs deps <name> [--tree] [--no-recommended] [--optional] @@ -128,7 +129,7 @@ CACHE_VERSION = 56 # v56: a skipped build variant does not keep its install st # tool + generated-script format versions. Bump SCRIPT_VERSION when the shape # of generated install scripts changes (phases, dispatcher, ...), and # CACHE_VERSION when the parsed/cached data structure changes. -BLFS_VERSION = "1.14.146" +BLFS_VERSION = "1.14.321" def _build_id(): @@ -838,8 +839,17 @@ def cache_dir(): def _book_stamp(path): + # THE CODE THAT EXTRACTED THE DATA IS PART OF THE KEY. + # + # The key was the book's size and mtime plus CACHE_VERSION, a number to + # bump by hand. Three fixes to the extractors -- 1.14.212, both halves + # of the poppler pair-skip, and the first try at shadow's pam.d blocks -- + # were each "proved" by regenerating a script that came out byte-identical + # and then reverted, because the script was rendered from a cache built + # by the OLD code. The edits never ran. _build_id changes whenever this + # file does, so an edited extractor re-indexes once, by construction. st = os.stat(path) - return f"{st.st_size}:{int(st.st_mtime)}:v{CACHE_VERSION}" + return f"{st.st_size}:{int(st.st_mtime)}:v{CACHE_VERSION}:{_build_id()}" def _cache_file_for(path): @@ -990,6 +1000,30 @@ def build_index(soup): anchor = a.get("id") or a.get("name") if anchor and anchor not in idx: idx[anchor] = header + # THE ID IS NOT ALWAYS INSIDE THE HEADER. + # + # A page whose anchor sits on the section DIV rather than on an <a> inside + # its title header is invisible to the loop above -- it is not "missing + # from the book", it is missing from the index, and the two look identical + # from outside: `packagemanager stack network` said NOT IN BOOK for + # dhcpcd, whose page is right there in basicnet. + # + # Only ids that name a section carrying a Package Information block are + # added, and only when nothing already claims the anchor, so this can add + # pages and can never move an existing one. + for div in soup.select("div.sect1[id], div.sect2[id]"): + anchor = div.get("id") + if not anchor or anchor in idx: + continue + if div.find("div", class_="package", recursive=False) is None: + continue + # the DIV is what gets indexed, not its header: for this shape the + # header sits INSIDE the section, so the "titlepage then sibling + # sect" walk that resolves the common shape finds div.package next + # and gives up. resolve_package takes a section div directly. + if div.find(lambda t: t.name in H_TAGS + and (_header_classes(t) & set(HEADER_CLASSES))) is not None: + idx[anchor] = div return idx @@ -1013,6 +1047,18 @@ def resolve_package(soup, anchor): header = get_index(soup).get(anchor) if header is None: return None + # A SECTION DIV THAT CARRIES THE ID IS ALREADY THE CONTENT. build_index + # adds these (see there); they are whole pages whose anchor is an + # attribute of the section rather than an <a> in its title. + if getattr(header, "name", None) == "div": + h = _content_title_header(header) + title = " ".join(h.get_text().split()) if h is not None else anchor + if header.find("div", class_="package", recursive=False) is None: + return None + if not extract_link(header).strip(): + return None + return {"anchor": anchor, "title": title, "content": header, + "is_module": "sect1" not in _header_classes(header)} classes = _header_classes(header) title = " ".join(header.get_text().split()) @@ -1057,7 +1103,7 @@ def resolve_package(soup, anchor): # This rejects in-package subsection anchors (foo-kernel "Kernel # Configuration", foo-config "Configuring foo", ...) that would otherwise # resolve to their enclosing package's data. - if sect_anchor(content) != anchor: + if sect_anchor(content) != anchor and content.get("id") != anchor: return None return {"anchor": anchor, "title": title, "content": content, "is_module": is_module} @@ -1396,6 +1442,28 @@ def extract_commands(sec, selector): if not elements: return "" out = [] + _emit_command_elements(elements, out, nested=False) + return "\n".join(out) + + +# A SUB-SECTION IS PART OF THE SECTION. +# +# Shadow's configure phase came out as prose and a bare `:` -- no +# /etc/pam.d/login, passwd, chage or the ten admin tools -- so `passwd` said +# `pam_start() failed, error 26` and login `PAM Failure, aborting`. Every one +# of those blocks is on the page, one level down: +# div.configuration > div.sect3 > div.sect4 > pre.root +# and this walk read the children of sect3 and dropped any div that was not a +# note. linux-pam's four system-* files sit directly in its sect3, which is +# why those four existed and shadow's sixteen did not. +# +# So a nested div.sect4/sect5 is walked with the same rules, and its title is +# kept as a comment. The TOP level is unchanged byte for byte: its own +# titlepage divs were never emitted and still are not. +_NESTED_SECT_CLASSES = {"sect4", "sect5"} + + +def _emit_command_elements(elements, out, nested): for el in elements: name = getattr(el, "name", None) if name == "h2": @@ -1419,7 +1487,7 @@ def extract_commands(sec, selector): # A conditional block is conditional wherever it appears. _g = _conditional_guard(el) if _g: - out.append("## OPTIONAL, skipped -- the book says: " + _g) + out.append(SKIP_MARK + _g) out.append(_commented(text)) else: out.append(text) @@ -1442,7 +1510,13 @@ def extract_commands(sec, selector): lines.append(child) out.append("## " + " ".join(lines).replace(" , ", ", ")) out.append("") - return "\n".join(out) + elif _NESTED_SECT_CLASSES & set(classes): + _emit_command_elements(el.find_all(True, recursive=False), + out, nested=True) + elif nested and "titlepage" in classes: + title = " ".join(el.get_text().split()) + if title: + out.append("\n\n## " + title) # The book's conditionals live in PROSE: "If you wish to build and install @@ -1470,8 +1544,53 @@ _COND_GUARD_RE = re.compile(r"^(if\b|optionally\b|should you\b)", re.I) # a failing command was ignored; the moment the dispatcher started honouring # set -e it would fail every such package. Tests belong to `test` # (PM_TEST=1), not to a build. +# ...AND THE BOOK DOES NOT ALWAYS START THE SENTENCE WITH "To test". +# numpy's is +# "The installation can be tested with the following commands:" +# which this matched none of, so a venv, `pip3 install hypothesis attrs` +# and `numpy.test()` were generated into the INSTALL phase. numpy itself +# installed (1320 files) and then the package failed on +# ModuleNotFoundError: No module named 'pytest' +# -- a test dependency, in an install. So these are searched for anywhere +# in the introducing sentence, not just at its start. _TEST_BLOCK_RE = re.compile( - r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)", re.I) + r"^(to test\b|to check\b|the test suite\b|tests? (can|may) be run\b)" + r"|\bcan be tested\b|\bto test the (results|installation|build)\b" + r"|\bto run the test", re.I) + +# THE COMMANDS SAY WHAT THEY ARE, AND PROSE IS A SECOND-HAND ACCOUNT. +# +# 1.14.211 widened the sentence match so numpy's "The installation can be +# tested with the following commands" would be recognised -- and the +# regenerated script STILL had the venv, `pip3 install hypothesis attrs` +# and `numpy.test()` in its install phase, because the book being read on +# that machine is 13.0 and words or nests it differently again. +# +# Chasing the prose is a losing game: `numpy.test()` IS a test suite +# whatever sentence introduces it. These patterns match the block's own +# commands, so a rewording upstream cannot bring the bug back. +_TEST_CONTENT_RE = re.compile( + r"""(?mx) + ^\s*(make|ninja)\s+(-\w+\s+)*(check|test)\b # make check / ninja test + | ^\s*meson\s+test\b + | ^\s*ctest\b + | ^\s*py\.test\b | ^\s*pytest\b + | ^\s*python3?\s+-m\s+(pytest|unittest)\b + | \.test\(\) # numpy.test() + | ^\s*python3?\s+-m\s+venv\b # a throwaway test venv + | ^\s*source\s+\S*testenv/bin/activate + """) + + +def _test_content_reason(pre): + """Why this block is a test suite, judged by its commands, or None.""" + txt = pre.get_text() + m = _TEST_CONTENT_RE.search(txt) + if not m: + return None + hit = " ".join(m.group(0).split()) + return "it IS the test suite -- these commands run it (`%s`)" % hit + def _conditional_guard(pre): """The conditional sentence introducing this <pre>, or None if the block @@ -1489,12 +1608,16 @@ def _conditional_guard(pre): Mixing local libtiff with shared libjpeg not supported "optional" wins over both: an optional tool is exactly what we do not install.""" + # content first: it does not depend on the book's wording or nesting + _by_content = _test_content_reason(pre) + if _by_content: + return _by_content prev = pre.find_previous(["p", "pre", "h3", "h4"]) if prev is None or prev.name != "p": return None txt = " ".join(prev.get_text().split()) - if _TEST_BLOCK_RE.match(txt): - return txt # a test, never part of installing + if _TEST_BLOCK_RE.search(txt): + return "the book says: " + txt # a test, never part of installing if not _COND_GUARD_RE.match(txt): return None low = txt.lower() @@ -1506,7 +1629,7 @@ def _conditional_guard(pre): return None if "optional" not in low and ("recommended" in low or "required" in low): return None # our contract satisfies it - return txt + return "the book says: " + txt def _commented(text): @@ -1585,7 +1708,7 @@ def extract_commands_phased(sec): is_root = "root" in classes and not guard stream = install if (is_root or install) else build if guard: - stream.append("## OPTIONAL, skipped -- the book says: " + guard) + stream.append(SKIP_MARK + guard) stream.append(_commented(text)) continue stream.append(text) @@ -2500,6 +2623,61 @@ _CONFIG_TEST_INTRO_RE = re.compile( r"|test your installation", re.I) +# A SITE CONFIGURATION FILE IS NOT AN INSTALL STEP. +# +# The dhcpcd page ends with an example interface file: +# cat > /etc/sysconfig/ifconfig.eth0 << "EOF" +# ONBOOT="yes" +# IFACE="eth0" +# SERVICE="dhcpcd" +# ... +# Run as a configure phase by p_dhcpcd it died with +# /etc/sysconfig/ifconfig.eth0: Permission denied +# -- correctly: /etc/sysconfig is drwxrwxr-t root:install and that file is +# root's. Granting the package user a collector group for /etc/sysconfig +# would "fix" it by letting a package write the machine's network +# configuration, which is worse than the failure. +# +# And the content is wrong anyway: `eth0` is the book's placeholder, and this +# machine's interfaces are `enp5s0` and `wlp3s0`. A file naming an interface +# that does not exist does nothing, or shadows the one you meant. +_SITE_CONFIG_RE = re.compile( + r"/etc/sysconfig/(ifconfig\.|network-devices|wpa_supplicant)" + r"|/etc/wpa_supplicant" + r"|/etc/sysconfig/console|/etc/sysconfig/clock", re.I) + + +def _comment_site_config_writes(cmds, name=""): + """Comment out writes to files that hold THIS machine's settings.""" + out, skipping, heredoc = [], False, None + for line in (cmds or "").splitlines(): + st = line.strip() + # PROSE IS NOT CODE: a comment mentioning the path is not a write. + if st.startswith("#") and not heredoc: + out.append(line) + continue + if heredoc is not None: + out.append("# " + line if skipping else line) + if st == heredoc or st == heredoc.strip('"\''): + heredoc, skipping = None, False + continue + if _SITE_CONFIG_RE.search(st) and re.search(r">|tee |install |cp |mv ", st): + out.append("## SKIPPED -- this writes a file that belongs to the" + " MACHINE, not to the package:") + out.append("## %s" % st[:120]) + out.append("## The interface name in the book is a placeholder, and" + " /etc/sysconfig is root's.") + out.append("## Write yours by hand; what is missing is reported by:") + out.append("## packagemanager stack network --only net-wifi --run") + out.append("# " + line) + m = re.search(r"<<-?\s*([\"']?)([A-Za-z_][A-Za-z0-9_]*)\1", st) + if m: + heredoc, skipping = m.group(2), True + continue + out.append(line) + return "\n".join(out) + + def _comment_config_tests(cmds): out, skipping = [], False for line in (cmds or "").splitlines(): @@ -3040,6 +3218,21 @@ def _drop_orphaned_variant_install(cmds): return "\n".join(out) +# ONE MARKER, ONE CONSTANT. `_enable_one_alternative` looked for +# "## OPTIONAL, skipped" and 1.14.212 renamed the marker to "## SKIPPED --" +# -- so the picker silently stopped firing, and webkitgtk's script came out +# with BOTH build blocks commented: a build phase of `:` and an install +# phase that runs `ninja install` against a directory with no build.ninja. +# Two strings for one fact, drifting apart, exactly as the rule says. +SKIP_MARK = "## SKIPPED -- " + + +def _is_skip_line(line): + """A skipped-block header, under either spelling this tool has used.""" + st = line.strip() + return st.startswith(SKIP_MARK.strip()) or st.startswith("## OPTIONAL, skipped") + + def _enable_one_alternative(cmds): """A page that offers a CHOICE must still build something. @@ -3058,8 +3251,7 @@ def _enable_one_alternative(cmds): if any(l.strip() and not l.strip().startswith("#") and l.strip() != ":" for l in lines): return cmds - starts = [i for i, l in enumerate(lines) - if l.strip().startswith("## OPTIONAL, skipped")] + starts = [i for i, l in enumerate(lines) if _is_skip_line(l)] if not starts: return cmds i = starts[0] @@ -3067,8 +3259,11 @@ def _enable_one_alternative(cmds): out = lines[:i] out.append("## Every build block on this page is conditional, so none " "would have run.") - out.append("## ENABLED HERE: %s" - % lines[i].split("the book says:", 1)[-1].strip()[:110]) + _why = lines[i] + for _sep in ("the book says:", SKIP_MARK.strip()): + if _sep in _why: + _why = _why.split(_sep, 1)[-1] + out.append("## ENABLED HERE: %s" % _why.strip()[:110]) out.append("## If you want one of the page's other variants instead, " "comment these") out.append("## lines and uncomment that block.") @@ -3131,6 +3326,7 @@ def render_script(data, source_url, book_version=""): config_cmds = _comment_profile_sourcing(data.get("config_cmds") or "") config_cmds = _comment_bootscript_installs(config_cmds, data.get("name")) config_cmds = _comment_config_tests(config_cmds) + config_cmds = _comment_site_config_writes(config_cmds, data.get("name")) data["config_cmds"] = config_cmds build_cmds = _comment_profile_sourcing(build_cmds) install_root = _comment_profile_sourcing(install_root) @@ -3355,6 +3551,29 @@ def cmd_fetch(args): print("cached: %s" % path) +def cmd_path(args): + """Print the local book file a selector reads -- for other tools. + + packagemanager asks this once per run for every book it needs (the + default, a stack's `from-book svn`). Resolving `svn` through + resolve_book asks the website every time, and a planner that passes + `--book svn` to each of its hundred blfs calls would ask a hundred + times. A cached book is answered from disk; only a book that was never + fetched goes to the network (or fails, with --offline).""" + sel = args.selector or getattr(args, "book", None) or default_selector() + if os.path.isfile(sel): + print(os.path.abspath(sel)) + return + p = _cached_for_selector(sel) + if not (p and os.path.isfile(p)): + if getattr(args, "offline", False): + sys.stderr.write("no cached book for '%s' (blfs fetch %s)\n" + % (sel, sel)) + sys.exit(1) + p = ensure_book(sel, quiet=True) + print(p) + + def cmd_import(args): """Add a local book file to the cache -- no network involved.""" src = os.path.expanduser(args.file) @@ -3816,18 +4035,69 @@ def cmd_debug(args): anchor = anchor_from_input(args.target) print(f"input : {args.target}") print(f"anchor : {anchor}") + # WHICH BOOK AM I EVEN READING. The first question when a page that + # plainly exists upstream is "not in the book" is whether this is the + # book you think it is, and how much of it parsed. + try: + _st = os.stat(book.path) + print("book : %s" % book.path) + print(" %.1f MB, modified %s" + % (_st.st_size / 1048576.0, + time.strftime("%Y-%m-%d %H:%M", time.localtime(_st.st_mtime)))) + except OSError: + print("book : %s (cannot stat it)" % book.path) + print("pages parsed : %d" % len(book.packages)) anchor = resolve_anchor(book, anchor) data = book.get(anchor) if data is None: - # not a package in the cache -- parse the soup to explain why + # NAME THE TEST THAT FAILED. "not an installable package (no package + # block + download link)" covered three different causes with three + # different fixes, and told you which of them applied: none. soup = book.soup + in_index = anchor in get_index(soup) node = soup.find(attrs={"id": anchor}) or soup.find("a", attrs={"name": anchor}) + print("in page index : %s" % ("yes" if in_index else "no")) if node is None: - print("resolve : NOT FOUND (no element with this id/name)") + print("resolve : NOT FOUND -- no element in this book carries " + "that id or name") + near = sorted({(e.get("id") or e.get("name")) + for e in soup.find_all(attrs={"id": True}) + if anchor in (e.get("id") or "")} + | {a.get("name") for a in soup.find_all("a", attrs={"name": True}) + if anchor in (a.get("name") or "")}) + near = [n for n in near if n and n != anchor][:8] + if near: + print("similar ids : %s" % ", ".join(near)) + print("hint : the page is there under another anchor; " + "use one of those") + elif re.search(re.escape(anchor), soup.get_text(" ", strip=True)[:4000000], + re.IGNORECASE): + print("hint : the NAME appears in the text but no anchor " + "matches -- likely a different book edition") + else: + print("hint : this book really does not have it. A tar or " + "git entry with build=/install= is the way in.") else: - print(f"resolve : found <{node.name} id={node.get('id')}> but it is " - f"not an installable package (no package block + download link)") + print("resolve : found <%s id=%s>, and it is not accepted as a " + "package. Which test fails:" % (node.name, node.get("id"))) + content = node if node.name == "div" else node.find_parent( + "div", class_=["sect1", "sect2", "sect3", "sect4", "sect5"]) + if content is None: + print(" content : no enclosing section div") + else: + pkg_div = content.find("div", class_="package", recursive=False) + print(" package block: %s" % ("present" if pkg_div is not None + else "MISSING (a direct child " + "<div class=package> is required)")) + link = extract_link(content).strip() if pkg_div is not None else "" + if pkg_div is not None: + print(" download link: %s" % (link or "MISSING (no upstream tarball url)")) + sa = sect_anchor(content) + print(" titles it : %s" % ("yes" if (sa == anchor + or content.get("id") == anchor) + else "no -- %r titles that section, so " + "this is a subsection anchor" % sa)) print("hint : blfs search " + anchor) return @@ -3924,6 +4194,13 @@ def main(): p.add_argument("selector", nargs="?", help="e.g. stable-systemd, 12.4, svn") p.set_defaults(func=cmd_fetch) + p = sub.add_parser("path", help="print the local file a book selector " + "reads (cached: no network)") + p.add_argument("selector", nargs="?", help="e.g. svn; default: the default") + p.add_argument("--offline", action="store_true", + help="fail instead of fetching a book that is not cached") + p.set_defaults(func=cmd_path) + p = sub.add_parser("import", help="add a local book file (no network)") p.add_argument("file") p.add_argument("--selector", help="name to file it under") diff --git a/kernel.conf b/kernel.conf index f9074e2..e779b54 100644 --- a/kernel.conf +++ b/kernel.conf @@ -54,6 +54,23 @@ CONFIG_FROM=running # Either way the tool reports how many lines changed. CONFIG_REFRESH=olddefconfig +# CONFIG_SET: options THIS MACHINE always wants, applied on top of whatever +# CONFIG_FROM gave, after the refresh -- so `running` + a new kernel version +# cannot quietly lose them. SYMBOL=VALUE, space separated, CONFIG_ optional: +# y / m / n enable, module, disable +# "text" a string option +# 123 a number +# Each is read back from the final .config; one the kernel refuses (a missing +# dependency, a symbol this version does not have) is named and stops the build. +# +# LEGACY_TIOCSTI=n no program may type into a terminal it does not own. +# With it allowed, shadow's su (4.20+) detaches the shell +# of every `su - <user>` run by root -- "cannot set +# terminal process group ... no job control" -- because +# that user could otherwise inject commands into root's +# terminal. Off, su keeps the terminal and job control. +CONFIG_SET="LEGACY_TIOCSTI=n" + # ---- firmware ------------------------------------------------------------ # # Most drivers need firmware blobs: amdgpu will not bring up a display @@ -86,12 +103,33 @@ FIRMWARE_GIT=https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firm # older releases kept iwlwifi-*.ucode at the top level, # which is what a FILE pattern like iwlwifi-cc-* is for. # rtl_bt Realtek Bluetooth +# iwlwifi-*.ucode THE INTEL WIRELESS BLOBS THEMSELVES. The driver asks +# the loader for a FLAT path -- iwlwifi-cc-a0-77.ucode, +# not intel/iwlwifi/... -- so `intel` above does not +# satisfy it, and the AX200 comes up with +# Direct firmware load for iwlwifi-cc-a0-77.ucode failed +# no suitable firmware found! +# and no wlan interface at all. Since 1.14.279 the +# pattern matches at any depth, and the flat names are +# made as links from linux-firmware's WHENCE -- so +# `intel` alone gets them too; the pattern does no harm. +# NOT here: regulatory.db. cfg80211 asks for it, but linux-firmware does +# not carry it -- it is released on its own as wireless-regdb -- and naming +# it here fails the whole firmware install. REGDB below fetches it. # Find what your machine asks for, once it is running, with: +# lfs-kernel firmware --missing # and --fix to install them # dmesg | grep -i firmware # (quoted: the value has spaces and a wildcard, and the config is sourced by # the shell -- unquoted, `intel` was run as a command and the whole # assignment came out empty, which meant "install everything") -FIRMWARE_ONLY="amdgpu amd-ucode amd intel" +FIRMWARE_ONLY="amdgpu amd-ucode amd intel iwlwifi-*.ucode" + +# REGDB: cfg80211's regulatory database, which is wireless-regdb rather than +# part of linux-firmware. yes installs the newest release from kernel.org +# into /lib/firmware, owned by the firmware package like everything else +# there. Without it: "failed to load regulatory.db" at boot, and the most +# conservative channels and transmit power the driver has. +REGDB=yes # ---- headers ------------------------------------------------------------- # diff --git a/lfs b/lfs index cd534f8..00d6793 100755 --- a/lfs +++ b/lfs @@ -60,7 +60,7 @@ except ImportError: " packagemanager pip install beautifulsoup4\n") sys.exit(2) -LFS_VERSION = "1.14.146" +LFS_VERSION = "1.14.321" def _build_id(): @@ -235,11 +235,11 @@ LAYOUT_DEFAULTS = { # the system being built "target_pkgusr_home": "/usr/src/pkgusr", "target_cfguser_home": "/usr/src/cfg", - "target_appuser_home": "/usr/src", # or /home/shared_users + "target_appuser_home": "/home/shared_user", # one subdir, not /home itself # this machine "pkgusr_home": "/usr/src/pkgusr", "cfguser_home": "/usr/src/cfg", - "appuser_home": "/usr/src", + "appuser_home": "/home/shared_user", } @@ -266,7 +266,10 @@ def _accounts_root(kind): target=True).rstrip("/") _c = {} try: - _c = read_config() or {} + # load_config, the function that exists: `read_config` did not, and + # the NameError was swallowed below -- a configured pkgusr_subdir / + # cfguser_subdir was silently ignored in favour of the default + _c = load_config() or {} except Exception: _c = {} sub = str(_c.get("pkgusr_subdir" if kind == "pkgusr" else "cfguser_subdir") @@ -1477,7 +1480,7 @@ _SESSION_KEYS = [ "/usr/src"), ("target_appuser_home", "where application users live on the BUILT system", - "/usr/src"), + "/home/shared_user"), ("locale", "system locale (see 'locale -a')", "en_US.UTF-8"), ("paper_size", "paper size for groff: A4 or letter", "A4"), # Everything below is a decision the build used to discover halfway @@ -3296,6 +3299,38 @@ def _handover_build_user_only(lfs, run=True): uid = _build_user_uid() if uid is None: return 0 + # THE HOST'S lfs UID CAN BE A PACKAGE USER'S UID IN THE TREE. + # + # This selects files by NUMBER, because that is the only thing that means + # the same on both sides of the chroot. Book 4.3 pins no uid for `lfs`, + # so the host's is whatever was free -- 10753 on one machine -- and + # package users here are created in the 10000-20000 range. If those + # numbers ever meet, this hands a package user's files to root and the + # package silently loses every one of them. + # + # Ask the tree's own passwd before touching anything. + _clash = [] + _pw = os.path.join(lfs, "etc", "passwd") + try: + with open(_pw, encoding="utf-8", errors="replace") as f: + for line in f: + parts = line.split(":") + if len(parts) > 2 and parts[2].strip() == str(uid): + _clash.append(parts[0]) + except OSError: + pass + if _clash: + warn("REFUSING to hand files back: the host's build user is uid %d," + % uid) + warn("and that is also %s in %s." % (", ".join(_clash), _pw)) + note(" Selecting by uid cannot tell them apart, so this would give") + note(" that account's files to root and the package would lose them.") + note(" See what it owns:") + note(" find %s -xdev -uid %d | head" % (lfs, uid)) + note(" Then renumber the HOST's lfs account -- it exists only to") + note(" build, so nothing of yours depends on its number:") + note(" usermod -u <free-uid> lfs # on the host, then retry") + return 0 dirs = [os.path.join(lfs, d) for d in LFS_HANDOVER_DIRS if os.path.exists(os.path.join(lfs, d))] if not dirs: @@ -5689,7 +5724,12 @@ def cmd_bs_gen_chroot_scripts(args): for nm, left in unresolved: sys.stderr.write(" %-18s %s\n" % (nm, ", ".join("<%s>" % x for x in left))) - print(f"# step order -> {pkgusr_state(lfs, "progress", "steporder")}") + # SINGLE QUOTES INSIDE THE BRACES. Reusing the f-string's own quote + # character is legal only from Python 3.12, and `lfs` is the one tool that + # runs on the HOST -- where Python is whatever the distribution ships + # (Debian 12: 3.11). There it did not fail at this line, it failed to + # start at all, with a SyntaxError pointing at a debug print. + print(f"# step order -> {pkgusr_state(lfs, 'progress', 'steporder')}") write_chroot_env(lfs) install_helper(lfs) _install_collector_groups(lfs) diff --git a/lfs-backup b/lfs-backup new file mode 100755 index 0000000..5d3e7f8 --- /dev/null +++ b/lfs-backup @@ -0,0 +1,694 @@ +#!/bin/bash +# lfs-backup -- back up the configuration files you changed, and put them on +# another system without overwriting anything there. +# +# lfs-backup list what the config names, and what exists +# lfs-backup create [--run] [ARCHIVE] write the archive (root; secrets inside) +# lfs-backup show ARCHIVE what an archive holds +# lfs-backup restore ARCHIVE [--run] put it on THIS system: +# missing file -> created +# same content -> left alone +# different -> <file>.new, listed +# lfs-backup clean ARCHIVE [--run] remove the .new files restore left, +# unless you edited one (kept, listed) +# lfs-backup merge ARCHIVE go through those .new files one by +# one: diff, then keep live / take new / +# edit both / skip +# lfs-backup sections [ARCHIVE] the section names to pick from +# lfs-backup plan ARCHIVE [FILE] write a restore plan to edit: one +# line per file, restore or skip, and +# where it goes +# lfs-backup restore ARCHIVE --plan FILE [--run] restore by that plan +# +# --only a,b restore/create/clean just these sections +# --skip a,b everything except these +# +# Without --run, create, restore and clean only print what they would do. +# +# THE CONFIG: /etc/pkgusr/backup.conf (LFS_BACKUP_CONF overrides). Sections, +# one path per line, shell globs allowed: +# [user] or [user:NAME] ~ is the main user's home (PKGUSR_SYS_MAIN_USER) +# [root] or [root:NAME] ~ is root's home +# [system] or [system:NAME] absolute paths +# NAME is what --only/--skip select ("accounts", "ssh", "vim" ...); a section +# without one is named after its kind. A directory entry takes every file +# below it. A line starting with ! excludes what matches it, anywhere in the +# config (* crosses /): !~/.vim/pack/*/.git/* !*~ +# +# WHAT IS STORED IS THE FILE, NOT THE LINK. A path that is a symlink is +# backed up as the file it points to, under the path you named. +# +# OWNERSHIP GOES BY NAME, NOT NUMBER. Ids differ between two systems; the +# archive records owner and group names and the mode. The old main user's +# name becomes this system's main user on restore; a name this system does +# not have becomes root, and is reported. +# +# NOTHING IS REPLACED. An existing file with different content gets a .new +# beside it, with the backed-up mode and owner, and the end of the run lists +# every one of them to merge by hand. /etc/passwd, group, shadow and gshadow +# ALWAYS go to .new, even when the file is missing -- a restored account +# database that does not match this system locks you out. +# +# THE ARCHIVE HOLDS SECRETS (shadow, wpa_supplicant keys): created 0600, root. +set -u + +VERSION="1.3.0" +PROG="lfs-backup" +CONF="${LFS_BACKUP_CONF:-/etc/pkgusr/backup.conf}" +# ROOT: the tree to read from (create) and write to (restore). / on a +# running system; a mounted new system's root when restoring into it. +ROOT="${LFS_BACKUP_ROOT:-/}" +ROOT="${ROOT%/}" +RUN=0 +PLAN="" +ONLY="" +SKIP="" +TMPD="" +trap '[ -n "$TMPD" ] && rm -rf "$TMPD"' EXIT + +# SECRETS ARE RESTORED FOR THEIR OWNER ONLY, whatever mode they had. The +# first real backup recorded /etc/gshadow 644 and most Wi-Fi keys 664; a +# restore that copies those modes spreads the mistake to the new system. +# Group and other bits are dropped (a 664 key becomes 600). +SECRET_RE='^/etc/(shadow|gshadow)$|^/etc/sysconfig/wpa_supplicant/|^/etc/ssh/ssh_host_.*_key$|^~/\.ssh/id_[^./]*$|^~/\.ssh/authorized_keys$' + +# always .new: account databases are merged, never dropped in +ALWAYS_NEW='^/etc/(passwd|group|shadow|gshadow)$' + +say() { printf '%s\n' "$*"; } +note() { printf '# %s\n' "$*"; } +warn() { printf '!! %s\n' "$*" >&2; } +die() { warn "$*"; exit 1; } + +usage() { sed -n '2,/^set -u/p' "$0" | sed '/^set -u/d; s/^# \{0,1\}//'; } + +need_root() { [ "$(id -u)" = 0 ] || die "$PROG $1 needs root (root's files, /etc/shadow)"; } + +# THE MAIN USER, FROM WHAT THE SYSTEM ALREADY SAYS. packagemanager writes +# PKGUSR_SYS_MAIN_USER to /etc/profile.d/pkgusr-system.sh; a root shell that +# did not source it still gets the answer from the file itself. +main_user() { + local u="${PKGUSR_SYS_MAIN_USER:-}" + if [ -z "$u" ] && [ -r "$ROOT/etc/profile.d/pkgusr-system.sh" ]; then + u="$(sed -n "s/^export PKGUSR_SYS_MAIN_USER=['\"]\{0,1\}\([^'\"]*\).*/\1/p" \ + "$ROOT/etc/profile.d/pkgusr-system.sh" | head -n1)" + fi + printf '%s' "$u" +} +home_of() { # home_of <user> -- from THIS tree's passwd + awk -F: -v u="$1" '$1 == u { print $6; exit }' "$ROOT/etc/passwd" 2>/dev/null +} +# NAMES AND IDS FROM THE TREE'S OWN FILES. With LFS_BACKUP_ROOT pointing at a +# mounted system, the host's accounts are the wrong ones to ask. +_name_in() { # _name_in <file> <id> -- the name, or the id itself when unknown + if [ -r "$1" ]; then + awk -F: -v i="$2" '$3 == i { print $1; f = 1; exit } END { if (!f) print i }' "$1" + else + printf '%s\n' "$2" + fi +} +uname_of() { _name_in "$ROOT/etc/passwd" "$1"; } +gname_of() { _name_in "$ROOT/etc/group" "$1"; } +uid_of() { awk -F: -v u="$1" '$1 == u { print $3; exit }' "$ROOT/etc/passwd" 2>/dev/null; } +gid_of() { awk -F: -v u="$1" '$1 == u { print $3; exit }' "$ROOT/etc/group" 2>/dev/null; } + +# SELECTION. A section matches --only/--skip by its name or its kind, so +# `--skip accounts` and `--only user` both do what they read like. +selected() { # selected <scope> <label> + local s="$1" l="$2" x + if [ -n "$ONLY" ]; then + local hit=0 + for x in ${ONLY//,/ }; do [ "$x" = "$l" ] || [ "$x" = "$s" ] && hit=1; done + [ "$hit" = 1 ] || return 1 + fi + for x in ${SKIP//,/ }; do [ "$x" = "$l" ] || [ "$x" = "$s" ] && return 1; done + return 0 +} + +# ---- config -> one resolved file per line: scope<TAB>name<TAB>source<TAB>label +# name what is recorded: ~/rel for user/root, the absolute path for system +# source where it is read on this system +resolve() { + [ -r "$CONF" ] || die "no config: $CONF" + local mu uhome rhome scope="" label="" line pat base rel f + # EXCLUDES FIRST, from the whole file: they apply to every section + EXCLUDES=() + while IFS= read -r line || [ -n "$line" ]; do + line="$(printf '%s' "${line%%#*}" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')" + case "$line" in '!'?*) EXCLUDES+=("${line#!}") ;; esac + done < "$CONF" + mu="$(main_user)" + uhome="$([ -n "$mu" ] && home_of "$mu")" + rhome="$(home_of root)"; rhome="${rhome:-/root}" + while IFS= read -r line || [ -n "$line" ]; do + line="${line%%#*}" + line="$(printf '%s' "$line" | sed 's/^[[:space:]]*//; s/[[:space:],]*$//')" + [ -n "$line" ] || continue + case "$line" in '!'*) continue ;; esac + case "$line" in + \[*\]) + local sec="${line#[}"; sec="${sec%]}" + scope="${sec%%:*}"; label="${sec#*:}" + [ "$label" = "$sec" ] && label="$scope" + case "$scope" in + user|root|system) ;; + *) warn "$CONF: unknown section $line (user, root or system)"; scope="" ;; + esac + continue ;; + esac + [ -n "$scope" ] || { warn "$CONF: '$line' is outside a section"; continue; } + [ "${ALL_SECTIONS:-0}" = 1 ] || selected "$scope" "$label" || continue + case "$scope" in + user) + [ -n "$uhome" ] || { warn "no main user (PKGUSR_SYS_MAIN_USER) -- [user] '$line' skipped"; continue; } + base="$uhome" ;; + root) base="$rhome" ;; + system) base="" ;; + esac + if [ "$scope" = system ]; then + case "$line" in /*) ;; *) warn "[system] '$line' is not absolute -- skipped"; continue ;; esac + pat="$ROOT$line" + else + case "$line" in "~/"*) rel="${line#\~/}" ;; *) warn "[$scope] '$line' does not start with ~/ -- skipped"; continue ;; esac + pat="$ROOT$base/$rel" + fi + local matched=0 + # the glob is expanded here, unquoted on purpose; no match stays literal + for f in $pat; do + if [ -d "$f" ]; then + while IFS= read -r g; do + matched=1 + emit "$scope" "$base" "$g" "$label" + done < <(find -L "$f" -type f 2>/dev/null | sort) + elif [ -e "$f" ]; then + matched=1 + emit "$scope" "$base" "$f" "$label" + fi + done + [ "$matched" = 1 ] || printf 'missing\t%s\t%s\t%s\n' "$scope" "$line" "$label" + done < "$CONF" +} +emit() { # emit <scope> <home> <path-in-ROOT> <label> + local scope="$1" base="$2" src="$3" name="${3#$ROOT}" label="$4" + case "$name" in *$'\t'*|*$'\n'*|*' | '*) warn "a tab, newline or ' | ' in '$name' -- skipped"; return ;; esac + [ "$scope" = system ] || name="~/${name#$base/}" + local x + for x in "${EXCLUDES[@]+"${EXCLUDES[@]}"}"; do + # shellcheck disable=SC2053 -- a pattern, on purpose + [[ "$name" == $x ]] && return + done + printf '%s\t%s\t%s\t%s\n' "$scope" "$name" "$src" "$label" +} + +# ---- list --------------------------------------------------------------- +do_list() { + local scope name src label n=0 m=0 + while IFS=$'\t' read -r scope name src label; do + if [ "$scope" = missing ]; then + printf ' %-9s %-48s (not there)\n' "$label" "$src"; m=$((m + 1)); continue + fi + n=$((n + 1)) + if [ -L "$src" ]; then + printf ' %-9s %-48s -> %s\n' "$label" "$name" "$(readlink -f "$src")" + else + printf ' %-9s %s\n' "$label" "$name" + fi + done < <(resolve) + say "" + note "$n file(s) to back up, $m entr(y/ies) with nothing there (config: $CONF)" +} + +# ---- create ------------------------------------------------------------- +do_create() { + need_root create + local out="${1:-}" host stage scope name src mode owner group n=0 m=0 link label + host="$(cat "$ROOT/etc/hostname" 2>/dev/null || hostname 2>/dev/null || echo lfs)" + [ -n "$out" ] || out="/root/lfs-backup-${host}-$(date +%Y%m%d-%H%M%S).tar.gz" + stage="$(mktemp -d)" || die "no temporary directory" + TMPD="$stage"; chmod 700 "$stage" + mkdir -p "$stage/files" + { + printf '# lfs-backup %s manifest\n' "$VERSION" + printf '# host=%s main_user=%s created=%s\n' "$host" "$(main_user)" "$(date -Iseconds)" + printf '# scope\tname\tmode\towner\tgroup\twas_link_to\tsection\n' + } > "$stage/manifest" + while IFS=$'\t' read -r scope name src label; do + if [ "$scope" = missing ]; then + note "not there: [$label] $src"; m=$((m + 1)); continue + fi + # the FILE: stat -L and cp -L read through a symlink + mode="$(stat -L -c %a "$src")" && owner="$(uname_of "$(stat -L -c %u "$src")")" \ + && group="$(gname_of "$(stat -L -c %g "$src")")" || { warn "cannot stat $src"; continue; } + link="-"; [ -L "$src" ] && link="$(readlink -f "$src")" + say " $label $name$([ "$link" != - ] && printf ' (link -> %s: the file is stored)' "$link")" + [ "$RUN" = 1 ] || { n=$((n + 1)); continue; } + mkdir -p "$stage/files/$scope/$(dirname "${name#\~/}")" + cp -L --preserve=mode,timestamps "$src" "$stage/files/$scope/${name#\~/}" \ + || { warn "could not copy $src"; continue; } + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$scope" "$name" "$mode" "$owner" "$group" "$link" "$label" >> "$stage/manifest" + n=$((n + 1)) + done < <(resolve) + say "" + if [ "$RUN" != 1 ]; then + note "dry run: $n file(s) would go into $out ($m entr(y/ies) with nothing there)" + note " write it: $PROG create --run${1:+ $1}" + return 0 + fi + ( umask 077; tar -C "$stage" -czf "$out" manifest files ) || die "could not write $out" + chown root:root "$out" && chmod 600 "$out" + note "wrote $out ($n file(s), mode 600: it holds secrets)" + [ "$m" -gt 0 ] && note "$m config entr(y/ies) had nothing to back up (listed above)" + return 0 +} + +# ---- show --------------------------------------------------------------- +do_show() { + local a="${1:-}" + [ -r "$a" ] || die "no archive: ${a:-(none given)}" + tar -xzOf "$a" manifest | awk -F'\t' ' + /^#/ { if (NR < 3) print; next } + { printf " %-9s %-48s %4s %s:%s%s\n", ($7 != "" ? $7 : $1), $2, $3, $4, $5, ($6 != "-" ? " (was a link)" : "") }' +} + +# ---- restore ------------------------------------------------------------ +map_owner() { # map_owner <name> <old-main> <new-main> -> a name this system has + local o="$1" + [ -n "$2" ] && [ "$o" = "$2" ] && o="$3" + if awk -F: -v u="$o" '$1 == u { f = 1 } END { exit !f }' "$ROOT/etc/passwd" 2>/dev/null; then + printf '%s' "$o" + else + printf 'root' + fi +} +map_group() { + local g="$1" + [ -n "$2" ] && [ "$g" = "$2" ] && g="$3" + if awk -F: -v u="$g" '$1 == u { f = 1 } END { exit !f }' "$ROOT/etc/group" 2>/dev/null; then + printf '%s' "$g" + else + printf 'root' + fi +} + +# WHO OWNS A RESTORED FILE. +# +# By kind first, names second. The host's root dotfiles were links into the +# main user's old config repo, so they were recorded as n76310's -- restored +# by name that is a /root/.bashrc the main user can edit, which root runs. +# [root] root, always +# [user] the main user; the group too, unless the file had another +# group that exists here +# [system] the recorded names where they exist here, else root -- but +# never the main user (an /etc file owned by a login is a mistake) +owner_for() { # owner_for <scope> <owner> <group> -> "owner group" + local scope="$1" o="$2" g="$3" + case "$scope" in + root) printf 'root root' ;; + user) + local ng="$new_main" + if [ "$g" != "$old_main" ] && [ "$g" != "$o" ] && [ "$g" != root ] \ + && awk -F: -v u="$g" '$1 == u { f = 1 } END { exit !f }' "$ROOT/etc/group" 2>/dev/null; then + ng="$g" + fi + printf '%s %s' "$new_main" "$ng" ;; + *) + [ "$o" = "$old_main" ] && o=root + [ "$g" = "$old_main" ] && g=root + printf '%s %s' "$(map_owner "$o" "" "")" "$(map_group "$g" "" "")" ;; + esac +} +target_for() { # target_for <scope> <name-or-planned-path> -> path in ROOT, or empty + local scope="$1" n="$2" + case "$n" in + "~/"*) + case "$scope" in + root) printf '%s' "$ROOT$rhome/${n#\~/}" ;; + *) [ -n "$uhome" ] && printf '%s' "$ROOT$uhome/${n#\~/}" ;; + esac ;; + /*) printf '%s' "$ROOT$n" ;; + esac +} + +# ---- plan ----------------------------------------------------------------- +# ONE LINE PER FILE, TO EDIT. The plan is how a restore is chosen file by +# file: change `restore` to `skip`, or change the target -- `~/` is the home +# of the section's kind on THIS system, anything else is an absolute path. +# action | state | section | name in the archive | target +do_plan() { + local a="${1:-}" out="${2:-lfs-backup-restore.plan}" tmp scope name mode owner group link label t state act + [ -r "$a" ] || die "no archive: ${a:-(none given)}" + [ -e "$out" ] && die "$out exists -- edit it, or remove it for a fresh plan" + tmp="$(mktemp -d)" || die "no temporary directory" + TMPD="$tmp"; chmod 700 "$tmp" + tar -C "$tmp" -xzf "$a" manifest files || die "cannot read $a" + old_main="$(sed -n 's/^# host=.* main_user=\([^ ]*\) .*/\1/p' "$tmp/manifest" | head -n1)" + new_main="$(main_user)" + uhome="$([ -n "$new_main" ] && home_of "$new_main")" + rhome="$(home_of root)"; rhome="${rhome:-/root}" + local n=0 k=0 + { + printf '# lfs-backup restore plan for %s\n' "$a" + printf '# written %s on %s -- archive main user %s, this system %s\n' \ + "$(date -Iseconds)" "$(cat "$ROOT/etc/hostname" 2>/dev/null || echo ?)" "${old_main:-?}" "${new_main:-?}" + printf '#\n' + printf '# action: restore or skip. target: where it goes on this system (an\n' + printf '# absolute path; ~/ also works: the home of that line'"'"'s kind).\n' + printf '# state is what restore would find NOW: new, differs (-> .new), same.\n' + printf '# Apply: lfs-backup restore %s --plan %s (add --run)\n' "$a" "$out" + printf '#\n# action | state | kind:section | name in the archive | target\n' + while IFS=$'\t' read -r scope name mode owner group link label; do + case "$scope" in ''|\#*) continue ;; esac + label="${label:-$scope}" + selected "$scope" "$label" || continue + t="$(target_for "$scope" "$name")" + if [ -z "$t" ]; then state="no-home"; act=skip + elif [ ! -e "$t" ] && [ ! -L "$t" ]; then state="new"; act=restore + elif cmp -s "$tmp/files/$scope/${name#\~/}" "$t"; then state="same"; act=skip + else state="differs"; act=restore + fi + [ "$act" = restore ] && k=$((k + 1)) + # KIND:SECTION AND AN ABSOLUTE TARGET. `~/.bashrc` is in both + # [user] and [root]: the plan named them identically, restore looked + # a name up and took the FIRST line -- root's file followed the + # user's decision. And a `~/` target read as root's home to anyone + # checking as root. The kind is part of the key; the target is + # the real path on this system (still editable, ~/ still accepted). + printf '%-7s | %-7s | %-14s | %s | %s\n' "$act" "$state" "$scope:$label" "$name" "${t#$ROOT}" + n=$((n + 1)) + done < "$tmp/manifest" + } > "$out" + note "wrote $out: $n file(s), $k set to restore (identical ones are set to skip)" + note " edit it, then: $PROG restore $a --plan $out" +} + +# PLAN LOOKUP: name -> "action<TAB>target" for the files of this archive +load_plan() { # load_plan <file> + [ -r "$1" ] || die "no plan: $1" + PLAN_MAP="$(awk -F' [|] ' ' + /^[[:space:]]*(#|$)/ { next } + NF < 5 { printf "bad\t%d\n", NR; next } + { a = $1; gsub(/[[:space:]]/, "", a); k = $3; gsub(/[[:space:]]/, "", k) + if (k !~ /^(user|root|system):/) { printf "bad\t%d\n", NR; next } + sub(/:.*/, "", k) + printf "%s:%s\t%s\t%s\n", k, $4, a, $5 }' "$1")" + local bad + bad="$(printf '%s\n' "$PLAN_MAP" | awk -F'\t' '$1 == "bad" { printf "%s ", $2 }')" + [ -z "$bad" ] || die "$1: lines $bad are not action | state | kind:section | name | target (a plan from lfs-backup 1.2.1 or older: write a new one)" + bad="$(printf '%s\n' "$PLAN_MAP" | awk -F'\t' '$2 != "restore" && $2 != "skip" { printf "%s ", $1 }')" + [ -z "$bad" ] || die "$1: an action is neither restore nor skip for: $bad" +} + +do_restore() { + need_root restore + local a="${1:-}" tmp old_main new_main uhome rhome + [ -r "$a" ] || die "no archive: ${a:-(none given)}" + tmp="$(mktemp -d)" || die "no temporary directory" + TMPD="$tmp"; chmod 700 "$tmp" + tar -C "$tmp" -xzf "$a" || die "cannot read $a" + [ -f "$tmp/manifest" ] || die "$a has no manifest -- not an $PROG archive" + old_main="$(sed -n 's/^# host=.* main_user=\([^ ]*\) .*/\1/p' "$tmp/manifest" | head -n1)" + new_main="$(main_user)" + uhome="$([ -n "$new_main" ] && home_of "$new_main")" + rhome="$(home_of root)"; rhome="${rhome:-/root}" + note "archive main user: ${old_main:-?} this system's: ${new_main:-?} ($uhome)" + [ "$ROOT" = "" ] || note "restoring into $ROOT" + + local scope name mode owner group link label rel target o g planned + local created=() same=() news=() skipped=() modediff=() unselected=0 planskip=0 unplanned=0 + [ -n "$PLAN" ] && { load_plan "$PLAN"; note "restoring by plan $PLAN"; } + while IFS=$'\t' read -r scope name mode owner group link label; do + case "$scope" in ''|\#*) continue ;; esac + label="${label:-$scope}" + selected "$scope" "$label" || { unselected=$((unselected + 1)); continue; } + rel="${name#\~/}" + case "$scope" in user|root|system) ;; *) skipped+=("$name (unknown scope $scope)"); continue ;; esac + planned="$name" + if [ -n "$PLAN" ]; then + local line + line="$(printf '%s\n' "$PLAN_MAP" | awk -F'\t' -v n="$scope:$name" '$1 == n { print; exit }')" + if [ -z "$line" ]; then unplanned=$((unplanned + 1)); continue; fi + [ "$(printf '%s' "$line" | cut -f2)" = restore ] || { planskip=$((planskip + 1)); continue; } + planned="$(printf '%s' "$line" | cut -f3-)" + fi + target="$(target_for "$scope" "$planned")" + if [ -z "$target" ]; then + skipped+=("$name ($([ -z "$uhome" ] && echo "no main user here" || echo "target '$planned' is neither ~/... nor absolute"))") + continue + fi + read -r o g <<< "$(owner_for "$scope" "$owner" "$group")" + if printf '%s' "$name" | grep -qE "$SECRET_RE"; then + local m2; m2="$(printf '%o' $(( 8#$mode & 8#700 )))" + if [ "$m2" != "$mode" ]; then + note "$name: a secret recorded $mode -- restored $m2" + mode="$m2" + fi + fi + [ "$scope" != system ] || [ "$o" = "$owner" ] \ + || note "$name: owner '$owner' -> $o" + local src="$tmp/files/$scope/$rel" dest="$target" why="" + if [ ! -e "$target" ] && [ ! -L "$target" ]; then + if [ "$scope" = system ] && printf '%s' "$name" | grep -qE "$ALWAYS_NEW"; then + dest="$target.new"; why="account database: always merged" + else + why="new" + fi + elif cmp -s "$src" "$target"; then + same+=("$target") + [ "$(stat -L -c %a "$target")" = "$mode" ] \ + || modediff+=("$target is $(stat -L -c %a "$target"), backup had $mode") + continue + else + dest="$target.new"; why="differs" + fi + if [ "$dest" != "$target" ] && [ -f "$dest" ] && cmp -s "$src" "$dest"; then + news+=("$dest ($why; already there from an earlier run)") + continue + fi + if [ "$RUN" = 1 ]; then + if [ "$dest" = "$target" ]; then + # parents that do not exist yet: under a home they belong to its user + local d="$(dirname "$target")" missing_dirs=() + while [ ! -d "$d" ]; do missing_dirs=("$d" "${missing_dirs[@]}"); d="$(dirname "$d")"; done + for d in "${missing_dirs[@]}"; do + mkdir "$d" || { warn "cannot create $d"; continue 2; } + case "$d" in "$ROOT$uhome"/*) [ -n "$uhome" ] && chown "$(uid_of "$new_main"):$(gid_of "$new_main")" "$d" 2>/dev/null ;; esac + done + fi + install -m "$mode" -o "$(uid_of "$o")" -g "$(gid_of "$g")" "$src" "$dest" \ + || { warn "could not write $dest"; continue; } + fi + if [ "$dest" = "$target" ]; then + created+=("$target ($mode $o:$g)") + else + news+=("$dest ($why)") + fi + done < "$tmp/manifest" + + say "" + local verb; [ "$RUN" = 1 ] && verb="" || verb="would be " + say "${verb}created: ${#created[@]}" + for x in "${created[@]}"; do say " $x"; done + say "identical, left alone: ${#same[@]}" + for x in "${modediff[@]}"; do say " mode only: $x"; done + [ "${#skipped[@]}" -gt 0 ] && { say "skipped: ${#skipped[@]}"; for x in "${skipped[@]}"; do say " $x"; done; } + [ "$unselected" -gt 0 ] && say "not selected (--only/--skip): $unselected" + [ "$planskip" -gt 0 ] && say "skip in the plan: $planskip" + [ "$unplanned" -gt 0 ] && say "not in the plan: $unplanned (left alone)" + say "" + if [ "${#news[@]}" -gt 0 ]; then + say "TO MERGE BY HAND -- ${verb}written as .new, the live file untouched: ${#news[@]}" + for x in "${news[@]}"; do say " $x"; done + say "" + note "compare: diff -u <file> <file>.new then keep what you want and rm the .new" + else + note "nothing to merge" + fi + [ "$RUN" = 1 ] || note "dry run -- nothing written. Apply: $PROG restore $a$([ -n "$PLAN" ] && printf ' --plan %s' "$PLAN") --run$(sel_args)" + [ "${#news[@]}" -gt 0 ] && note "merged everything? $PROG clean $a$(sel_args) removes the .new files" + return 0 +} + +sel_args() { [ -n "$ONLY" ] && printf ' --only %s' "$ONLY"; [ -n "$SKIP" ] && printf ' --skip %s' "$SKIP"; } + +# ---- sections ----------------------------------------------------------- +do_sections() { + local a="${1:-}" + if [ -n "$a" ]; then + [ -r "$a" ] || die "no archive: $a" + tar -xzOf "$a" manifest | awk -F'\t' '!/^#/ { l = ($7 != "" ? $7 : $1); n[l]++; k[l] = $1 } + END { for (l in n) printf " %-12s %-7s %d file(s)\n", l, k[l], n[l] }' | sort + else + ALL_SECTIONS=1 resolve | awk -F'\t' '$1 != "missing" { n[$4]++; k[$4] = $1 } + END { for (l in n) printf " %-12s %-7s %d file(s)\n", l, k[l], n[l] }' | sort + fi +} + +# ---- merge -------------------------------------------------------------- +# ONE .new AT A TIME, WITH THE DIFF IN FRONT OF YOU. +# k keep the live file -> the .new is removed +# n take the backed-up file -> its CONTENT replaces the live file's; +# the live owner and mode stay (they are +# this system's), then the .new is removed +# e edit: vimdiff live .new (MERGE_TOOL overrides), then ask again +# s skip, decide later q stop here +# The account databases (passwd, group, shadow, gshadow) cannot be taken +# whole: an old system's copy locks out this one's accounts. Edit them. +do_merge() { + need_root merge + local a="${1:-}" tmp new_main uhome rhome scope name mode owner group link label target + local kept=0 took=0 skipped=0 ans tool + [ -r "$a" ] || die "no archive: ${a:-(none given)}" + tmp="$(mktemp -d)" || die "no temporary directory" + TMPD="$tmp"; chmod 700 "$tmp" + tar -C "$tmp" -xzf "$a" manifest || die "cannot read $a" + new_main="$(main_user)" + uhome="$([ -n "$new_main" ] && home_of "$new_main")" + rhome="$(home_of root)"; rhome="${rhome:-/root}" + [ -n "$PLAN" ] && load_plan "$PLAN" + tool="${MERGE_TOOL:-}" + [ -n "$tool" ] || { command -v vimdiff >/dev/null 2>&1 && tool=vimdiff; } + # answers from the terminal, or from stdin when there is none + { exec 3</dev/tty; } 2>/dev/null || exec 3<&0 + local list=() t + while IFS=$'\t' read -r scope name mode owner group link label; do + case "$scope" in ''|\#*) continue ;; esac + selected "$scope" "${label:-$scope}" || continue + local planned="$name" + if [ -n "$PLAN" ]; then + local line + line="$(printf '%s\n' "$PLAN_MAP" | awk -F'\t' -v n="$scope:$name" '$1 == n { print; exit }')" + [ -n "$line" ] && [ "$(printf '%s' "$line" | cut -f2)" = restore ] || continue + planned="$(printf '%s' "$line" | cut -f3-)" + fi + t="$(target_for "$scope" "$planned")" + [ -n "$t" ] && [ -f "$t.new" ] && list+=("$t") + done < "$tmp/manifest" + [ "${#list[@]}" -gt 0 ] || { note "no .new files from this archive -- nothing to merge"; return 0; } + local i=0 total="${#list[@]}" + for target in "${list[@]}"; do + i=$((i + 1)) + while :; do + say "" + say "=== [$i/$total] $target" + if [ ! -e "$target" ]; then + say " (no live file: only the .new exists)" + else + diff -u --label "live: $target" --label "backup: $target.new" \ + "$target" "$target.new" | sed 's/^/ /' | head -n "${MERGE_LINES:-60}" + local nl; nl="$(diff "$target" "$target.new" | wc -l)" + [ "$nl" -gt "${MERGE_LINES:-60}" ] && say " ... ($nl diff lines; e shows all)" + fi + local takeable=1 + printf '%s' "$target" | sed "s#^$ROOT##" | grep -qE "$ALWAYS_NEW" && takeable=0 + if [ "$takeable" = 1 ]; then + printf ' [k]eep live [n] take backup [e]dit [s]kip [q]uit: ' + else + printf ' [k]eep live [e]dit [s]kip [q]uit (account database: no whole take): ' + fi + read -r ans <&3 || ans=q + case "$ans" in + k) rm -f "$target.new"; note "kept the live file; removed the .new"; kept=$((kept + 1)); break ;; + n) if [ "$takeable" != 1 ]; then warn "not for an account database -- edit it"; continue; fi + if [ -e "$target" ]; then + cat "$target.new" > "$target" || { warn "could not write $target"; continue; } + else + mv "$target.new" "$target" || { warn "could not move it into place"; continue; } + fi + rm -f "$target.new" + note "took the backup's content (the live owner and mode stay)"; took=$((took + 1)); break ;; + e) if [ -n "$tool" ] && [ -e "$target" ]; then + $tool "$target" "$target.new" <&3 + else + ${EDITOR:-vi} "$target.new" <&3 + fi ;; + s) skipped=$((skipped + 1)); break ;; + q) say ""; note "stopped at $target"; break 2 ;; + *) ;; + esac + done + done + exec 3<&- + say "" + note "kept live: $kept took backup: $took left for later: $((total - kept - took))" + return 0 +} + +# ---- clean -------------------------------------------------------------- +# THE .new FILES OF THIS ARCHIVE, AND ONLY THOSE. For each file the archive +# holds, <target>.new is removed when its content is still exactly what +# restore wrote. A .new you edited while merging differs from the archive +# and is kept, and so is any .new this archive did not produce (a package's +# config, `make install`'s). +do_clean() { + need_root clean + local a="${1:-}" tmp new_main uhome rhome scope name mode owner group link label rel target + local removed=() kept=() gone=0 + [ -r "$a" ] || die "no archive: ${a:-(none given)}" + tmp="$(mktemp -d)" || die "no temporary directory" + TMPD="$tmp"; chmod 700 "$tmp" + tar -C "$tmp" -xzf "$a" || die "cannot read $a" + new_main="$(main_user)" + uhome="$([ -n "$new_main" ] && home_of "$new_main")" + rhome="$(home_of root)"; rhome="${rhome:-/root}" + [ -n "$PLAN" ] && load_plan "$PLAN" + while IFS=$'\t' read -r scope name mode owner group link label; do + case "$scope" in ''|\#*) continue ;; esac + selected "$scope" "${label:-$scope}" || continue + rel="${name#\~/}" + local planned="$name" + if [ -n "$PLAN" ]; then + local line + line="$(printf '%s\n' "$PLAN_MAP" | awk -F'\t' -v n="$scope:$name" '$1 == n { print; exit }')" + [ -n "$line" ] && [ "$(printf '%s' "$line" | cut -f2)" = restore ] || continue + planned="$(printf '%s' "$line" | cut -f3-)" + fi + target="$(target_for "$scope" "$planned")" + [ -n "$target" ] || continue + [ -f "$target.new" ] || { gone=$((gone + 1)); continue; } + if cmp -s "$tmp/files/$scope/$rel" "$target.new"; then + [ "$RUN" = 1 ] && rm -f "$target.new" + removed+=("$target.new") + else + kept+=("$target.new (edited since restore -- yours to delete)") + fi + done < "$tmp/manifest" + local verb; [ "$RUN" = 1 ] && verb="removed" || verb="would remove" + say "$verb: ${#removed[@]}" + for x in "${removed[@]}"; do say " $x"; done + if [ "${#kept[@]}" -gt 0 ]; then + say "kept: ${#kept[@]}" + for x in "${kept[@]}"; do say " $x"; done + fi + [ "$RUN" = 1 ] || note "dry run -- nothing removed. Remove: $PROG clean $a --run$(sel_args)" + return 0 +} + +# ---- main ----------------------------------------------------------------- +cmd="${1:-}"; [ $# -gt 0 ] && shift +args=() +while [ $# -gt 0 ]; do + case "$1" in + --run) RUN=1 ;; + --plan) PLAN="${2:?--plan needs a file}"; shift ;; + --plan=*) PLAN="${1#--plan=}" ;; + --only) ONLY="${2:?--only needs section names}"; shift ;; + --only=*) ONLY="${1#--only=}" ;; + --skip) SKIP="${2:?--skip needs section names}"; shift ;; + --skip=*) SKIP="${1#--skip=}" ;; + *) args+=("$1") ;; + esac + shift +done +case "$cmd" in + list) do_list ;; + create) do_create "${args[@]+"${args[@]}"}" ;; + show) do_show "${args[@]+"${args[@]}"}" ;; + restore) do_restore "${args[@]+"${args[@]}"}" ;; + clean) do_clean "${args[@]+"${args[@]}"}" ;; + plan) do_plan "${args[@]+"${args[@]}"}" ;; + merge) do_merge "${args[@]+"${args[@]}"}" ;; + sections) do_sections "${args[@]+"${args[@]}"}" ;; + --version|version) say "$PROG $VERSION (build $(md5sum "$0" 2>/dev/null | cut -c1-7))" ;; + ""|-h|--help|help) usage ;; + *) die "unknown command '$cmd' -- $PROG --help" ;; +esac diff --git a/lfs-helper b/lfs-helper index 2e69f94..20582ad 100755 --- a/lfs-helper +++ b/lfs-helper @@ -49,7 +49,7 @@ set -u # Bump on every change that alters behaviour. `lfs-helper --version` makes it # possible to tell at a glance whether the copy inside the chroot is the one # that was just fixed -- guessing at that has wasted a lot of time. -LFS_HELPER_VERSION="1.14.146" +LFS_HELPER_VERSION="1.14.321" # A fingerprint of this file's own contents. A hand-maintained version number # goes stale the moment someone forgets to bump it -- and the reason for @@ -980,7 +980,7 @@ cmd_add_user() { pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" if user_exists "$name"; then - say "# user '$name' already exists" + [ "${_QUIET_EXISTING:-0}" = 1 ] || say "# user '$name' already exists" else if have_shadow_tools; then # Let Shadow allocate the ids. Choosing them by hand means racing @@ -1056,14 +1056,20 @@ load_snapshot() { have_su() { command -v su >/dev/null 2>&1; } snapshot_dirs() { + # PRUNE, DO NOT FILTER. `-not -path "$SRCROOT/*"` hides a path AFTER find + # has walked into it, so every scan still descended /usr/src -- every + # package's unpacked source and build tree, LLVM's and WebKit's included + # -- and threw the answer away. Each phase ended with five such walks: + # nvtop's two-second install sat for minutes on "scanning the tree". + # -prune does not descend. The pruned directory itself is still listed, + # exactly as the filter listed it. local r="${SNAP_ROOT%/}" - find "${SNAP_ROOT:-/}" -xdev -type d \ - -not -path "$SRCROOT/*" \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -not -path "$r/tmp/*" -not -path "$r/sources/*" \ - -not -path "$r/build/*" \ - -not -path "$r/root/*" -not -path "$r/home/*" \ + find "${SNAP_ROOT:-/}" -xdev \ + \( -path "$SRCROOT" -o -path "$r/dev" -o -path "$r/proc" \ + -o -path "$r/sys" -o -path "$r/run" -o -path "$r/tmp" \ + -o -path "$r/sources" -o -path "$r/build" \ + -o -path "$r/root" -o -path "$r/home" \) -type d -print -prune \ + -o -type d -print \ 2>/dev/null | LC_ALL=C sort } @@ -1078,13 +1084,13 @@ snapshot() { # since comm cannot tell. A package user's profile sets LC_ALL=POSIX and # root's shell may not, so the two sides really can differ. local r="${SNAP_ROOT%/}" - find "${SNAP_ROOT:-/}" -xdev \( -type f -o -type l \) \ - -not -path "$SRCROOT/*" \ - -not -path "$r/dev/*" -not -path "$r/proc/*" \ - -not -path "$r/sys/*" -not -path "$r/run/*" \ - -not -path "$r/tmp/*" -not -path "$r/sources/*" \ - -not -path "$r/build/*" \ - -not -path "$r/root/*" -not -path "$r/home/*" \ + # pruned, not filtered -- see snapshot_dirs + find "${SNAP_ROOT:-/}" -xdev \ + \( -path "$SRCROOT" -o -path "$r/dev" -o -path "$r/proc" \ + -o -path "$r/sys" -o -path "$r/run" -o -path "$r/tmp" \ + -o -path "$r/sources" -o -path "$r/build" \ + -o -path "$r/root" -o -path "$r/home" \) -type d -prune \ + -o \( -type f -o -type l \) -print \ 2>/dev/null | LC_ALL=C sort } @@ -2492,6 +2498,64 @@ unwritable_dirs_from_log() { | sort -u)" fi + # CARGO NAMES THE DESTINATION IN BACKTICKS, ON THE LINE ABOVE THE REASON. + # + # Error: CliError { error: Some(failed to copy `.../libgstrsrtp.a` to + # `/usr/lib/gstreamer-1.0/libgstrsrtp.a` + # Caused by: + # Permission denied (os error 13) + # + # No quotes, no "Permission denied" on the same line, and the path that + # matters is the SECOND backticked one -- so nothing matched, the repair + # ran zero rounds, and the box still said "grant access and retry". + local _cargodirs="" + if grep -qE "failed to copy \`|os error 13" "$log" 2>/dev/null; then + _cargodirs="$(tail -c 524288 "$log" 2>/dev/null \ + | grep -aoE 'to `/[^`]+`' \ + | sed 's/^to `//; s/`$//' \ + | sed 's|/[^/]*$||' \ + | sort -u)" + fi + + # AN ABSOLUTE PATH IN THE MESSAGE IS THE ANSWER, AND NOBODY READ IT. + # + # install: cannot create regular file + # '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied + # + # The file does not exist, so the handover has nothing to give away; the + # directory is another package's and not writable, so the grant is what + # is needed -- and the `cd`-based extractor below only fires when the + # build printed a `cd` and then a RELATIVE path. With an absolute path + # right there in the message, nothing was extracted and light's udev + # rule stopped the install. + # CMAKE WRAPS ITS MESSAGE. + # + # file INSTALL cannot copy file + # "/usr/src/pkgusr/p_nvtop/.../io.github.syllo.nvtop.metainfo.xml" + # to "/usr/share/metainfo/io.github.syllo.nvtop.metainfo.xml": Permission + # denied. + # "Permission" and "denied" on two lines, so no line matched and the second + # round found nothing to grant: nvtop stopped after one round with a plain + # permission error on the screen. Join each `file INSTALL` block (up to + # the blank line CMake ends it with) and read the destination from that. + local _cmakedirs="" + _cmakedirs="$(tail -c 524288 "$log" 2>/dev/null | awk ' + /file INSTALL cannot/ { blk = $0; inb = 1; next } + inb && /^[[:space:]]*$/ { print blk; inb = 0; next } + inb { blk = blk " " $0 } + END { if (inb) print blk }' \ + | tr -s ' ' \ + | grep -aE 'Permission denied|Operation not permitted' \ + | grep -aoE '(to|directory) "/[^"]+"' | sed 's/^[a-z]* "//; s/"$//' \ + | while IFS= read -r _cp; do + if [ -d "$_cp" ]; then echo "$_cp"; else echo "${_cp%/*}"; fi + done | sort -u)" + local _instdirs="" + _instdirs="$(tail -c 524288 "$log" 2>/dev/null \ + | grep -aoE "cannot create (regular file|directory) '/[^']+'" \ + | sed "s/.*'\(\/[^']*\)'.*/\1/" \ + | sed 's|/[^/]*$||' | sort -u)" + # (plain awk: match() with a capture array is a gawk extension) _cdpaths="$(tail -c 524288 "$log" 2>/dev/null | awk ' /cd .\/|cd \x27\// { @@ -2506,6 +2570,9 @@ unwritable_dirs_from_log() { }' 2>/dev/null | sort -u)" { printf '%s\n' "$_mesondirs" + printf '%s\n' "$_cargodirs" + printf '%s\n' "$_instdirs" + printf '%s\n' "$_cmakedirs" printf '%s\n' "$_cdpaths" printf '%s\n' "$_tail" | grep -aoE "'[^']+'|\"[^\"]+\"" | tr -d "'\"" printf '%s\n' "$_tail" | grep -aoE "/[A-Za-z0-9._/+-]{6,}"; } \ @@ -2884,7 +2951,7 @@ meson_takeover_from_log() { # meson's chmod succeeds. Only files it is actually installing (named on # meson's own "Installing SRC to DIR" lines) and only where the current # owner is another PACKAGE USER: root's files are never taken. - local log="$1" user="$2" took=0 src dest f owner + local log="$1" user="$2" took=0 src dest f owner _t_owner="" _t_src="" grep -qE "Unhandled python OSError|\[Errno 13\]|\[Errno 1\]" "$log" 2>/dev/null || return 1 while IFS= read -r line; do src="${line#Installing }"; src="${src%% to *}" @@ -2899,8 +2966,47 @@ meson_takeover_from_log() { esac pkgusr_kind "$owner" >/dev/null 2>&1 || true say " $f belongs to $owner -- handing it to $user (the installer owns what it installs)" - soft "could not hand $f to $user" -- real_chown "$user:$user" "$f" && took=$((took+1)) + soft "could not hand $f to $user" -- real_chown "$user:$user" "$f" \ + && { took=$((took+1)); _t_owner="$owner"; _t_src="$src"; } done <<< "$(tail -c 2097152 "$log" 2>/dev/null | grep '^Installing .* to /')" + # THE WHOLE INSTALL PLAN, NOT THE LINE IT STOPPED ON. + # + # libglvnd replaces Mesa's GL, GLES, EGL and KHR headers and its libGL: + # a legitimate move of dozens of files from p_mesa to p_libglvnd. meson + # stops at the first one, so the log named one file per round, and after + # three the round limit called it a bundled subproject -- it was not. + # meson writes its complete plan to meson-info/intro-installed.json + # (source -> destination) at setup, so once one file has had to change + # hands, every destination still owned by that same package changes + # hands in this round. Same guards: package users only, never root. + # Not for a subproject's file: that is the bundled-copy case below, and + # handing its whole plan over would hide it. + if [ "$took" -ge 1 ] && [ -n "$_t_owner" ] \ + && [ "${_t_src#*/subprojects/}" = "$_t_src" ]; then + local _plan="" _dir _home _n=0 _d _o + # walk up from the file meson was installing to the tree that holds + # meson-info, and no further than the package user's home + _home="$(getent passwd "$user" | cut -d: -f6)" + _dir="$(dirname "$_t_src")" + while [ -n "$_dir" ] && [ "$_dir" != / ]; do + _plan="$(find "$_dir" -maxdepth 3 -path '*/meson-info/intro-installed.json' 2>/dev/null | head -1)" + [ -n "$_plan" ] && break + [ "$_dir" = "$_home" ] && break + _dir="$(dirname "$_dir")" + done + if [ -n "$_plan" ] && command -v python3 >/dev/null 2>&1; then + while IFS= read -r _d; do + [ -e "$_d" ] && [ ! -L "$_d" ] || continue + _o="$(stat -c %U "$_d" 2>/dev/null)" || continue + [ "$_o" = "$_t_owner" ] || continue + real_chown "$user:$user" "$_d" 2>/dev/null && _n=$((_n+1)) + done <<< "$(python3 -c 'import json,sys; print("\n".join(json.load(open(sys.argv[1])).values()))' "$_plan" 2>/dev/null)" + if [ "$_n" -gt 0 ]; then + say " ...and $_n more file(s) of $_t_owner that meson's install plan puts there" + took=$((took + _n)) + fi + fi + fi # ONE FILE PER ROUND IS A LOOP, NOT PROGRESS. # # meson stops at the FIRST file it may not replace, so its log names only @@ -2909,8 +3015,9 @@ meson_takeover_from_log() { # .mo file per round. When a package keeps taking a single file from the # same other package, it is installing a BUNDLED COPY of it, and the fix # is upstream of the ownership question. - if [ "$took" = 1 ] && [ -n "${owner:-}" ]; then - LAST_SINGLE_HANDOVER="$owner" + # ...and only a file that comes FROM a subproject is a bundled copy. + if [ "$took" = 1 ] && [ -n "$_t_owner" ] && [ "${_t_src#*/subprojects/}" != "$_t_src" ]; then + LAST_SINGLE_HANDOVER="$_t_owner" else LAST_SINGLE_HANDOVER="" fi @@ -2918,6 +3025,255 @@ meson_takeover_from_log() { } +# A RULE ON DISK IS NOT A RULE IN FORCE. +# +# light installs 90-backlight.rules, and until udevd is told, nothing has +# changed: /sys/class/backlight/*/brightness keeps its old group, light still +# cannot write it, and the only symptom is "it does not work" -- on a system +# where every file is exactly where it should be. A reboot fixes it, which +# is how this kind of thing stays hidden for a week. +# +# Reload always (cheap, and cannot break a running device), then re-trigger +# ONLY the subsystems the new rules actually mention. A bare `udevadm +# trigger` re-adds every device on the machine; on a desktop that can reset +# input devices and drop a session, which is not a thing a package install +# should do. +_udev_reload_if_rules_changed() { # <manifest> + local man="$1" rules subs + [ -r "$man" ] || return 0 + rules="$(grep -E '/udev/rules\.d/[^/]+\.rules$' "$man" 2>/dev/null)" + [ -n "$rules" ] || return 0 + if ! command -v udevadm >/dev/null 2>&1; then + note "# installed a udev rule, and there is no udevadm here to load it" + return 0 + fi + # IN A CHROOT THERE IS NO udevd TO TELL. Saying so beats a confusing + # "Failed to send reload request". + if [ "$(stat -c %d:%i /proc/1/root/. 2>/dev/null)" \ + != "$(stat -c %d:%i / 2>/dev/null)" ]; then + note "# a udev rule was installed; this is a chroot, so nothing was" + note "# reloaded -- it takes effect when you boot this tree" + return 0 + fi + say "# a udev rule changed -- telling udevd:" + if udevadm control --reload 2>/dev/null; then + say "# rules reloaded" + else + warn "udevadm control --reload failed; the rule is on disk but not in force" + return 0 + fi + # THE ACTION MATTERS AS MUCH AS THE SUBSYSTEM. + # + # `udevadm trigger` sends **change** unless told otherwise, and light's + # rule is `ACTION=="add", SUBSYSTEM=="backlight", RUN+="chgrp video ..."` + # -- so the trigger fired, udevd re-examined the device, the rule did not + # match, and /sys/class/backlight/*/brightness stayed root:root. Reload + # plus trigger LOOKED like it had done the job. + # + # So take the actions out of the rules too, and trigger each one. + acts="$(printf '%s\n' "$rules" | while IFS= read -r r; do + [ -r "$r" ] || continue + sed -n 's/.*ACTION=="\([a-z]*\)".*/\1/p' "$r" + done | sort -u | tr '\n' ' ')" + [ -n "$acts" ] || acts="change" + subs="$(printf '%s\n' "$rules" | while IFS= read -r r; do + [ -r "$r" ] || continue + sed -n 's/.*SUBSYSTEM=="\([a-z0-9_-]*\)".*/\1/p' "$r" + done | sort -u | tr '\n' ' ')" + if [ -z "$subs" ]; then + say "# no SUBSYSTEM== in the new rule(s), so nothing was re-triggered." + say "# If a device needs it: udevadm trigger --action=add --subsystem-match=<what>" + return 0 + fi + for sub in $subs; do + for act in $acts; do + if udevadm trigger --action="$act" --subsystem-match="$sub" 2>/dev/null; then + say "# re-triggered '$sub' with action '$act'" + else + warn "could not re-trigger '$sub' ($act); by hand:" + warn " udevadm trigger --action=$act --subsystem-match=$sub" + fi + done + done + udevadm settle --timeout=10 >/dev/null 2>&1 || true +} + +_missing_py_module() { # _missing_py_module <log> -> module name or "" + grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ + "$1" 2>/dev/null | head -n1 | cut -d"'" -f2 +} + +# A TEST'S DEPENDENCY IS NOT A BUILD'S DEPENDENCY, and installing it would +# have hidden a generator bug for two releases. numpy's install phase +# wrongly contained its test suite; the missing module was `pytest`, and an +# eager "install what is missing" would have installed pytest, run a +# multi-minute test suite inside an install, and left the real fault -- +# a test block in the wrong phase -- undiscovered. +_PY_TEST_ONLY="pytest py.test hypothesis attrs pytest_xdist pytest-xdist + nose coverage tox mock flaky" + +_py_module_provider() { # _py_module_provider <module> -> BLFS page or "" + case "$1" in + mesonpy) echo meson_python ;; + setuptools_scm) echo setuptools_scm ;; + Cython|cython) echo Cython ;; + numpy) echo NumPy ;; + flit_core) echo flit_core ;; + hatchling) echo hatchling ;; + scikit_build_core) echo scikit-build-core ;; + poetry.core|poetry_core) echo poetry-core ;; + wheel) echo wheel ;; + setuptools) echo setuptools ;; + *) echo "" ;; + esac +} + +# INSTALL A MISSING BUILD BACKEND, IF ASKED TO. Opt-in (--auto-modules or +# PM_AUTO_MODULES=1): a build that cannot proceed without a python backend +# is a dependency problem the tool can fix, but installing packages nobody +# asked for is not something to do by default -- and never for a module +# only the TEST suite wants. +_install_missing_module() { # _install_missing_module <log> + local mod prov + mod="$(_missing_py_module "$1")" + [ -n "$mod" ] || return 1 + case " $_PY_TEST_ONLY " in + *" $mod "*) + warn "'$mod' is a TEST dependency, not a build one -- NOT installing it" + note " a test suite in an install phase is the real fault here." + note " Regenerate the script and the test moves to \`test\`:" + note " packagemanager regenerate-script $(unprefix_pkg_user "${name:-}") --overwrite" + return 1 ;; + esac + prov="$(_py_module_provider "$mod")" + if [ -z "$prov" ]; then + warn "no known BLFS page provides the python module '$mod'" + note " find it: packagemanager search $mod" + return 1 + fi + say "# '$mod' is missing; installing $prov (asked for with --auto-modules) ..." + if packagemanager install "$prov" --run --yes --recursive; then + printf '%s\n' "$prov ($mod)" >> "${_AUTO_MODULES_LOG:-/dev/null}" + return 0 + fi + warn "installing $prov failed; $mod is still missing" + return 1 +} + +_take_cmake_dest() { # _take_cmake_dest <log> <package user> + # CMAKE NAMES THE DESTINATION ACROSS THREE LINES. + # + # CMake Error at tools/llvm-ar/cmake_install.cmake:52 (file): + # file INSTALL cannot copy file + # ".../build/bin/llvm-ar" to + # "/usr/bin/llvm-ar": Permission denied. + # + # /usr/bin is drwxrwxr-t root:install -- writable by the group and + # STICKY, so only the file's owner may replace it, and /usr/bin/llvm-ar + # belonged to p_rust. No grant fixes that: the directory was never the + # problem. The meson handover next door reads "Installing SRC to DIR" + # lines and needs "Unhandled python OSError", so it saw nothing either, + # and the repair ran zero rounds. + # + # Ownership follows the installer. llvm-ar is llvm's file; take it, and + # only from another PACKAGE USER -- root's files are never taken. + # the log is joined into one line before matching, because the message + # is three lines and the destination is on the third + local log="$1" user="${2:-}" dest owner took=0 + [ -r "$log" ] || return 1 + [ -n "$user" ] || return 1 + # ...AND A SYMLINK IT WILL NOT OVERWRITE AT ALL. + # + # file INSTALL cannot duplicate symlink + # ".../build/libtiff/libtiff.so.6" + # at + # "/usr/lib/libtiff.so.6" + # because: File exists + # + # Round 1 handed the .pc file over and the retry then died on this: the + # ownership was fine, cmake simply refuses to replace an existing + # symlink. Handing it over changes nothing -- it has to GO. Only + # another package user's, never root's, and only after saying whose it + # was. + # THE GATE HAS TO KNOW THE WORDING TOO. The extractor learned meson's + # "Operation not permitted: '<path>'" and this gate did not, so the + # function returned before looking -- the same shape as the 1.14.222 + # cmake detector: two places, one fact. + grep -qE "file INSTALL cannot (copy file|duplicate symlink)|Operation not permitted: '" \ + "$log" 2>/dev/null || return 1 + while IFS= read -r dest; do + [ -n "$dest" ] || continue + case "$dest" in /*) ;; *) continue ;; esac + [ -e "$dest" ] || continue + owner="$(stat -c %U "$dest" 2>/dev/null)" || continue + [ "$owner" = "$user" ] && continue + case "$owner" in + root|"") + # A ROOT FILE THIS PACKAGE'S MANIFEST CLAIMS (a setid bit we + # applied) is handed over BEFORE the phase by the lease, and + # restored if the phase fails. 1.14.261 did it here, after + # the failure, which needed this log's wording to match. + # Nothing root owns is taken here any more. + note "# $dest is root's -- left alone"; continue ;; + esac + if [ -L "$dest" ]; then + # cmake will not overwrite a symlink whatever it is owned by + say " $dest is $owner's symlink -- removing it ($user installs it now)" + rm -f "$dest" 2>/dev/null || { + warn "could not remove $dest"; continue; } + took=1 + continue + fi + say " $dest belongs to $owner -- handing it to $user (the installer owns what it installs)" + chown "$user:$user" "$dest" 2>/dev/null || { + warn "could not hand $dest to $user"; continue; } + took=1 + done <<EOF +$(tail -c 262144 "$log" 2>/dev/null | tr '\n' ' ' \ + | grep -aoE 'file INSTALL cannot copy file[[:space:]]+"[^"]+"[[:space:]]+to[[:space:]]+"[^"]+"' \ + | sed 's/.*[[:space:]]to[[:space:]]*"//; s/"$//' | sort -u +tail -c 262144 "$log" 2>/dev/null | tr '\n' ' ' \ + | grep -aoE 'cannot duplicate symlink[[:space:]]+[^[:space:]]+[[:space:]]+at[[:space:]]+[^[:space:]]+' \ + | awk '{print $NF}' | sort -u +# meson/python say it differently, and EPERM on a sticky directory is the +# same problem: "Operation not permitted: '/usr/sbin/unix_chkpwd'" +tail -c 262144 "$log" 2>/dev/null \ + | grep -aoE "Operation not permitted: '[^']+'" \ + | sed "s/.*'\([^']*\)'.*/\1/" | sort -u) +EOF + [ "$took" = 1 ] +} + +_clear_stale_dest() { # _clear_stale_dest <log> <package user> + # `mv`/`cp` refusing an existing destination is not a permission problem + # and no grant will fix it. Remove the stale copy IF the package being + # installed owns it, and say so; anything else is left alone, because a + # destination owned by someone else is a real conflict. + local log="$1" user="${2:-}" path owner cleared=0 + [ -r "$log" ] || return 1 + [ -n "$user" ] || return 1 + while IFS= read -r path; do + [ -n "$path" ] || continue + case "$path" in /*) ;; *) continue ;; esac + [ -e "$path" ] || continue + owner="$(stat -c %U "$path" 2>/dev/null)" + if [ "$owner" != "$user" ]; then + warn "$path exists and belongs to $owner, not $user --" + warn " leaving it alone: that is a conflict, not stale output" + continue + fi + note "# $path is $user's own output from a previous install;" + note "# removing it so this install can replace it" + rm -rf "$path" || { warn "could not remove $path"; continue; } + cleared=1 + done <<EOF +$(tail -c 262144 "$log" 2>/dev/null \ + | grep -aoE "cannot overwrite '[^']+'|cannot move '[^']+' to '[^']+': Directory not empty" \ + | grep -aoE "'[^']+'" | tr -d "'" | sort -u) +EOF + [ "$cleared" = 1 ] +} + auto_grant_from_log() { local log="$1" user="$2" need d granted=0 need="$(unwritable_dirs_from_log "$log" "$user")" @@ -3310,6 +3666,181 @@ pkg_users_list() { done < <(pkgusr_roots) } +# --------------------------------------------------------------------------- +# setuid/setgid bits: recorded during the build, applied by root afterwards. +# +# A PACKAGE USER OWNS FILES, NOT IDENTITY. The install and chmod wrappers +# refuse setid modes, which is right -- a package user CAN set setuid on its +# own file, and the result runs as the PACKAGE USER, not as root. That is +# how `su` came to say +# setgid: Operation not permitted +# instead of working: shadow's `install -m 4755 su` was downgraded to 755, +# and nothing set it afterwards. +# +# So: chown to root FIRST, then the mode. Both, or neither. +# --------------------------------------------------------------------------- + +# What upstream installs with a setid bit, for the case where no request was +# recorded (an older build, or a package installed by hand). +# path|mode|group|who asks for it -- checked only if the file exists, and +# always reported with the mode and owner actually found. +# +# `-` IN THE GROUP COLUMN MEANS KEEP WHAT IS THERE, and for a setUID file +# that is the right answer on this system: only the owner has to be root for +# the bit to mean root, and the group records which package installed the +# file. These all said `root`, so this table ran immediately after the +# request table and undid it -- `/usr/bin/su is now 4755 root:p_shadow` +# followed by `/usr/bin/su is now 4755 root:root`, seventeen changes to +# reach the wrong one. +# +# A setGID entry is different: THERE the group is the whole point +# (dbus-daemon-launch-helper must be group messagebus, fcrontab group +# fcron), so those keep a named group. +_setid_known() { + # ONE TABLE: setid.conf. This was a second list, embedded here, and it + # disagreed with setid.conf on the mode -- 4755 here, 4711 there -- so + # `cfg shadow-pam` set 4711 and the next `lfs-helper setid --run` set + # 4755, each correcting the other (the 1.14.260 shape again). Now the + # config file is the list and this reads it. Only root-owned lines are + # setid policy. Output: path|mode|group|why + local conf + if ! conf="$(_setid_conf)"; then + warn "no setid.conf found (looked in \$SETID_CONF, ${PKGUSR_ETC:-/etc/pkgusr}/stacks, beside lfs-helper)" + return 0 + fi + awk -F'|' -v src="$(basename "$conf")" ' + /^[[:space:]]*(#|$)/ { next } + { f = $1; m = $2; o = $3; g = $4 + gsub(/[[:space:]]/, "", f); gsub(/[[:space:]]/, "", m) + gsub(/[[:space:]]/, "", o); gsub(/[[:space:]]/, "", g) + if (f == "" || m == "") next + if (o != "" && o != "root") next + print f "|" m "|" (g == "" ? "-" : g) "|" src }' "$conf" +} + +_setid_state() { # -> "<mode> <owner>:<group>" + stat -c '%a %U:%G' "$1" 2>/dev/null +} + +_setid_apply() { # <mode> <group> <path> (root; reads the result back) + local mode="$1" grp="$2" f="$3" got="" + chown "root:$grp" "$f" || { warn " chown root:$grp $f failed"; return 1; } + chmod "$mode" "$f" || { warn " chmod $mode $f failed"; return 1; } + # CONFIRM YOUR OWN EFFECTS: chmod returns 0 on a nosuid mount too, and + # the bit is then not there. + got="$(_setid_state "$f")" + case "$got" in + "${mode#0} root:$grp"|"$mode root:$grp") + ok " $f is now $got"; return 0 ;; + esac + warn " $f still reads '$got', not '$mode root:$grp'" + warn " (a nosuid mount cannot hold a setid bit -- check: findmnt -T $f)" + return 1 +} + +cmd_setid() { + need_root + local run=0 a + for a in "$@"; do case "$a" in --run) run=1 ;; esac; done + local outstanding=0 applied=0 known=0 line mode grp why got home u f + + say "Setuid/setgid bits refused during the build (a package user cannot" + say "set them; root can):" + say "" + while IFS= read -r u; do + home="$(pkgusr_home_for "$u" 2>/dev/null)" || continue + # a ledger left root-owned by an older release blocks the NEXT + # build, not this one, so repair it whenever we are here anyway + for _lf in "$home/.pkgusr-setid.requests" "$home/.pkgusr-setid.applied"; do + [ -e "$_lf" ] || continue + [ "$(stat -c %U "$_lf" 2>/dev/null)" = "$u" ] && continue + chown "$u:$u" "$_lf" 2>/dev/null \ + && note "gave $u back $(basename "$_lf")" + done + [ -s "$home/.pkgusr-setid.requests" ] || continue + say " $u:" + while IFS= read -r line; do + [ -n "$line" ] || continue + # the recorded line is literally `chmod <mode> <path> [path ...]` + set -- $line + [ "${1:-}" = chmod ] || continue + mode="${2:-}"; shift 2 + for f in "$@"; do + case "$f" in -*) continue ;; esac + if [ ! -e "$f" ]; then + warn " $f: recorded, but not on disk (package removed?)" + continue + fi + got="$(_setid_state "$f")" + say " $f is $got, asked for $mode" + outstanding=$((outstanding+1)) + # KEEP THE GROUP THE FILE ALREADY HAS. Only the USER has + # to be root for a setuid bit to mean root; the group is how + # this system records which package owns a file, and + # root:root threw that away -- `/usr/bin/su is now 4755 + # root:root` when it should read root:p_shadow. setid.conf + # uses `-` for exactly this; the request path did not. + _grp_now="$(stat -c '%G' "$f" 2>/dev/null)" + [ -n "$_grp_now" ] || _grp_now=root + # A PATH setid.conf NAMES IS DECIDED THERE. The package asked + # for 4755 on su, the table says 4711; applying both flipped + # the file twice in one run. + if _setid_policy_for "$f" >/dev/null; then + say " (setid.conf decides this one -- see below)" + continue + fi + [ "$run" = 1 ] && { _setid_apply "$mode" "$_grp_now" "$f" \ + && applied=$((applied+1)); } + done + done < "$home/.pkgusr-setid.requests" + if [ "$run" = 1 ]; then + # moved aside, not deleted: what was applied stays answerable + cat "$home/.pkgusr-setid.requests" >> "$home/.pkgusr-setid.applied" 2>/dev/null + : > "$home/.pkgusr-setid.requests" 2>/dev/null || true + # ...and hand both back: they live in the package user's home and + # the next build appends to them as that user + chown "$u:$u" "$home/.pkgusr-setid.requests" \ + "$home/.pkgusr-setid.applied" 2>/dev/null || true + fi + done < <(pkg_users_list) + [ "$outstanding" = 0 ] && say " (none recorded)" + + say "" + say "Programs upstream installs with a setid bit, as found on disk:" + say "" + while IFS='|' read -r f mode grp why; do + [ -n "$f" ] || continue + [ -e "$f" ] || continue + got="$(_setid_state "$f")" + # `-` MEANS KEEP THE GROUP THE FILE HAS. With `root` hard-coded + # here, this table undid the request table one line later. + if [ "$grp" = "-" ]; then + grp="$(stat -c '%G' "$f" 2>/dev/null)" + [ -n "$grp" ] || grp=root + fi + [ "$got" = "${mode#0} root:$grp" ] && continue + known=$((known+1)) + say " $f" + say " is $got" + say " want $mode root:$grp ($why)" + [ "$run" = 1 ] && { _setid_apply "$mode" "$grp" "$f" \ + && applied=$((applied+1)); } + done < <(_setid_known) + [ "$known" = 0 ] && say " (each one is already root-owned with its bit)" + + say "" + if [ "$run" = 1 ]; then + say "Applied $applied change(s). Try what was broken: su -" + return 0 + fi + if [ $((outstanding + known)) -gt 0 ]; then + say "Nothing was changed. Apply it: lfs-helper setid --run" + return 1 + fi + ok "every setid bit this system needs is in place" + return 0 +} + cmd_find_user_site() { need_root local run=0; [ "${1:-}" = "--run" ] && run=1 @@ -3514,7 +4045,22 @@ EOF # install: cannot change permissions of '/usr/sbin': Operation not permitted # The directory is already there and correctly set up, so there is nothing to do. real=/usr/bin/install -args=(); skipped="" + +# CHECK BEFORE REDIRECTING. `printf >> file 2>/dev/null || echo ...` does +# not work: the shell itself reports the failed redirection ("Permission +# denied") before printf ever runs, and 2>/dev/null on printf cannot +# suppress it -- which is how the recorder's own message came to fail an +# install that had finished. +_sid_record() { # _sid_record <line> + local f="$HOME/.pkgusr-setid.requests" + if [ -w "$f" ] || { [ ! -e "$f" ] && [ -w "$HOME" ]; }; then + printf '%s\n' "$1" >> "$f" + else + echo "*** (could not record: $1)" >&2 + echo "*** apply it as root instead: lfs-helper setid --run" >&2 + fi +} +args=(); skipped=""; _setid_mode="" # BUNDLED SHORT OPTIONS FIRST, or none of the rules below see what they are # looking for. Everything here matched `-d` only as a whole argument, and @@ -3589,7 +4135,16 @@ while [ $# -gt 0 ]; do -m|--mode) case "$2" in [2467][0-7][0-7][0-7]) # setuid/setgid/sticky requested - echo "*** install -m $2 (using 755 instead; set the bit as root if you want it)" >&2 + # WRITE DOWN WHAT WAS REFUSED -- the chmod wrapper does, + # and this is the same refusal. shadow installs su with + # `install -m 4755`, so the bit was dropped here and + # nothing recorded it; months later `su -` failed with + # setgid: Operation not permitted + # which names neither su's mode nor its owner. The + # targets are known after the parse loop, so the mode is + # remembered here and written at the end. + _setid_mode="$2" + echo "*** install -m $2 (using 755 instead; apply it as root: lfs-helper setid --run)" >&2 args+=(-m 755); shift 2 ;; *) args+=("$1" "$2"); shift 2 ;; esac ;; @@ -3655,7 +4210,32 @@ case "$_dest" in exit 0 fi ;; esac -exec "$real" "${args[@]}" +# RECORD THE REFUSED BIT AGAINST THE FILE THAT ACTUALLY LANDED. +# +# Only after the install succeeded: a request naming a file that was never +# installed sends root to chmod something that is not there. +if [ -z "$_setid_mode" ]; then + exec "$real" "${args[@]}" +fi +"$real" "${args[@]}" || exit $? +_f="$HOME/.pkgusr-setid.requests" +_last="${args[$((${#args[@]} - 1))]}" +_srcs2=(); _n2=${#args[@]} +for ((_i = 0; _i < _n2 - 1; _i++)); do + case "${args[$_i]}" in -*) continue ;; esac + if [ "$_i" -gt 0 ]; then + case "${args[$((_i - 1))]}" in -m|--mode|-o|--owner|-g|--group|-t|-S) continue ;; esac + fi + _srcs2+=("${args[$_i]}") +done +if [ -d "$_last" ]; then + for _s in ${_srcs2[@]+"${_srcs2[@]}"}; do + _sid_record "chmod $_setid_mode ${_last%/}/$(basename "$_s")" + done +else + _sid_record "chmod $_setid_mode $_last" +fi +exit 0 EOF # A SERVICE ACCOUNT IS ROOT'S JOB. @@ -3706,6 +4286,21 @@ EOF #!/bin/bash # Allow everything except setuid/setgid, which a package user must not set. real=/usr/bin/chmod + +# CHECK BEFORE REDIRECTING. `printf >> file 2>/dev/null || echo ...` does +# not work: the shell itself reports the failed redirection ("Permission +# denied") before printf ever runs, and 2>/dev/null on printf cannot +# suppress it -- which is how the recorder's own message came to fail an +# install that had finished. +_sid_record() { # _sid_record <line> + local f="$HOME/.pkgusr-setid.requests" + if [ -w "$f" ] || { [ ! -e "$f" ] && [ -w "$HOME" ]; }; then + printf '%s\n' "$1" >> "$f" + else + echo "*** (could not record: $1)" >&2 + echo "*** apply it as root instead: lfs-helper setid --run" >&2 + fi +} me="$(id -un)" for a in "$@"; do case "$a" in @@ -3720,7 +4315,8 @@ for a in "$@"; do # its setgid bit at install time. A refusal nobody records is a # decision that has to be re-derived from its consequences. echo "*** chmod $* (skipped: refuses to set setuid/setgid; do it as root if needed)" >&2 - printf 'chmod %s\n' "$*" >> "$HOME/.pkgusr-setid.requests" 2>/dev/null + # A RECORDER THAT CANNOT WRITE IS A NOTE, NOT A FAILURE. + _sid_record "chmod $*" exit 0 ;; esac done @@ -4174,7 +4770,7 @@ is_never_claimed() { } _drop_other_packages_files() { - local owner="$1" p cur + local owner="$1" p cur _sp _tgt _to while IFS= read -r p; do [ -n "$p" ] || continue # a shared index or a generated cache is nobody's file @@ -4184,6 +4780,32 @@ _drop_other_packages_files() { && [ "$cur" != UNKNOWN ] && user_exists "$cur"; then continue # another package installed this fi + # CURRENT OWNERSHIP IS NOT PROOF OF PROVENANCE (1.14.310). + # + # The test above asks who owns the file NOW, and that is circular: + # once the lease had handed /usr/bin/su and six /usr/sbin/iptables* + # links to p_webkitgtk, they read as "the owner's own" and stayed in + # its manifest -- which is the list the NEXT lease reads. The + # manifest fed the lease and the lease fed the manifest, and the + # damage repeated on every rebuild. Two facts break the loop, and + # neither depends on who holds the file today: + # + # * setid.conf names the package that installs a setid file. + # * a symlink is installed by whoever installed its target. + if _sp="$(_setid_pkg_for "$p" 2>/dev/null)" \ + && [ -n "$_sp" ] && [ "$_sp" != "$(unprefix_pkg_user "$owner")" ]; then + continue + fi + if [ -L "$p" ]; then + _tgt="$(readlink -f "$p" 2>/dev/null)" + if [ -n "$_tgt" ] && [ "$_tgt" != "$p" ]; then + _to="$(stat -c %U "$_tgt" 2>/dev/null)" + if [ -n "$_to" ] && [ "$_to" != "$owner" ] && [ "$_to" != root ] \ + && [ "$_to" != UNKNOWN ] && user_exists "$_to"; then + continue + fi + fi + fi printf '%s\n' "$p" done } @@ -4416,13 +5038,26 @@ _claim_candidates() { # <owner> <manifest> ; paths on stdout printf '%s\n' "$r$p" done < "$man" > "$tmp" # stat in batches: a path that does not exist simply produces no line - xargs -d '\n' -r -a "$tmp" stat -c '%U %n' -- 2>/dev/null | awk -v o="$owner" ' + # A SETUID FILE IS NEVER CLAIMED. + # + # ~ > su - + # su: Authentication service cannot retrieve authentication info + # + # This ran before EVERY phase, configure included, and shadow's manifest + # lists /usr/bin/su: "gave 'p_shadow' 16 path(s)" was su, passwd, chage + # and the rest going 4711 root -> 755 p_shadow, because chown clears the + # bit. Nothing put them back -- the lease only sees a file someone + # ELSE owns, and after this p_shadow owned them. Root's setid files are + # the lease's to hand over: it copies them aside, restores them on + # failure and applies setid.conf on success. + xargs -d '\n' -r -a "$tmp" stat -c '%U %A %n' -- 2>/dev/null | awk -v o="$owner" ' { - u = $1 - i = index($0, " ") - path = substr($0, i + 1) + u = $1; a = $2 + i = index($0, " "); rest = substr($0, i + 1) + j = index(rest, " "); path = substr(rest, j + 1) if (u == o) next if (u != "root" && u != "lfs" && u != "UNKNOWN") next + if (substr(a, 4, 1) ~ /[sS]/ || substr(a, 7, 1) ~ /[sS]/) next print path }' rm -f "$tmp" @@ -4467,8 +5102,30 @@ _claim_earlier_stages() { detail "# claiming files from earlier stages of $owner ..." local chownbin; chownbin="$(_real_tool chown)" local batch; batch="$(mktemp)" || return 0 + # AN EARLIER STAGE IS EARLIER THAN THIS PACKAGE'S OWN INSTALLS. + # + # Chapter 7 builds gettext, perl, Python ... as root under the same names + # chapter 8 uses, so the step's OWN manifest holds root-owned leftovers + # the first package-user build must take over. After that first + # successful install, a root-owned entry in its own manifest is not a + # leftover -- it is something recorded by mistake: LLVM's listed + # /usr/bin/lfs, /usr/bin/packagemanager and /etc/pam.d/su_u_user (root + # wrote them during its hours-long install window), and this step + # "gave 'p_llvm' 13 path(s)", making a script root runs on every su + # writable by p_llvm. The -tmp/-pass/libstdcpp stages are always earlier. + local _home_done="" + _home_done="$(pkgusr_home_for "$(unprefix_pkg_user "$owner")" 2>/dev/null)" + local _installed_before=0 + [ -n "$_home_done" ] && { [ -s "$_home_done/.pkgusr-installed" ] || [ -s "$_home_done/VERSION" ]; } \ + && _installed_before=1 for base in $seen; do man="$MANIFESTS/$base.files" + if [ "$_installed_before" = 1 ]; then + case "$base" in + *-tmp|*-pass1|*-pass2|libstdcpp|libstdc++) ;; + *) continue ;; + esac + fi if [ -s "$man" ]; then _claim_candidates "$owner" "$man" > "$batch" if [ -s "$batch" ]; then @@ -4641,6 +5298,410 @@ EOF return 0 } +# --------------------------------------------------------------------------- +# THE LEASE: files a phase will overwrite that someone else owns. (1.14.263) +# +# The old way was fail, read the log, repair, retry. Every tool words the +# refusal differently -- cmake, meson, python, a bash redirect -- and each +# needed its own parser. Shadow's configure died on +# line 153: /etc/pam.d/su: Permission denied +# which no parser recognised, so the round granted the DIRECTORY, which was +# never the problem, and stopped. And the setid pass and the 1.14.261 +# take-back were two halves of one cycle: root gets the file for its bit, +# the next install cannot replace it, a repair hands it back. +# +# So, around every install/configure phase, as root: +# before copy each file the phase will write that another account owns, +# with its owner and mode, and hand it to the installer. COPY, +# never move: the phase itself starts with `su - p_<pkg>`, and +# su, its pam file and unix_chkpwd must exist at that moment. +# failure put every copy back, owner and mode included, and read it back. +# success a setid.conf file gets its policy; a file the phase did not +# change goes back to its owner; a file it rewrote belongs to the +# installer (the rule everywhere else). +# +# WHICH FILES, before anything runs: +# * install/all/update: the package's own manifest -- ROOT-owned entries +# only. Those are files it installed that root took for a setid bit. +# * configure/all/update: the paths configure_pkg writes, read from the +# script (script_write_targets). +# A root-owned file is leased ONLY when this package's manifest claims it. +# The script is editable by the package user; without that rule, one edited +# line could hand it /etc/shadow. Files of non-package accounts (the main +# user, u_*) are never leased. The first install of a file another package +# owns cannot be known in advance, so the repair loop stays as the fallback. +# --------------------------------------------------------------------------- +LEASES="$STATE/leases" + +# script_write_targets <script> [function ...] -> absolute paths, one a line +# +# What a shell function WRITES, read from its text: redirect targets +# (> >> and tee), the destination of install/cp/mv/ln, the files of sed -i. +# A `for VAR in a b c` loop expands ${VAR}/$VAR; any other expansion, a +# glob or a brace is dropped rather than guessed. Comments and heredoc +# BODIES are skipped -- a heredoc body is data, and "prose is not code". +script_write_targets() { + local script="$1"; shift + [ -r "$script" ] || return 1 + [ $# -gt 0 ] || set -- configure_pkg + local fn + for fn in "$@"; do + sed -n "/^${fn}() {/,/^}/p" "$script" + done | awk ' + function emit(p, n, i, x, v, out) { + gsub(/^["\047]|["\047]$/, "", p) + if (p !~ /^\//) return + if (p ~ /[*?[{}]/) { + # ${VAR} is the one brace we understand + if (p !~ /\$\{[A-Za-z_][A-Za-z_0-9]*\}/) return + tmp = p; gsub(/\$\{[A-Za-z_][A-Za-z_0-9]*\}/, "", tmp) + if (tmp ~ /[*?[{}]/) return + } + if (p ~ /\$/) { + for (v in lists) { + if (index(p, "${" v "}") || p ~ ("\\$" v "([^A-Za-z_0-9]|$)")) { + n = split(lists[v], x, " ") + for (i = 1; i <= n; i++) { + out = p + gsub("\\$\\{" v "\\}", x[i], out) + gsub("\\$" v, x[i], out) + if (out !~ /\$/) emit(out) + } + return + } + } + return + } + if (p ~ /^\/(dev|proc|sys|tmp|run)(\/|$)/) return + print p + } + # skip a heredoc body until its delimiter + hd != "" { d = $0; if (strip) sub(/^\t+/, "", d); if (d == hd) hd = ""; next } + /^[[:space:]]*#/ { next } + { + buf = buf $0 + if (sub(/\\$/, "", buf)) next + line = buf; buf = "" + # the opener line still counts; its BODY does not + if (match(line, /<<-?[[:space:]]*["\047]?[A-Za-z_][A-Za-z_0-9]*["\047]?/)) { + h = substr(line, RSTART, RLENGTH) + strip = (h ~ /^<<-/) + gsub(/^<<-?[[:space:]]*|["\047]/, "", h) + hd = h + line = substr(line, 1, RSTART - 1) " " substr(line, RSTART + RLENGTH) + } + # redirect targets (not 2> / &> / <>, and not >&2) + l = line + while (match(l, /(^|[^0-9&<>])>>?[[:space:]]*[^[:space:];|&<>()]+/)) { + t = substr(l, RSTART, RLENGTH); l = substr(l, RSTART + RLENGTH) + sub(/^[^>]*>>?[[:space:]]*/, "", t) + if (t !~ /^&/) emit(t) + } + nseg = split(line, seg, /&&|\|\||;|\|/) + for (s = 1; s <= nseg; s++) { + # drop empty fields: a segment ending in a space splits into + # a trailing "" that would become the "last word" + nn = split(seg[s], ww, /[[:space:]]+/); n = 0 + for (j = 1; j <= nn; j++) if (ww[j] != "") w[++n] = ww[j] + k = 1; while (k <= n && (w[k] ~ /^(do|then|else|elif|!|sudo|\{|\()$/)) k++ + if (k > n) continue + cmd = w[k] + if (cmd == "for" && k + 2 <= n && w[k+2] == "in") { + v = w[k+1]; lists[v] = "" + for (j = k + 3; j <= n; j++) { x1 = w[j]; sub(/;$/, "", x1); if (x1 != "" && x1 != "do") lists[v] = lists[v] " " x1 } + continue + } + if (cmd ~ /^(install|cp|mv|ln)$/) { + last = ""; isdir = 0; tdir = "" + for (j = k + 1; j <= n; j++) { + if (w[j] ~ /^>/) break + if (w[j] == "-t" && j < n) { tdir = w[j+1]; j++; continue } + if (cmd == "install" && w[j] ~ /^-[A-Za-z]*d/) isdir = 1 + if (w[j] ~ /^-/) { + # install -m 644 / -o x / -g y take a value + if (cmd == "install" && w[j] ~ /^-[mog]$/) j++ + continue + } + last = w[j] + } + if (!isdir && tdir == "" && last != "") emit(last) + continue + } + if (cmd == "tee") { + for (j = k + 1; j <= n; j++) if (w[j] !~ /^-/) emit(w[j]) + continue + } + if (cmd == "sed") { + inplace = 0; havescript = 0 + for (j = k + 1; j <= n; j++) { + if (w[j] ~ /^-i/ || w[j] == "--in-place") { inplace = 1; continue } + if (w[j] == "-e" || w[j] == "-f") { havescript = 1; j++; continue } + if (w[j] ~ /^-/) continue + if (!havescript) { havescript = 1; continue } + if (inplace) emit(w[j]) + } + } + } + }' | awk '!seen[$0]++' +} + +cmd_script_targets() { + local script="${1:-}" + case "$script" in + ""|-h|--help) + say "lfs-helper script-targets <install script> [function ...]" + say "" + say "Lists the absolute paths the named functions write (default" + say "configure_pkg): redirects, tee, install/cp/mv/ln destinations," + say "sed -i. for-loops are expanded; other expansions are dropped." + say "This is the list the lease reads before a phase runs." + [ -n "$script" ]; return ;; + esac + shift + [ -r "$script" ] || die "script-targets: cannot read $script" + script_write_targets "$script" "$@" +} + +# the single policy table: setid.conf, wherever this install keeps it +_setid_conf() { + local c + for c in "${SETID_CONF:-}" "${PKGUSR_ETC:-/etc/pkgusr}/stacks/setid.conf" \ + "$(dirname "$(readlink -f "$0" 2>/dev/null || echo "$0")")/stacks/setid.conf"; do + [ -n "$c" ] && [ -r "$c" ] && { printf '%s\n' "$c"; return 0; } + done + return 1 +} + +# _setid_policy_for <path> -> "mode|owner|group" from setid.conf, or nothing +_setid_policy_for() { + local conf; conf="$(_setid_conf)" || return 1 + awk -F'|' -v p="$1" ' + /^[[:space:]]*(#|$)/ { next } + { f = $1; gsub(/[[:space:]]/, "", f) + if (f == p) { m = $2; o = $3; g = $4 + gsub(/[[:space:]]/, "", m); gsub(/[[:space:]]/, "", o); gsub(/[[:space:]]/, "", g) + print m "|" (o == "" ? "root" : o) "|" (g == "" ? "-" : g); found = 1; exit } } + END { exit !found }' "$conf" +} + +# _setid_pkg_for <path> -> the PACKAGE setid.conf names as the file's +# provider (the optional 5th column), or nothing. +# +# WHY A SETID FILE NEEDS AN OWNER'S NAME. The lease hands a root-owned file +# to a package user for the length of a phase, and one of the doors it may +# come through is "the file carries a setid bit". That door had no lock: +# ANY package whose time-window manifest happened to list /usr/bin/su -- and +# a webkitgtk build runs for hours, so its manifest listed whatever root +# wrote meanwhile -- was handed su, 0711 and no longer setuid, until its +# phase ended. `su` is broken for those hours and, if the build is +# interrupted, for good. +# +# Only shadow installs su. With the provider written down, the lease can +# say so: shadow may reclaim it, nobody else may borrow it. +_setid_pkg_for() { + local conf; conf="$(_setid_conf)" || return 1 + awk -F'|' -v p="$1" ' + /^[[:space:]]*(#|$)/ { next } + { f = $1; gsub(/[[:space:]]/, "", f) + if (f == p) { k = $5; gsub(/[[:space:]]/, "", k); print k; found = 1; exit } } + END { exit !found }' "$conf" +} + +_lease_dir() { printf '%s/%s\n' "$LEASES" "$(unprefix_pkg_user "$1")"; } + +# _lease_candidates <owner> <script> <phase> +_lease_candidates() { + local owner="$1" script="$2" phase="$3" pkg man + pkg="$(unprefix_pkg_user "$owner")" + man="$MANIFESTS/$pkg.files" + case "$phase" in + install|all|update) + if [ -r "$man" ]; then + # A PACKAGE RECLAIMS ITS OWN FILES. Every manifest entry + # someone else owns now -- root (a setid bit applied) or + # another package (a later install took it) -- is leased. + # Only root's took this path, so shadow's own file held by + # another package still failed on a bash redirect or cp, + # whose wording no repair parses. lease_files keeps the + # guards: package accounts and manifest-claimed root only. + grep -v '^[[:space:]]*$' "$man" | xargs -r -d '\n' stat -c '%U|%n' 2>/dev/null \ + | awk -F'|' -v o="$owner" '$1 != o { sub(/^[^|]*\|/, ""); print }' + fi ;; + esac + case "$phase" in + configure|all|update) script_write_targets "$script" configure_pkg ;; + esac +} + +# lease_files <owner> <script> <phase> (root; before the phase) +lease_files() { + local owner="$1" script="$2" phase="$3" dir ledger n=0 p u g m accounts _sp _lt + [ "$(id -u)" = 0 ] || return 0 + case "$phase" in install|configure|all|update) ;; *) return 0 ;; esac + user_exists "$owner" || return 0 + dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" + # A LEDGER LEFT BEHIND means the last phase never got to settle (killed, + # power cut). Its copies are the only record of what was there: restore + # them first, then lease afresh. + if [ -s "$ledger" ]; then + note "# $owner: a lease from an interrupted run is still open -- restoring it first" + lease_restore "$owner" || { + warn "!! could not restore the old lease in $dir -- not leasing again" + return 1; } + fi + local man="$MANIFESTS/$(unprefix_pkg_user "$owner").files" + accounts="$(pkg_users_list 2>/dev/null)" + mkdir -p "$dir/files" 2>/dev/null && chmod 700 "$LEASES" "$dir" 2>/dev/null + : > "$ledger" || { warn "!! cannot write $ledger -- no lease"; return 1; } + local _cfg_targets="" + case "$phase" in + configure|all|update) _cfg_targets="$(script_write_targets "$script" configure_pkg 2>/dev/null)" ;; + esac + while IFS= read -r p; do + [ -n "$p" ] || continue + [ -e "$p" ] || [ -L "$p" ] || continue + [ -d "$p" ] && [ ! -L "$p" ] && continue + grep -qF "$p|" "$ledger" 2>/dev/null && continue + # NO SYMLINK IN THE WAY. If a directory on the path is a link a + # package user controls, the name the manifest vouches for is not + # the file that would be handed over. + if [ "$(cd "$(dirname "$p")" 2>/dev/null && pwd -P)/$(basename "$p")" != "$p" ]; then + detail "# $p goes through a symlinked directory -- left alone" + continue + fi + u="$(stat -c %U "$p" 2>/dev/null)" || continue + [ "$u" = "$owner" ] && continue + if [ "$u" = root ]; then + if ! grep -qxF "$p" "$man" 2>/dev/null; then + detail "# $p is root's and $owner's manifest does not claim it -- left alone" + continue + fi + # A MANIFEST CLAIM IS NOT ENOUGH FOR ROOT'S FILES. Manifests are + # built by time window, and a long phase swallowed what root + # wrote meanwhile -- LLVM's claimed /usr/bin/lfs and + # /etc/pam.d/su_u_user, and the lease handed them to p_llvm. A + # package's file is root's for two reasons only: a setid bit + # (the policy put it there) or its own configure commands write + # it. Anything else root owns is not the package's to borrow. + if [ -u "$p" ] || [ -g "$p" ] || _setid_policy_for "$p" >/dev/null 2>&1; then + # ...AND "IT IS SETID" IS NOT A REASON EITHER, unless the + # package is the one that installs it (1.14.310). + # + # This was the widest door in the lease and it had no lock. + # /usr/bin/su is root 4711; any package whose time-window + # manifest listed it -- webkitgtk's did, because `cfg + # shadow-pam` ran during its build -- came through here, and + # `chown` to the package user clears the setuid bit as a + # side effect the kernel never reports. Three times over + # three months the answer to "why is su owned by + # p_webkitgtk and 0711" was this line. + # + # setid.conf names the provider (5th column). shadow may + # reclaim its own su; for anyone else a setid file is the + # system's, not a package's. + _sp="$(_setid_pkg_for "$p" 2>/dev/null)" + if [ -z "$_sp" ] || [ "$_sp" != "$(unprefix_pkg_user "$owner")" ]; then + warn "# $p is setuid/setgid and ${_sp:-no package} installs it -- not leasing it to $owner" + continue + fi + elif ! printf '%s\n' "$_cfg_targets" | grep -qxF "$p"; then + detail "# $p is root's, not setid and not written by $owner's configure -- left alone" + continue + fi + elif ! printf '%s\n' "$accounts" | grep -qxF "$u"; then + detail "# $p belongs to $u, not a package account -- left alone" + continue + fi + g="$(stat -c %G "$p")"; m="$(stat -c %a "$p")" + n=$((n + 1)) + cp -a "$p" "$dir/files/$n" 2>/dev/null || { + warn "!! could not copy $p aside -- not leasing it"; continue; } + printf '%s|%s|%s|%s|%s\n' "$p" "$m" "$u" "$g" "$n" >> "$ledger" + if chown -h "$owner:$owner" "$p" 2>/dev/null; then + detail "# leased $p ($m $u:$g) to $owner for this phase" + else + warn "!! could not hand $p to $owner" + fi + done < <(_lease_candidates "$owner" "$script" "$phase") + [ -s "$ledger" ] || rm -rf "$dir" 2>/dev/null + return 0 +} + +# lease_restore <owner> -- failure: every copy back, owner and mode, read back +lease_restore() { + local owner="$1" dir ledger p m u g n bad=0 got want + dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" + [ -s "$ledger" ] || return 0 + while IFS='|' read -r p m u g n; do + [ -n "$p" ] || continue + rm -f "$p" 2>/dev/null + if cp -a "$dir/files/$n" "$p" 2>/dev/null; then + chown -h "$u:$g" "$p" 2>/dev/null + [ -L "$p" ] || chmod "$m" "$p" 2>/dev/null + fi + got="$(stat -c '%a %U:%G' "$p" 2>/dev/null)" + want="$m $u:$g" + [ -L "$p" ] && { got="${got#* }"; want="$u:$g"; } + if [ "$got" = "$want" ] && cmp -s "$dir/files/$n" "$p" 2>/dev/null \ + || { [ -L "$p" ] && [ "$(readlink "$p")" = "$(readlink "$dir/files/$n")" ]; }; then + detail "# restored $p ($want)" + else + warn "!! $p did not come back as it was ($got, wanted $want)" + warn " the original is still at $dir/files/$n" + bad=1 + fi + done < "$ledger" + [ "$bad" = 0 ] && rm -rf "$dir" + return "$bad" +} + +# lease_settle <owner> -- success +lease_settle() { + local owner="$1" dir ledger p m u g n pol pm po pg + dir="$(_lease_dir "$owner")"; ledger="$dir/ledger" + [ -s "$ledger" ] || return 0 + while IFS='|' read -r p m u g n; do + [ -n "$p" ] || continue + if [ ! -e "$p" ] && [ ! -L "$p" ]; then + note "# $p was leased and the phase removed it" + continue + fi + if pol="$(_setid_policy_for "$p")"; then + IFS='|' read -r pm po pg <<<"$pol" + [ "$pg" = "-" ] && pg="$(stat -c %G "$p")" + # _setid_apply is root:<group> + mode, read back; the owner + # column is root for every setid line, and anything else is + # not a setid policy this code should apply + if [ "$po" = root ]; then + _setid_apply "$pm" "$pg" "$p" >/dev/null || warn "!! $p: setid.conf policy did not apply" + continue + fi + fi + # A SYMLINK IS COMPARED BY WHERE IT POINTS, NOT BY `cmp` (1.14.310). + # + # `cp -a` copies a symlink AS a symlink, so the copy in the lease + # directory carries the same *relative* target -- and relative to + # $LEASES/<pkg>/files, not to /usr/sbin. /usr/sbin/iptables -> + # xtables-nft-multi copied aside is a dangling link, `cmp` cannot + # open it, and a failed compare reads here as "the phase rewrote + # it". So every leased symlink was declared the package's own and + # kept -- which is how six /usr/sbin/iptables* links, made by `cfg + # iptables-nft`, ended up owned by p_webkitgtk and written into its + # pkg.lst. lease_restore already compared links with readlink; this + # is the same test, on the other side of the branch. + if { [ -L "$p" ] && [ "$(readlink "$p")" = "$(readlink "$dir/files/$n")" ]; } \ + || { [ ! -L "$p" ] && cmp -s "$dir/files/$n" "$p" 2>/dev/null; }; then + # the phase did not change it: the lease was not needed + chown -h "$u:$g" "$p" 2>/dev/null + [ -L "$p" ] || chmod "$m" "$p" 2>/dev/null + detail "# $p unchanged -- back to $u:$g" + else + say "# $p rewritten by $owner -- it owns it now (was $u:$g)" + fi + done < "$ledger" + rm -rf "$dir" +} + report_setid_requests() { # Printed after the phase, by root, because only root can act on it. local owner="${1:-}" home f @@ -4648,6 +5709,32 @@ report_setid_requests() { home="$(pkgusr_home_for "$owner" 2>/dev/null)" f="$home/.pkgusr-setid.requests" [ -s "$f" ] || return 0 + # APPLY THEM, DO NOT RECITE THEM. + # + # This function runs AS ROOT, right after the install -- the one moment + # when the thing that is missing can simply be done. It printed nine + # chmod lines for shadow and left `su` non-setuid until somebody typed + # them, which is the same "warn and walk away" that + # /etc/pam.d/su_u_user and the mode of that file were both guilty of. + # + # `lfs-helper setid --run` is the routine that already knows how: it + # applies each recorded request and READS THE MODE BACK. Call it, and + # report what it did rather than what the reader should do. + if [ "$(id -u)" = 0 ]; then + warn "" + warn "This package asked for setuid/setgid bits, which a package user" + warn "cannot set -- applying them now (root):" + while IFS= read -r line; do + [ -n "$line" ] && warn " $line" + done < "$f" + if cmd_setid --run; then + warn " applied. 'lfs-helper setid' lists anything still outstanding." + else + warn " SOME COULD NOT BE APPLIED -- see above; the requests are kept." + warn " retry at any time: lfs-helper setid --run" + fi + return 0 + fi warn "" warn "This package asked for setuid/setgid bits, which a package user" warn "cannot set. Nothing else refuses them; the binaries are installed" @@ -4657,7 +5744,14 @@ report_setid_requests() { while IFS= read -r line; do [ -n "$line" ] && warn " $line" done < "$f" - : > "$f" 2>/dev/null || true + warn "" + warn "All of them at once, at any later time:" + warn " lfs-helper setid # what is outstanding" + warn " lfs-helper setid --run # apply it" + # NOT CLEARED HERE. Printing once and emptying the file made this a + # notice you had to be watching for: six hours of build output later + # there was no way to ask what had been refused. `setid --run` is what + # clears a line, once it has applied it and read the result back. } run_phase_as() { @@ -4696,6 +5790,13 @@ run_phase_as() { case "$phase" in all|update) clear_phases "$track" ;; esac fi mkdir -p "$(dirname "$log")" 2>/dev/null || true + # THE LEASE (1.14.263): files this phase will overwrite that root or + # another package owns are handed to the installer first, and settled or + # restored below. Root writes anything, so a root run needs none. + local _leased=0 + if [ "$as_root" != 1 ] && have_su && user_exists "$owner"; then + lease_files "$owner" "$script" "$phase" && _leased=1 + fi if [ "$as_root" = 1 ] || ! have_su || ! user_exists "$owner"; then ( [ -n "$cwd" ] && cd "$cwd"; eval "env $envpass bash '$script' '$phase'" ) \ 2>&1 | tee "$log" @@ -4705,6 +5806,22 @@ run_phase_as() { su - "$owner" -c "env $envpass bash '$script' '$phase'" 2>&1 | tee "$log" rc=${PIPESTATUS[0]} fi + # A SINGLE-PHASE RUN KNOWS WHICH PHASE FAILED: ITS OWN. + # + # LAST_FAILED_PHASE was only set in the all|update branch below, and it + # is cleared at the top of this function -- so the FIRST retry (which + # runs just `install`) left it empty, and cmd_build's second round fell + # back to `$phase` = all. opencv's round 2 therefore re-cloned and + # rebuilt a tree that had reached 100%, to redo an install that needed + # one more directory grant. Rounds 1 and 2 printed the difference: + # round 1 ... (phase install only -- the build is done) + # round 2 ... (phase all) + if [ "$rc" != 0 ]; then + case "$phase" in + all|update) ;; # parsed from the log below + *) LAST_FAILED_PHASE="$phase" ;; + esac + fi if [ -n "$track" ]; then local _p case "$phase" in @@ -4774,7 +5891,23 @@ run_phase_as() { fi if [ "$rc" = 0 ] && [ -n "$log" ] && [ -f "$log" ]; then local _swallowed="" - _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^install: cannot create|: Permission denied$" \ + # CMAKE SAYS IT DIFFERENTLY, AND IT SAID IT. + # + # CMake Error at CMakeLists.txt:6 (set): + # There is an unterminated variable reference. + # -- Configuring incomplete, errors occurred! + # + # ...and the run printed "llvm: done (all phases complete)". The + # pattern below knows make and ninja; cmake fails before either runs, + # so nothing matched and a configure error became a finished build. + # That is the exact failure this check was written for, in another + # tool's words. + # + # And the `ninja` in the first alternative never matched anything: + # ninja prints `ninja: build stopped: subcommand failed.` with no + # `***`, so it needs its own branch -- the one place this check was + # supposed to work, and did not. + _swallowed="$(grep -m1 -E "^(make(\[[0-9]+\])?|ninja): \*\*\* .*(Error [0-9]+|build stopped)|^ninja: build stopped|^install: cannot create|: Permission denied$|^CMake Error|^-- Configuring incomplete|^configure: error:|^meson\.build:[0-9]+:[0-9]+: ERROR:|^ERROR: Unhandled python|^error: could not compile|^fatal: could not read Username|^fatal: repository .* not found|^fatal: Authentication failed|BackendUnavailable: Cannot import|^ModuleNotFoundError: No module named" \ "$log" 2>/dev/null)" if [ -n "$_swallowed" ]; then warn "!! this phase exited 0, but its log says otherwise:" @@ -4784,6 +5917,17 @@ run_phase_as() { rc=1 fi fi + # SETTLE ONLY AFTER THE LOG HAS HAD ITS SAY: a zero exit with "Permission + # denied" in the log is a failure, and a failure puts every leased file + # back. + if [ "$_leased" = 1 ]; then + if [ "$rc" = 0 ]; then + lease_settle "$owner" + else + lease_restore "$owner" \ + || warn "!! some leased files could not be restored -- see above" + fi + fi return "$rc" } @@ -4867,6 +6011,37 @@ _version_from_script() { | sed 's/^.*-\([0-9][0-9.]*\)$/\1/' } +# WHAT IS INSTALLED IS WHAT FINISHED -- ON EVERY PATH THAT FINISHES. +# +# `update` reads the installed version from .pkgusr-installed. It was +# written in ONE branch: a single-phase run completing the last phase. A +# normal `--phase all` install (every packagemanager install) took the other +# branch and never wrote it; and VERSION, beside it, parsed only +# name_version="..." while the generator writes name_version='...'. So no +# record was written on success, the reader fell back to install_last (staged +# BEFORE the build), and a failed build advanced the version. 1.14.273 then +# seeded the marker with the OLD version before a build -- and with nothing +# updating it on success, LLVM 22.1.8 installed and kept reading 21.1.8, and +# every stack run planned it again. +# +# Called after every phase has succeeded, whichever branch got there. +_record_installed() { # <name> <owner> <script> + local name="$1" owner="$2" script="$3" nv home + nv="$(sed -n "s/^name_version=['\"]\([^'\"]*\)['\"].*/\1/p" "$script" 2>/dev/null | head -n1)" + [ -n "$nv" ] || return 0 + home="$(pkgusr_home_for "$owner" 2>/dev/null)" + [ -n "$home" ] && [ -d "$home" ] || home="$(pkgusr_home_for "$name" 2>/dev/null)" + [ -n "$home" ] && [ -d "$home" ] || return 0 + { + echo "$nv" + echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" + [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" + } > "$home/VERSION" + printf '%s\n' "$nv" > "$home/.pkgusr-installed" + # benign: notes in the package's own home, rewritten every install + real_chown "$owner:$owner" "$home/VERSION" "$home/.pkgusr-installed" 2>/dev/null || true +} + _write_pkgusr_info() { local name="$1" owner="$2" source="${3:-lfs}" version="${4:-}" local home="$(pkgusr_home_for "$name")" @@ -4997,6 +6172,8 @@ _package_already_owns_files() { cmd_build() { need_root local name="" phase="all" force=0 run_tests=0 jobs="" auto_fix=1 + local auto_modules="${PM_AUTO_MODULES:-0}" + local _AUTO_MODULES_LOG="" local _force_stage=auto script_override="" as_root=0 while [ $# -gt 0 ]; do case "$1" in @@ -5013,6 +6190,7 @@ cmd_build() { --tests) run_tests=1; shift ;; --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; --no-auto-fix) auto_fix=0; shift ;; + --auto-modules) auto_modules=1; shift ;; # A CONFIGURATION STEP THAT NEEDS ROOT (1.14.79) told people to run # lfs-helper build fcron --phase configure --force --as-root # and cmd_build did not take --as-root: the flag existed on @@ -5200,7 +6378,9 @@ cmd_build() { else pkgusr_ready || die "run 'lfs-helper init-pkgusr --run' first" owner="$(pkg_owner_name "$name")" - cmd_add_user "$owner" + # an existing account is the normal case for every build after the + # first -- say nothing about it (`lfs-helper add-user` still does) + _QUIET_EXISTING=1 cmd_add_user "$owner" stagedir="$(pkgusr_home_for "$owner")" staged="$stagedir/install_$name" # ALREADY THERE IS ALREADY STAGED. @@ -5283,7 +6463,20 @@ cmd_build() { mkdir -p "$(dirname "$GRANTED")" 2>/dev/null || true : > "$GRANTED" 2>/dev/null || true create_service_accounts_from_script "$staged" - : > "$(pkgusr_home_for "$owner" 2>/dev/null)/.pkgusr-setid.requests" 2>/dev/null || true + # ROOT MUST NOT LEAVE A FILE THE PACKAGE USER CANNOT WRITE. + # + # This truncation runs as root, so the ledger came out root:root -- and + # then the install wrapper, running as the package user, could not + # append to it: + # /usr/lib/pkgusr/install: .pkgusr-setid.requests: Permission denied + # The swallowed-failure check saw "Permission denied" and failed the + # build, correctly, over the RECORDER rather than the package. Every + # root write to this file hands it back. + _sidf="$(pkgusr_home_for "$owner" 2>/dev/null)/.pkgusr-setid.requests" + if [ -n "${_sidf:-}" ]; then + : > "$_sidf" 2>/dev/null || true + chown "$owner:$owner" "$_sidf" 2>/dev/null || true + fi say "# building $name ..." local rc=0 make_wrappers @@ -5417,6 +6610,32 @@ cmd_build() { && ! printf '%s' "$envpass" | grep -q "MAKEFLAGS="; then envpass="$envpass MAKEFLAGS='$MAKEFLAGS'" fi + # THREE BUILD SYSTEMS, THREE VARIABLES, ONE NUMBER. + # + # make reads MAKEFLAGS; cmake reads CMAKE_BUILD_PARALLEL_LEVEL; cargo + # reads CARGO_BUILD_JOBS; ninja reads NOTHING and defaults to nproc+2, + # which is why machine.conf patches its -j into the script. Setting + # MAKEFLAGS alone limited one of the four, so `--jobs 13` was not a + # limit -- a cmake or rust package still ran wide and got OOM-killed. + # READ THE VALUE THAT WILL ACTUALLY BE PASSED. + # + # This looked at ${MAKEFLAGS:-} -- the AMBIENT one, in root's shell -- + # while the number lives in /etc/pkgusr/build.env, which is already in + # $envpass by now. In a chroot root shell MAKEFLAGS is unset, so + # `MAKEFLAGS=-j13` in build.env produced no CARGO_BUILD_JOBS at all and + # rust built on every core. + _jn="$jobs" + [ -n "$_jn" ] || _jn="$(printf '%s' "$envpass" \ + | grep -oE "MAKEFLAGS='?-j ?[0-9]+" \ + | grep -oE '[0-9]+' | head -n1)" + [ -n "$_jn" ] || _jn="$(printf '%s' "${MAKEFLAGS:-}" \ + | grep -oE '\-j ?[0-9]+' | grep -oE '[0-9]+' | head -n1)" + if [ -n "$_jn" ]; then + printf '%s' "$envpass" | grep -q "CMAKE_BUILD_PARALLEL_LEVEL=" \ + || envpass="$envpass CMAKE_BUILD_PARALLEL_LEVEL='$_jn'" + printf '%s' "$envpass" | grep -q "CARGO_BUILD_JOBS=" \ + || envpass="$envpass CARGO_BUILD_JOBS='$_jn'" + fi # Keep pip out of the package user's ~/.local. PIP_USER=0 stops it # installing there, and PYTHONNOUSERSITE=1 stops it *seeing* an existing # per-user copy and concluding the requirement is "already satisfied" -- @@ -5530,9 +6749,10 @@ cmd_build() { # and it then reports a diff that is simply wrong. The sort and the # comparison have to agree; pinning either one alone guarantees they do not. newfiles="$(LC_ALL=C comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after"))" - newdirs="$(LC_ALL=C comm -13 <(printf '%s\n' "$before_dirs") <(snapshot_dirs))" + local _after_dirs; _after_dirs="$(snapshot_dirs)" # one walk, used twice + newdirs="$(LC_ALL=C comm -13 <(printf '%s\n' "$before_dirs") <(printf '%s\n' "$_after_dirs"))" printf '%s\n' "$after" > "$SNAPSHOT" - printf '%s\n' "$(snapshot_dirs)" > "$SNAPDIRS" + printf '%s\n' "$_after_dirs" > "$SNAPDIRS" # Everything this build wrote: created OR modified since the stamp. # Always work out what the build TOUCHED, even when we are not going to @@ -5569,20 +6789,23 @@ cmd_build() { # knew that; this one did not. Same rule, one more place. local _bld="${BUILD_ROOT:-$r/build}" say "# scanning the tree for what this step touched (a fresh tree takes a moment) ..." - touched_dirs="$(find "${r:-/}" -xdev -type d -newer "$stamp" \ - -not -path "$SRCROOT/*" \ - -not -path "$r/sources/*" -not -path "$_bld/*" \ - -not -path "$_bld" -not -path "$r/dev/*" \ - -not -path "$r/proc/*" -not -path "$r/sys/*" \ - -not -path "$r/run/*" -not -path "$r/tmp/*" \ - 2>/dev/null | sort -u)" - touched="$(find "${r:-/}" -xdev \( -type f -o -type l \) -newer "$stamp" \ - -not -path "$SRCROOT/*" \ - -not -path "$r/sources/*" -not -path "$_bld/*" \ - -not -path "$r/dev/*" \ - -not -path "$r/proc/*" -not -path "$r/sys/*" \ - -not -path "$r/run/*" -not -path "$r/tmp/*" \ - 2>/dev/null | sort -u)" + # ONE pruned walk for both lists (it was two unpruned ones). A + # pruned directory is reported only as itself, as before: $SRCROOT, + # sources, dev, proc, sys, run and tmp when newer; never the build root. + local _tw + _tw="$(find "${r:-/}" -xdev \ + \( -path "$_bld" \) -prune \ + -o \( -path "$SRCROOT" -o -path "$r/sources" -o -path "$r/dev" \ + -o -path "$r/proc" -o -path "$r/sys" -o -path "$r/run" \ + -o -path "$r/tmp" \) -type d -newer "$stamp" -printf 'd %p\n' -prune \ + -o \( -path "$SRCROOT" -o -path "$r/sources" -o -path "$r/dev" \ + -o -path "$r/proc" -o -path "$r/sys" -o -path "$r/run" \ + -o -path "$r/tmp" \) -prune \ + -o -newer "$stamp" \( -type d -printf 'd %p\n' \ + -o \( -type f -o -type l \) -printf 'f %p\n' \) \ + 2>/dev/null)" + touched_dirs="$(printf '%s\n' "$_tw" | sed -n 's/^d //p' | sort -u)" + touched="$(printf '%s\n' "$_tw" | sed -n 's/^f //p' | sort -u)" fi rm -f "$stamp" @@ -5590,14 +6813,38 @@ cmd_build() { # Record what was touched regardless of whether ownership changed -- a # config step that rewrites /etc/hosts installs no NEW file but has very # much done its job. - { [ -f "$man" ] && cat "$man"; printf '%s\n' "$newfiles" - printf '%s\n' "$touched"; } \ + # A PACKAGE USER CANNOT HAVE WRITTEN A ROOT-OWNED FILE. + # + # Attribution is by time window. LLVM's install phase ran for hours + # (its compile is generated into install_pkg), and in that window root + # ran `make install` of these tools and `cfg app-users` -- so + # /usr/bin/lfs, /usr/bin/packagemanager and /etc/pam.d/su_u_user went + # into LLVM's manifest. The lease then treated them as LLVM's own + # root-owned files and handed su_u_user, a script root runs on every + # su, to p_llvm for the length of each install round. + # When the phase ran AS the package user, it could only have written + # files it can write: whatever is root's now was not its doing. (A + # file it rewrote under a lease is its own by now -- lease_settle ran + # before this scan.) Root-run phases and chown-after builds still record + # root files, which is what they produce. + local _root_ok=0 + { [ "${as_root:-0}" = 1 ] || [ "${chown_after:-0}" = 1 ]; } && _root_ok=1 + { [ -f "$man" ] && cat "$man" + { printf '%s\n' "$newfiles"; printf '%s\n' "$touched"; } \ + | while IFS= read -r _tf; do + [ -n "$_tf" ] || continue + if [ "$_root_ok" != 1 ] && [ "$(stat -c %u "$_tf" 2>/dev/null)" = 0 ]; then + continue + fi + printf '%s\n' "$_tf" + done; } \ | grep -v '^$' | sort -u \ | _drop_other_packages_files "$owner" > "$man.tmp" \ && mv "$man.tmp" "$man" { [ -f "$mand" ] && cat "$mand"; printf '%s\n' "$newdirs"; } \ | grep -v '^$' | sort -u > "$mand.tmp" && mv "$mand.tmp" "$mand" say "# tracked $(wc -l < "$man") file(s) for '$name'" + _udev_reload_if_rules_changed "$man" # ...and the owner-scanned record too, once ownership makes it meaningful. # One door: cmd_record_install decides which strategy applies (item 6). @@ -5628,7 +6875,7 @@ cmd_build() { # Directories that belong to the install group are shared infrastructure # (root:install) and must stay that way -- only take the ones this # package created itself. - local n=0 p cur skipped_foreign=0 + local n=0 p cur skipped_foreign=0 _sp _lt _lo while IFS= read -r p; do [ -n "$p" ] || continue # NEVER take a file that already belongs to another package user. @@ -5647,6 +6894,61 @@ cmd_build() { skipped_foreign=$((skipped_foreign+1)) continue fi + # A PACKAGE USER CANNOT HAVE WRITTEN A ROOT-OWNED FILE -- HERE TOO. + # + # The manifest above already drops root-owned paths when the phase + # ran AS the package user (_root_ok). This loop did not: it walked + # the RAW lists again, so a file the manifest correctly refused was + # chowned anyway. One rule, applied in one of the two places that + # needed it. + if [ "$_root_ok" != 1 ] && [ "$cur" = root ]; then + skipped_foreign=$((skipped_foreign+1)) + continue + fi + # CHOWN DESTROYS A SETUID BIT, AND NOTHING SAYS SO. + # + # Attribution is by TIME WINDOW, and a webkitgtk build runs for + # hours. Anything that touched /usr/bin/su in that window -- + # `lfs-helper setid`, a shadow reinstall, `cfg shadow-pam` -- made + # it "newer than the stamp", so this loop took a root-owned file + # that was fair game by every rule above and chowned it: + # -rwx--x--x 1 p_webkitgtk p_webkitgtk /bin/su + # The kernel clears setuid/setgid on chown, so one command changed + # the owner AND removed the bit that makes su work. A file + # carrying either bit is never this sweep's to take, whoever ran + # the phase: the modes in setid.conf are the system's, not a + # package's. + case "$(stat -c %a "$p" 2>/dev/null)" in + [1-7][0-7][0-7][0-7]) + warn "# not taking $p -- it is setuid/setgid ($(stat -c %a "$p")); chown would clear that" + skipped_foreign=$((skipped_foreign+1)) + continue ;; + esac + # ...AND THE SAME TWO PROVENANCE RULES THE MANIFEST USES + # (1.14.310): a setid file belongs to the package setid.conf + # names, and a symlink belongs to whoever owns its target. The + # mode test above cannot see either -- `stat -c %a` on a symlink + # reports the link's own 777, so no link has ever looked setuid + # to it, and a file whose setuid bit a previous chown already + # cleared does not look setuid either. + if _sp="$(_setid_pkg_for "$p" 2>/dev/null)" \ + && [ -n "$_sp" ] && [ "$_sp" != "$name" ]; then + warn "# not taking $p -- setid.conf says $_sp installs it" + skipped_foreign=$((skipped_foreign+1)) + continue + fi + if [ -L "$p" ]; then + _lt="$(readlink -f "$p" 2>/dev/null)" + if [ -n "$_lt" ] && [ "$_lt" != "$p" ]; then + _lo="$(stat -c %U "$_lt" 2>/dev/null)" + if [ -n "$_lo" ] && [ "$_lo" != "$owner" ] && [ "$_lo" != root ] \ + && [ "$_lo" != UNKNOWN ] && user_exists "$_lo"; then + warn "# not taking $p -- it points at $_lo's $_lt" + skipped_foreign=$((skipped_foreign+1)) + continue + fi + fi + fi real_chown -h "$owner:$owner" "$p" 2>/dev/null && n=$((n+1)) done <<< "$(printf '%s\n%s\n' "$newfiles" "$touched" | grep -v '^$' | sort -u)" while IFS= read -r p; do @@ -5684,8 +6986,31 @@ cmd_build() { # that stops because the USER must do something -- "mount your EFI System # Partition first" -- must never be answered by handing directories away. local _perm_ish=0 - grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system" \ + # "file INSTALL cannot": CMake wraps "Permission denied" over two lines. + grep -qiE "permission denied|operation not permitted|cannot (create|make|remove|change|touch)|couldn't copy|read-only file system|file INSTALL cannot" \ "$log" 2>/dev/null && _perm_ish=1 + # A BOOK COMMAND THAT WORKS ONCE. + # + # libtiff's page ends by moving its html docs into a versioned + # directory. The first install creates + # /usr/share/doc/libtiff-4.7.1/tiff; the second cannot: + # mv: cannot overwrite '/usr/share/doc/libtiff-4.7.1/tiff': Directory not empty + # Nothing is wrong with the system -- the destination is this package's + # own output from last time, and a REINSTALL means replacing it. The + # gate did not recognise it, so 38 rebuilds stopped on a directory the + # package itself had put there. + # + # Only ever the exact path the message names, only when it belongs to + # THIS package, and only for a phase that is reinstalling. + # THE GATE DECIDES; THE LOOP ACTS. + # + # This used to call _clear_stale_dest here. It removed the directory -- + # correctly, and said so -- and then the retry loop ran zero rounds, + # because the loop only counts what IT fixed. The repair happened and + # nothing re-ran the phase, so the package failed anyway with the + # obstacle already gone. Detect here, act in the loop. + grep -qE "cannot overwrite '[^']+'|: Directory not empty" "$log" 2>/dev/null \ + && _perm_ish=1 # "cannot make directory ...: No such file or directory" is not a # permission problem and granting will never fix it -- but it IS # automatically fixable, so it must reach the retry loop. @@ -5728,6 +7053,18 @@ cmd_build() { && _fixed_something=1 fi create_missing_dirs_from_log "$log" "$owner" && _fixed_something=1 + # the package's own leftovers from a previous install of the same + # version: removing them IS the fix, and it counts as one, so the + # phase is retried + _clear_stale_dest "$log" "${owner:-}" && _fixed_something=1 + # a destination file another package owns, in a sticky directory: + # handing it over is the fix, and it counts as one + _take_cmake_dest "$log" "${owner:-}" && _fixed_something=1 + if [ "${auto_modules:-0}" = 1 ]; then + [ -n "$_AUTO_MODULES_LOG" ] \ + || _AUTO_MODULES_LOG="$(mktemp -t pm-auto-modules.XXXXXX)" + _install_missing_module "$log" && _fixed_something=1 + fi # a file another package owns, which THIS package is installing: # ownership follows the installer (granting cannot fix EPERM) takeover_files_from_log "$log" "$owner" && _fixed_something=1 @@ -5768,11 +7105,21 @@ cmd_build() { # THAT phase; the sources and the build tree are untouched and still # there, which is exactly why the grant is all that was missing. local _retry_phase="$phase" - if [ "$phase" = all ]; then - case "${LAST_FAILED_PHASE:-}" in - install|configure) _retry_phase="$LAST_FAILED_PHASE" ;; - esac - fi + # `update` IS A COMPOUND PHASE TOO. + # + # This mapped only `all`, so a failure inside `update` retried + # `update` -- and poppler's retry, after a grant for + # /usr/share/gir-1.0, re-unpacked the tarball and re-ran cmake from + # scratch. 145 files were already installed; the one that needed + # doing again was the install step. The log says which inner phase + # failed (`!! phase 'install' FAILED`); both compound phases should + # use it. + case "$phase" in + all|update) + case "${LAST_FAILED_PHASE:-}" in + install|configure) _retry_phase="$LAST_FAILED_PHASE" ;; + esac ;; + esac say "" if [ "$_retry_phase" != "$phase" ]; then say "# retrying $name (phase $_retry_phase only -- the build is done), round $_round ..." @@ -5866,6 +7213,42 @@ cmd_build() { fail "! auto-repair: $([ "$auto_fix" = 1 ] && echo enabled || echo DISABLED)," \ "gate: $([ "$_perm_ish" = 1 ] && echo "recognised" || echo "did not recognise this failure")," \ "rounds run: ${_round:-0}" + # A PYTHON BUILD BACKEND IS A PACKAGE, AND ITS NAME IS NOT THE + # MODULE'S. numpy died on + # BackendUnavailable: Cannot import 'mesonpy' + # and the page that provides `mesonpy` is called meson_python -- not + # guessable, and the traceback is 60 lines of pip internals with the + # one useful word at the bottom. + local _pymod _pyprov + _pymod="$(grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ + "$log" 2>/dev/null | head -n1 | cut -d"'" -f2)" + if [ -n "$_pymod" ]; then + case "$_pymod" in + mesonpy) _pyprov="meson_python" ;; + setuptools_scm) _pyprov="setuptools_scm" ;; + Cython|cython) _pyprov="Cython" ;; + numpy) _pyprov="NumPy" ;; + flit_core) _pyprov="flit_core" ;; + hatchling) _pyprov="hatchling" ;; + scikit_build_core) _pyprov="scikit-build-core" ;; + poetry.core|poetry_core) _pyprov="poetry-core" ;; + *) _pyprov="" ;; + esac + if [ -n "${_AUTO_MODULES_LOG:-}" ] && [ -s "$_AUTO_MODULES_LOG" ]; then + fail "! installed on the way (--auto-modules), not asked for:" + while IFS= read -r _am; do fail "! $_am"; done < "$_AUTO_MODULES_LOG" + fi + fail "! a python module is missing, not a permission: '$_pymod'" + if [ -n "$_pyprov" ]; then + fail "! the BLFS page that provides it is '$_pyprov':" + fail "! packagemanager install $_pyprov --run --recursive" + else + fail "! find the page that provides it:" + fail "! packagemanager search $_pymod" + fi + fail "! then rebuild this package. A plain \`install\` builds ONE" + fail "! package: --recursive is what builds what it needs." + fi local _md_now _md_now="$(missing_dirs_from_log "$log" 2>/dev/null | tr '\n' ' ')" if [ -n "$_md_now" ]; then @@ -6653,27 +8036,13 @@ EOF fi fi - # Record what is installed, in the package user's own home. - # - # The scripts are regenerated from the book whenever it moves on, so their - # name_version tells you what the BOOK has, not what is on disk. Without a - # record of the latter, "is this up to date?" is unanswerable. - local _nv _home - _nv="$(sed -n 's/^name_version="\(.*\)"$/\1/p' "$s" 2>/dev/null | head -n1)" - _home="$(pkgusr_home_for "$owner")" - if [ -n "$_nv" ] && [ -d "$_home" ]; then - { - echo "$_nv" - echo "# installed $(date '+%Y-%m-%d %H:%M:%S') by lfs-helper" - [ -n "${_book_of_script:-}" ] && echo "# from: $_book_of_script" - } > "$_home/VERSION" - # benign: a note in the package's own home, rewritten every install - real_chown "$owner:$owner" "$_home/VERSION" 2>/dev/null || true - fi - + # Record what is installed, in the package user's own home: VERSION and + # .pkgusr-installed, both in _record_installed (below), for the branch + # that finished every phase. if [ "$phase" = "all" ] || [ "$phase" = "update" ]; then _write_pkgusr_info "$name" "$owner" "${LFS_PKG_SOURCE:-lfs}" \ "$(_version_from_script "$staged")" + _record_installed "$name" "$owner" "$s" mark_done "$name" ok "# $name: done" report_granted_groups @@ -6696,7 +8065,18 @@ EOF # running again on every build-all. [ -f "$MANIFESTS/$name.files" ] && \ _mark_adopted "$name" "$(_manifest_size "$MANIFESTS/$name.files")" + _record_installed "$name" "$owner" "$s" ok "# $name: done (all phases complete)" + # AND WHAT IT INSTALLED ON THE WAY, on the SUCCESS path too: the + # whole point of reporting it is that the run changed more than + # the package that was asked for, and a successful run is + # exactly when nobody goes looking. + if [ -n "${_AUTO_MODULES_LOG:-}" ] && [ -s "$_AUTO_MODULES_LOG" ]; then + say "# installed on the way (--auto-modules), not asked for:" + while IFS= read -r _am; do say "# $_am"; done \ + < "$_AUTO_MODULES_LOG" + say "# they are real package users now: packagemanager info <name>" + fi report_granted_groups fi fi @@ -6933,6 +8313,153 @@ cmd_rename_group() { ok "renamed: $old -> $new" } +# REGROUP ONE DIRECTORY. +# +# /etc/pam.d p_bash-completion:nimgnu_bash-completion drwxrwxr-x +# +# A directory granted through the wrong group -- here one named after a +# package that has nothing to do with pam -- cannot be fixed with chgrp +# alone: every member of the old group keeps write access to it, and the +# packages that DO install there have to join the new one. So: +# +# 1. who installs INTO it: every package whose manifest or pkg.lst lists a +# direct child, every account owning an entry in it, and its owner +# 2. the new group's name, asked the way a grant asks (owner-named, +# directory-named, or typed) +# 3. which members of the OLD group still need it: those installing into +# another directory that keeps the old group (or owning one) +# 4. the plan, then yes/no -- nothing changes before the answer +# +# The old group itself is left as it is (even when empty): deleting a group +# is a separate decision, and the plan says when it has become unused. +cmd_regroup_dir() { + need_root + local dir="${1:?usage: lfs-helper regroup-dir <directory>}" + dir="$(readlink -f "$dir")" || die "no such directory: $1" + [ -d "$dir" ] || die "not a directory: $dir" + local owner old + owner="$(stat -c %U "$dir")"; old="$(stat -c %G "$dir")" + [ "$owner" = root ] && die "$dir is owned by root -- that is not a package's directory to share" + + say "$dir ($(stat -c '%A %U:%G' "$dir"))" + + # 1. installers into THIS directory (direct children only) + local installers + installers="$( + { for man in "$MANIFESTS"/*.files; do + [ -f "$man" ] || continue + if awk -v d="$dir" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$man"; then + pkg_owner_name "$(basename "${man%.files}")" + fi + done + for _r in "$PKGUSR_ROOT" "$CFGUSR_ROOT"; do + [ -d "$_r" ] || continue + for _l in "$_r"/*/pkg.lst; do + [ -f "$_l" ] || continue + awk -v d="$dir" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$_l" \ + && basename "$(dirname "$_l")" + done + done + find "$dir" -mindepth 1 -maxdepth 1 -printf '%u\n' 2>/dev/null + echo "$owner" + } | while IFS= read -r _u; do + [ -n "$_u" ] || continue + # package and config accounts only: never root, a login or a u_ account + case "$_u" in + "${PKGUSR_PREFIX:-p_}"?*|"${CFGUSR_PREFIX:-cfg_}"?*) ;; + *) continue ;; + esac + getent passwd "$_u" >/dev/null 2>&1 && echo "$_u" + done | sort -u)" + + # 2. the new name + local by_owner by_dir new ans + by_owner="$(sanitise_group_name "$(collector_group_for "$owner")")" + by_dir="$(sanitise_group_name "${COLLECTOR_PREFIX}_$(basename "$dir")")" + say "" + say " installs into it: $(echo $installers)" + say " current group: $old (members: $(getent group "$old" | cut -d: -f4))" + say "" + say " New group for $dir:" + say " 1) ${C_B}${by_dir}${C_OFF} (named after the directory)" + say " 2) ${C_B}${by_owner}${C_OFF} (named after its owner)" + say " or type another name, or q to cancel" + printf ' [1/2/name/q] (default 1): ' > /dev/tty + read -e -r ans < /dev/tty || ans=q + case "$ans" in + ""|1) new="$by_dir" ;; + 2) new="$by_owner" ;; + q|Q) say "cancelled -- nothing changed"; return 0 ;; + *) new="$(sanitise_group_name "$ans")" + [ -n "$new" ] || die "'$ans' is not a usable group name -- nothing changed" ;; + esac + [ "$new" = "$old" ] && { say "$dir already has group $new -- nothing to do"; return 0; } + + # 3. who still needs the OLD group, for the directories that keep it + local others keep="" drop="" m d2 needs + # EVERY real filesystem, not just /'s: a directory of $old on another + # mount that is not seen here would cost its installers their membership. + local _mnts + _mnts="$(findmnt -rno TARGET -t ext2,ext3,ext4,xfs,btrfs,f2fs,jfs,reiserfs 2>/dev/null | tr '\n' ' ')" + [ -n "$_mnts" ] || _mnts="/" + others="$(find $_mnts -xdev \( -path "$SRCROOT" -o -path /proc -o -path /sys -o -path /dev \ + -o -path /run -o -path /tmp \) -prune \ + -o -type d -group "$old" -print 2>/dev/null | sort -u | grep -vxF "$dir")" + for m in $(getent group "$old" | cut -d: -f4 | tr ',' ' '); do + needs="" + while IFS= read -r d2; do + [ -n "$d2" ] || continue + if [ "$(stat -c %U "$d2")" = "$m" ] \ + || find "$d2" -mindepth 1 -maxdepth 1 -user "$m" -print -quit 2>/dev/null | grep -q .; then + needs="$d2"; break + fi + for man in "$MANIFESTS/$(unprefix_pkg_user "$m").files" "$PKGUSR_ROOT/$m/pkg.lst" "$CFGUSR_ROOT/$m/pkg.lst"; do + [ -f "$man" ] || continue + if awk -v d="$d2" '{ p = $0; sub(/\/[^\/]*$/, "", p); if (p == d) { f = 1; exit } } END { exit !f }' "$man"; then + needs="$d2"; break 2 + fi + done + done <<< "$others" + if [ -n "$needs" ]; then keep="$keep $m"; else drop="$drop $m"; fi + done + + # 4. the plan + say "" + say " PLAN" + say " chgrp $new $dir (mode g+rwx kept for the installers)" + group_exists "$new" && say " group $new exists (gid $(getent group "$new" | cut -d: -f3))" \ + || say " CREATE group $new (collector range)" + say " add to $new: $(echo $installers)" + [ -n "$drop" ] && say " remove from $old: $(echo $drop) (no other directory of $old is theirs)" + [ -n "$keep" ] && say " stay in $old: $(echo $keep) (they still install into a directory of $old)" + [ -z "$others" ] && say " $old is then on no directory at all -- delete it yourself if unused: groupdel $old" + printf ' apply? [y/N]: ' > /dev/tty + read -e -r ans < /dev/tty || ans=n + case "$ans" in [Yy]*) ;; *) say "cancelled -- nothing changed"; return 0 ;; esac + + ensure_group "$new" || die "could not create group $new -- nothing changed" + for m in $installers; do add_to_group "$m" "$new" || warn "could not add $m to $new"; done + real_chgrp "$new" "$dir" || die "chgrp $new $dir failed" + real_chmod g+rwx "$dir" 2>/dev/null + for m in $drop; do + gpasswd -d "$m" "$old" >/dev/null 2>&1 || warn "could not remove $m from $old" + done + # the records a later grant consults + local rel="${dir#${SNAP_ROOT%/}}" + if [ -f "$COLLECTOR_MAP" ]; then + grep -v "^$rel|" "$COLLECTOR_MAP" > "$COLLECTOR_MAP.tmp" 2>/dev/null; mv "$COLLECTOR_MAP.tmp" "$COLLECTOR_MAP" + fi + mkdir -p "$(dirname "$COLLECTOR_MAP")"; echo "$rel|$new" >> "$COLLECTOR_MAP" + for m in $installers; do + grep -qxF "$new|$m|$owner" "$GRANTED" 2>/dev/null || echo "$new|$m|$owner" >> "$GRANTED" + done + say "" + ok "$dir is now $(stat -c '%A %U:%G' "$dir")" + say " $new: $(getent group "$new" | cut -d: -f4)" + say " $old: $(getent group "$old" | cut -d: -f4)" + say " (membership changes apply to NEW logins and builds; a running shell keeps its groups)" +} + # Which package installed this path? # # The manifests are the record of who installed what, so a file that should @@ -8387,6 +9914,21 @@ cmd_verify() { ok "every file has a real owner" fi + # ---- 3c. setid bits the build refused --------------------------------- # + # A CHECK MUST SAY WHAT IS MISSING RATHER THAN WAIT TO BE ASKED. These + # were printed once, hours earlier, in the middle of a build log. + local _sid=0 _sf _su + while IFS= read -r _su; do + _sf="$(pkgusr_home_for "$_su" 2>/dev/null)/.pkgusr-setid.requests" + [ -s "$_sf" ] && _sid=$((_sid + $(grep -c . "$_sf"))) + done < <(pkg_users_list) + if [ "$_sid" -gt 0 ]; then + warn "$_sid setuid/setgid bit(s) were refused during the build and never applied" + warn " what and why: lfs-helper setid" + warn " apply them: lfs-helper setid --run" + problems=$((problems+1)) + fi + detail "# checking homes, the build user and the uid order ..." # ---- 4b. the passes the dead copy had and this one did not ------------- # # homes owned by their own user, the build user's leftovers, and the uid @@ -8782,12 +10324,28 @@ cmd_pm_install() { say "Install $user_name" say "-------------------------------" say "install file: $install_script" - if ! assume_yes; then + # "CHECK THAT THIS IS THE RIGHT ACCOUNT" IS A FIRST-INSTALL QUESTION. + # + # It is worth asking once: a typo in the account name on a FIRST install + # scatters a package across the wrong home and is tedious to undo. It is + # worth nothing on an update -- the account exists, it already owns this + # package's files, and the answer was settled the day it was installed. + # Asking anyway turned every `update <pkg> --run` into a keypress, and a + # prompt people press through is a prompt that has stopped protecting + # anything. + local _known=0 + if user_exists "$user_name"; then + [ -s "$(pkgusr_home_for "$user_name")/pkg.lst" ] && _known=1 + [ -f "$(pkgusr_home_for "$user_name")/.pkgusr-installed" ] && _known=1 + fi + if ! assume_yes && [ "$_known" != 1 ]; then warn "First check the install file for: link, name, wrong commands" warn "check that \"$user_name\" is the right account" printf 'Press enter to continue ' read -r -n1 _ say "" + elif [ "$_known" = 1 ]; then + say "# $user_name is an existing package with files recorded -- not asking" fi # Phased scripts carry their metadata (install_groups, nimgnu_groups_dir, @@ -8854,6 +10412,24 @@ cmd_pm_install() { warn " used -- move your edits into $base (or delete the old file)" fi fi + # RECORD WHAT IS INSTALLED BEFORE OVERWRITING THE RECORD. + # + # `update` reads the installed version from .pkgusr-installed, written + # only after a successful install -- and, for a package built before that + # marker existed, from install_last, which is replaced HERE, before the + # build. LLVM 21.1.8 (no marker) failed on 22.1.8's patch, and the next + # plan dropped it as current: spirv-llvm-translator then asked for LLVM 22 + # and found 21.1.8. So the first time through, the old install_last's + # version becomes the marker, and a failure leaves it there. + if [ ! -s "$home/.pkgusr-installed" ] && [ -f "$home/install_last" ] \ + && ! [ "$install_script" -ef "$home/install_last" ]; then + local _was + _was="$(sed -n "s/^name_version=['\"]\{0,1\}\([^'\"]*\).*/\1/p" "$home/install_last" 2>/dev/null | head -n1)" + if [ -n "$_was" ]; then + printf '%s\n' "$_was" > "$home/.pkgusr-installed" \ + && chown "$user_name:" "$home/.pkgusr-installed" 2>/dev/null || true + fi + fi if ! [ "$install_script" -ef "$home/install_last" ]; then rm -f "$home/install_last" 2>/dev/null cp "$install_script" "$home/install_last" @@ -9785,11 +11361,29 @@ EOF [ -z "${INPUTRC:-}" ] && [ -f /etc/inputrc ] && export INPUTRC=/etc/inputrc EOF + # THE LOCALE COMES FROM A FILE, NOT FROM A LINE YOU EDIT INTO A SCRIPT. + # + # This used to write `: # export LANG=en_US.UTF-8` -- commented out, on + # the grounds that an unset LANG is a choice. It is not a choice anyone + # makes: nothing else on the system sets LANG either, so every machine + # built this way sits in the C locale until somebody notices a terminal + # complaining ("foot: 'C' is not a UTF-8 locale"), and the only way to + # fix it is to hand-edit a generated file. BLFS's own version has the + # same shape and the same outcome -- a placeholder nobody fills in. + # + # So it reads /etc/locale.conf, which is where LFS puts the locale and + # what `cfg shell-env` writes from `lfs config locale`. With no + # locale.conf this does nothing, exactly as before. _write_if_absent "$d/i18n.sh" <<'EOF' -# Locale. Set LANG here (e.g. en_US.UTF-8); an unset LANG means the C -# locale, which is a choice, not an accident. -if [ -z "${LANG:-}" ] && command -v locale >/dev/null 2>&1; then - : # export LANG=en_US.UTF-8 +# written by lfs-helper (shell startup files) -- safe to replace +# Locale. LANG lives in /etc/locale.conf, one file for the whole system: +# echo 'LANG=en_US.UTF-8' > /etc/locale.conf +# `packagemanager stack services --only shell-env --run` writes it for you, +# from `lfs config locale`, and checks the locale really exists. +if [ -r /etc/locale.conf ]; then + . /etc/locale.conf + if [ -n "${LANG:-}" ]; then export LANG; fi + if [ -n "${LC_ALL:-}" ]; then export LC_ALL; fi fi export LESSCHARSET="${LESSCHARSET:-utf-8}" EOF @@ -10410,6 +12004,11 @@ Builds one package as its own package user, tracking every file it installs. expect some failures) --jobs N override MAKEFLAGS for this build --no-auto-fix do not grant collector-group access on a permission failure + --auto-modules on a missing python build backend, install the BLFS page + that provides it (--recursive) and retry. Never for a + module only a TEST needs (pytest, hypothesis, ...): a test + suite in an install phase is a script bug, not a missing + package. Off by default; PM_AUTO_MODULES=1 also works. EOF ;; strip) @@ -10470,10 +12069,21 @@ Ownership and permissions (the package-user model): lfs-helper verify check the tree against what the manifests say it should be lfs-helper verify --fix and repair what it can + lfs-helper setid [--run] setuid/setgid bits the build refused + (su, passwd, dbus-launch-helper, ...) lfs-helper fix-perms <name> --run grant one package the access it needs lfs-helper grant-dir <dir> <user> --run lfs-helper rename-group <old> <new> --run # fix a mistyped group name + lfs-helper regroup-dir <directory> # move a directory to another group (asks, shows the plan) give one user access to one directory + lfs-helper undone steps of the chroot build not yet done + lfs-helper sort-users --run sort passwd and group by id + lfs-helper check-login can root and the main user log in + + The chroot build's own steps -- init-accounts, establish-ownership, + seal-install-dirs, install-profile, repair-adoption -- are dispatched by + `build-all` and listed by `lfs-helper list`. They are not wrong to run by + hand, but nothing needs to. Collector groups (shared directories): lfs-helper export-groups [<file>] save the decisions for the next build @@ -10507,6 +12117,8 @@ Look around / tidy up: run one phase as the package user (the one runner both drivers use) lfs-helper find-user-site --run find pip installs hidden in ~/.local + lfs-helper script-targets <script> [fn ...] + the files a phase writes (the lease list) Any command: -h explains it. Dry run is the default; --run applies. State: $STATE @@ -10537,6 +12149,8 @@ case "${1:-}" in manifests) shift; cmd_manifests "$@" ;; fix-perms) shift; cmd_fix_perms "$@" ;; clean-sources) shift; cmd_clean_sources "$@" ;; + setid) shift; cmd_setid "$@" ;; + script-targets) shift; cmd_script_targets "$@" ;; find-user-site) shift; cmd_find_user_site "$@" ;; check-toolchain) shift; cmd_check_toolchain "$@" ;; verify) shift; cmd_verify "$@" ;; @@ -10550,6 +12164,7 @@ case "${1:-}" in which-package) shift; cmd_which_package "$@" ;; grant-dir) shift; cmd_grant_dir "$@" ;; rename-group) shift; cmd_rename_group "$@" ;; + regroup-dir) shift; cmd_regroup_dir "$@" ;; make-wrappers) shift; cmd_make_wrappers "$@" ;; wrapper-dir) shift; cmd_wrapper_dir "$@" ;; export-groups) shift; cmd_export_groups "$@" ;; diff --git a/lfs-kernel b/lfs-kernel index 5a4c91b..bcd52d9 100755 --- a/lfs-kernel +++ b/lfs-kernel @@ -7,6 +7,14 @@ # lfs-kernel latest the newest stable, once # lfs-kernel detect-esp print the EFI directory this would use # lfs-kernel verify [ver] check that everything is installed +# lfs-kernel firmware install what FIRMWARE_ONLY names, no kernel build +# lfs-kernel firmware --missing [--fix] [--all] +# firmware this machine asks for and has not got, +# from the boot log, this tree's modules and the +# config; --fix installs it at exactly the path +# the driver asked for, --all includes the files +# a module merely DECLARES (other chips' -- large +# and mostly unused) # # Settings: /etc/pkgusr/kernel.conf, INSTALLED by `make install` so it can be # edited before the first run. The build commands are in one block below, @@ -48,8 +56,25 @@ case "${1:-}" in printf '%s\n' "$(_detect_esp)"; exit 0 ;; verify) shift; _VERIFY=1; VERSION="${1:-}" ;; + firmware) + # firmware alone, with no kernel build: the blobs are a separate + # package and a missing one is a run-time problem, so fixing it must + # not mean rebuilding a kernel that is already installed and booted. + shift; _FWONLY=1; _FWARG=""; _FWFIX=0; _FWALL=0 + for _a in "$@"; do + case "$_a" in + --fix) _FWFIX=1 ;; + --all) _FWALL=1 ;; + --missing|missing) _FWARG="--missing" ;; + *) echo "!! lfs-kernel firmware: unknown argument '$_a'"; exit 1 ;; + esac + done + [ "$_FWFIX" = 1 ] || [ "$_FWALL" = 1 ] && _FWARG="--missing" ;; -h|--help) - sed -n '2,14p' "$0" | sed 's/^# \?//'; exit 0 ;; + # the slice is the header above, which grew: derive the end from + # the header itself rather than hard-coding a line number that a + # new command silently pushes out of range. + sed -n '2,/^# Settings:/p' "$0" | sed '$d' | sed 's/^# \?//'; exit 0 ;; "") ;; -*) echo "!! unknown option: $1 (try: lfs-kernel --help)"; exit 1 ;; *) VERSION="$1" ;; @@ -314,91 +339,6 @@ if [ "${_VERIFY:-0}" = 1 ]; then exit 1 fi -echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" - -# -------------------------------------------------------------------------- -# THE COMMANDS. Everything above chooses the version and checks the paths; -# everything below is the build, and is meant to be read. -# -------------------------------------------------------------------------- -install -d "$SRC" -if [ ! -d "$tree" ]; then - [ -f "$tarball" ] || { - echo "# fetching $MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" - wget -q --show-progress -O "$tarball.part" \ - "$MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" \ - && mv "$tarball.part" "$tarball" \ - || { rm -f "$tarball.part" - echo "!! download failed -- KERNEL_LIST=1 shows what exists"; exit 1; } - } - tar -xf "$tarball" -C "$SRC" || exit 1 - chown -R "$acct:$acct" "$tree" "$tarball" 2>/dev/null || true -fi -[ -d "$tree" ] || { echo "!! $tree is missing"; exit 1; } - -# boot logo (optional) -if [ -n "${BOOTLOGO:-}" ]; then - logo="$SRC/$(basename "$BOOTLOGO")" - [ -f "$logo" ] || { wget -qO "$logo" "$BOOTLOGO" && chown "$acct:$acct" "$logo"; } || logo="" - [ -n "$logo" ] && _as_pkg "cp -v '$logo' drivers/video/logo/logo_linux_clut224.ppm" -fi - -# .config -case "$CONFIG_FROM" in - running) _as_pkg "gunzip < /proc/config.gz > .config" || { - echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; - file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; - keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; - *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; -esac -# ---- refreshing the config for a NEW kernel ------------------------------ -# -# A config from one kernel is never complete for another: every version adds -# symbols the old file has no answer for. Something must decide them, and -# which something matters: -# -# olddefconfig take the DEFAULT for each new symbol, no questions. The -# right choice for an unattended run, and what this tool has -# always done -- but the defaults are upstream's, not yours. -# oldconfig ask about each new symbol, one at a time. Slow, and needs -# someone at the terminal, but nothing is decided silently. -# menuconfig open the editor and decide everything at once. -# none use the .config exactly as it is; `make` will then ask, or -# fail, depending on what is missing. -# -# It also says how many symbols the refresh had to answer, because "47 new -# symbols were set to their defaults" is worth knowing before a boot. -: "${CONFIG_REFRESH:=olddefconfig}" -_cfg_before="$(mktemp)"; cp "$tree/.config" "$_cfg_before" 2>/dev/null || true -case "$CONFIG_REFRESH" in - none) echo "# not refreshing the config (CONFIG_REFRESH=none)" ;; - olddefconfig|oldconfig|menuconfig|*config) - echo "# refreshing the config for $VERSION (make $CONFIG_REFRESH) ..." - _as_pkg "make $CONFIG_REFRESH" || exit 1 ;; - *) echo "!! CONFIG_REFRESH=$CONFIG_REFRESH is not understood"; exit 1 ;; -esac -if [ -s "$_cfg_before" ] && [ "$CONFIG_REFRESH" != none ]; then - _newsym="$(diff <(sort "$_cfg_before") <(sort "$tree/.config") 2>/dev/null \ - | grep -c '^>' || true)" - [ "${_newsym:-0}" -gt 0 ] \ - && echo "# $_newsym config line(s) differ from the config you started from" -fi -rm -f "$_cfg_before" - -# what a Wayland compositor needs; a warning, not a refusal -for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do - grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" -done -# Drivers that need firmware are no use without the driver. These are the -# ones a laptop misses most often; the firmware for each is in FIRMWARE_ONLY. -for opt in CONFIG_IWLWIFI:"Intel wireless (AX200 and friends)" \ - CONFIG_IWLMVM:"the AX200's own MVM firmware layer" \ - CONFIG_BT_HCIBTUSB:"USB Bluetooth (the AX200's other half)" \ - CONFIG_DRM_AMDGPU:"AMD graphics"; do - _o="${opt%%:*}"; _why="${opt#*:}" - grep -qE "^$_o=(y|m)" "$tree/.config" \ - || echo "!! $_o is not set -- no $_why" -done - # ---- firmware, as its own package ---------------------------------------- # # The drivers need it at RUN time and CONFIG_EXTRA_FIRMWARE needs it at BUILD @@ -426,6 +366,9 @@ _sparse_args() { out="$out $e" fi done + # WHENCE lists the flat-name links (see _fw_whence_links). Cone mode + # always checks out top-level files; pattern mode only what it is told. + [ -n "$mode" ] && out="$out /WHENCE" printf '%s' "${out# }" } @@ -447,7 +390,11 @@ _firmware_not_found() { # the question actually being asked. local _hits="" if [ -d "$src/.git" ]; then - _hits="$(git -C "$src" ls-files 2>/dev/null | grep -i -- "$stem" | head -12)" + _hits="$(_fw_index "$src" | grep -i -- "$stem" | head -12)" + if [ -n "$_FW_INDEX_ERR" ]; then + echo " (could not read the repository index, so this is what is on" + echo " disk only: $_FW_INDEX_ERR)" + fi fi [ -n "$_hits" ] || _hits="$(find "$src" -maxdepth 2 -iname "*$stem*" \ -printf '%P\n' 2>/dev/null | head -12)" @@ -466,6 +413,64 @@ _firmware_not_found() { fi } +# THE REGULATORY DATABASE IS NOT IN linux-firmware. +# +# cfg80211 asks the loader for regulatory.db, linux-firmware does not carry +# it, and no FIRMWARE_ONLY entry can conjure it -- worse, an entry naming it +# makes the whole firmware install FAIL, because a pattern matching nothing +# in the repository is an error. It is released on its own as +# wireless-regdb, and it belongs to the same package user as the rest of +# /lib/firmware: one directory, one owner. +REGDB_BASE="${REGDB_BASE:-https://www.kernel.org/pub/software/network/wireless-regdb}" +_install_regdb() { + local acct=p_linux-firmware home tar url ver work + case "${REGDB:-yes}" in yes|1|true) ;; *) return 0 ;; esac + [ -e /lib/firmware/regulatory.db ] && [ "${1:-}" != force ] && return 0 + id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 + home="$(getent passwd "$acct" | cut -d: -f6)" + # ASK, DO NOT ASSUME A VERSION: the directory listing names the releases, + # and a hard-coded date goes stale between them. + ver="$(wget -qO- "$REGDB_BASE/" 2>/dev/null \ + | sed -n 's/.*wireless-regdb-\([0-9.]*\)\.tar\.xz.*/\1/p' \ + | sort -V | tail -n1)" + if [ -z "$ver" ]; then + echo "!! could not read $REGDB_BASE/ -- no network, or the layout moved." + echo " Fetch a wireless-regdb tarball by hand, then:" + echo " su - $acct -c 'cp regulatory.db regulatory.db.p7s /lib/firmware/'" + return 1 + fi + url="$REGDB_BASE/wireless-regdb-$ver.tar.xz" + tar="$home/src/wireless-regdb-$ver.tar.xz" + install -d -o "$acct" -g "$acct" -m 755 "$home/src" 2>/dev/null + # CONFIRM THE DIRECTORY BEFORE DOWNLOADING INTO IT. `install -d -o` on + # an account that does not exist fails quietly, and the next line then + # reports a download failure for a URL that was never the problem. + [ -d "$home/src" ] || { + echo "!! $home/src does not exist and could not be created" + echo " (does the account $acct have that home? lfs-helper fix-home $acct --run)" + return 1; } + echo "# wireless-regdb $ver (the regulatory database, not in linux-firmware)" + [ -s "$tar" ] || su - "$acct" -c "wget -qO '$tar' '$url'" || { + echo "!! could not download $url"; rm -f "$tar"; return 1; } + work="$home/src/wireless-regdb-$ver" + su - "$acct" -c "cd '$home/src' && tar -xf '$tar'" || return 1 + [ -s "$work/regulatory.db" ] || { + echo "!! $tar unpacked without a regulatory.db in it"; return 1; } + install -d -m 755 /lib/firmware 2>/dev/null + [ "$(stat -c %U /lib/firmware 2>/dev/null)" = "$acct" ] \ + || chown "$acct:$acct" /lib/firmware 2>/dev/null + su - "$acct" -c "cp -a '$work/regulatory.db' '$work/regulatory.db.p7s' /lib/firmware/" \ + || { echo "!! could not install the database as $acct"; return 1; } + if [ -s /lib/firmware/regulatory.db ]; then + echo "# installed /lib/firmware/regulatory.db ($(stat -c %s /lib/firmware/regulatory.db) bytes)" + echo "# /lib/firmware/regulatory.db.p7s" + packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true + return 0 + fi + echo "!! /lib/firmware/regulatory.db is absent after the copy" + return 1 +} + _install_firmware() { local acct="p_linux-firmware" src="" only="${FIRMWARE_ONLY:-}" id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 @@ -538,14 +543,49 @@ _install_firmware() { su - "$acct" -c "cp -a '$src'/* /lib/firmware/" || return 1 else for d in $only; do + # AN ENTRY FOR SOMETHING THIS REPOSITORY DOES NOT HAVE IS NOT AN + # ERROR WHEN WE KNOW WHERE IT DOES LIVE. `regulatory.db*` in + # FIRMWARE_ONLY failed the entire firmware install, and the + # message blamed the pattern. + _skip=0 + while IFS='|' read -r _ep _epkg _eurl; do + [ -n "${_ep:-}" ] || continue + case "$_ep" in + $d) _skip=1 ;; + esac + done < <(_fw_elsewhere) + if [ "$_skip" = 1 ]; then + echo "# '$d' is not in linux-firmware -- it is wireless-regdb;" + echo "# installing that instead (REGDB=${REGDB:-yes})" + _install_regdb || echo "# ...which did not work; see above" + continue + fi # a pattern matches files at the top level; a name is a directory case "$d" in *[*?[]*) - if ! su - "$acct" -c "ls -d $src/$d >/dev/null 2>&1"; then + # A PATTERN MATCHES AT ANY DEPTH, AND A LINK NAME COUNTS. + # + # linux-firmware moved iwlwifi-*.ucode into intel/iwlwifi/ + # and keeps the flat names the driver asks for as `Link:` + # lines in WHENCE. The sparse checkout (gitignore rules) + # already fetched them at their new depth; this looked + # only at the top level and stopped the kernel build. + local _hits _links + _hits="$(cd "$src" && find . -name .git -prune -o \( -type f -o -type l \) -name "$d" -print 2>/dev/null | sed 's|^\./||')" + _links="$(_fw_whence_link_names "$src" | while IFS= read -r _ln; do + case "$_ln" in $d) echo "$_ln" ;; esac; done)" + if [ -z "$_hits" ] && [ -z "$_links" ]; then _firmware_not_found "$src" "$d"; return 1 fi - echo "# installing firmware: $d" - su - "$acct" -c "cp -a $src/$d /lib/firmware/" || return 1 ;; + echo "# installing firmware: $d ($(printf '%s\n' "$_hits" | grep -c .) file(s)$([ -n "$_links" ] && printf ', %s flat link(s) from WHENCE' "$(printf '%s\n' "$_links" | grep -c .)"))" + # --parents, and from inside the checkout: `cp -a + # $src/intel/iwlwifi /lib/firmware/` dropped the intel/ + # and landed as /lib/firmware/iwlwifi, where nothing + # looks for it. An entry keeps its path. + if [ -n "$_hits" ]; then + printf '%s\n' "$_hits" \ + | su - "$acct" -c "cd '$src' && xargs -r -d '\\n' cp -a --parents -t /lib/firmware/" || return 1 + fi ;; *) # ...the same help for a NAME. Only the pattern case # searched, so `iwlwifi` (a directory that does not @@ -553,14 +593,637 @@ _install_firmware() { # the reader exactly where the pattern had. [ -e "$src/$d" ] || { _firmware_not_found "$src" "$d"; return 1; } echo "# installing firmware: $d" - su - "$acct" -c "cp -a '$src/$d' /lib/firmware/" || return 1 ;; + su - "$acct" -c "cd '$src' && cp -a --parents '$d' /lib/firmware/" || return 1 ;; esac done fi + [ -n "$only" ] && _fw_whence_links "$src" "$acct" + _install_regdb || true packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true return 0 } +# WHENCE's `Link: <name> -> <target>` lines are how linux-firmware keeps the +# names drivers ask for after moving a blob (iwlwifi-cc-a0-77.ucode -> +# intel/iwlwifi/iwlwifi-cc-a0-77.ucode). Its own copy-firmware.sh makes +# them; a selective install has to as well. The target is relative to the +# link's directory. +_fw_whence_link_names() { + [ -r "$1/WHENCE" ] || return 0 + sed -n 's/^Link:[[:space:]]*\([^[:space:]]*\)[[:space:]]*->.*/\1/p' "$1/WHENCE" +} +_fw_whence_links() { + local src="$1" acct="$2" name target made=0 + [ -r "$src/WHENCE" ] || { echo "# (no WHENCE in the checkout -- flat firmware names not linked)"; return 0; } + while read -r name target; do + [ -n "$name" ] && [ -n "$target" ] || continue + # only for blobs that were installed, and never over a real file + [ -e "/lib/firmware/$(dirname "$name")/$target" ] || continue + [ -e "/lib/firmware/$name" ] || [ -L "/lib/firmware/$name" ] && continue + su - "$acct" -c "mkdir -p '/lib/firmware/$(dirname "$name")' && ln -s '$target' '/lib/firmware/$name'" \ + && made=$((made + 1)) + done < <(sed -n 's/^Link:[[:space:]]*\([^[:space:]]*\)[[:space:]]*->[[:space:]]*\([^[:space:]]*\).*/\1 \2/p' "$src/WHENCE") + [ "$made" -gt 0 ] && echo "# linked $made flat firmware name(s) from WHENCE" + return 0 +} + +# REPORT WHAT YOU FOUND, NOT WHAT YOU EXPECTED. +# +# FIRMWARE_ONLY="... intel" installed intel/, and the AX200 still had no +# firmware, because the driver asks for a FLAT path: +# iwlwifi 0000:03:00.0: Direct firmware load for iwlwifi-cc-a0-77.ucode failed +# iwlwifi 0000:03:00.0: no suitable firmware found! +# Four releases went into guessing linux-firmware's layout from the outside. +# The running kernel already knows the answer -- it printed the exact path it +# wanted -- and the checkout's git index knows where that file lives. So ask +# both, instead of reasoning about directory names. +_fw_requests() { + # every firmware path THIS boot asked for and did not get + dmesg 2>/dev/null | sed -n \ + -e 's/.*Direct firmware load for \([^ ]*\) failed.*/\1/p' \ + -e 's/.*firmware: failed to load \([^ ]*\).*/\1/p' \ + | sed 's/[,:]$//' | sort -u +} + +# ASK THE INDEX -- AND NOTICE WHEN THE ASKING FAILED. +# +# `git -C "$src" ls-files 2>/dev/null` returned nothing and rc 128, because +# the repository belongs to p_linux-firmware and this runs as root: +# fatal: detected dubious ownership in repository at '...' +# The empty output was then reported as a FACT -- "linux-firmware does not +# have a file by that name at all" -- about a file that is right there. An +# error hidden behind 2>/dev/null becomes a confident wrong answer. +_FW_INDEX_ERR="" +_fw_index() { # _fw_index <repo> [<basename to match>] + local src="$1" want="${2:-}" out="" err="" + _FW_INDEX_ERR="" + if [ -d "$src/.git" ]; then + # -c safe.directory: the checkout is the firmware account's and this + # runs as root, which git refuses to touch without being told. + # the scratch file goes in /tmp, not in the repository: leaving + # untracked files in someone else's checkout is not this tool's business + local tmp; tmp="$(mktemp)" + # ls-TREE, not ls-files. With a sparse index git collapses every + # excluded directory into one entry, so `ls-files` answered `intel/` + # and every nested blob under it looked absent -- 56 files reported + # as "this checkout has no file by that name" while they were all + # right there. ls-tree reads the commit, which no sparse setting + # touches. + err="$(git -c "safe.directory=$src" -C "$src" ls-tree -r --name-only HEAD 2>&1 >"$tmp")" + if [ -n "$err" ] && [ ! -s "$tmp" ]; then + _FW_INDEX_ERR="$err" + rm -f "$tmp" + return 1 + fi + out="$(cat "$tmp")" + rm -f "$tmp" + else + out="$(cd "$src" && find . -type f -printf '%P\n' 2>/dev/null)" + fi + if [ -n "$want" ]; then + printf '%s\n' "$out" | grep -- "\(^\|/\)$want$" + else + printf '%s\n' "$out" + fi +} + +_fw_repo() { # the linux-firmware checkout, or empty + local home + home="$(getent passwd p_linux-firmware 2>/dev/null | cut -d: -f6)" + case "$FIRMWARE_FROM" in + copy:*) printf '%s' "${FIRMWARE_FROM#copy:}" ;; + *) [ -n "$home" ] && printf '%s' "$home/src/linux-firmware" ;; + esac +} + +# WHICH KERNEL'S MODULES? In a chroot the running kernel is the host's, so +# `uname -r` is the wrong answer. Prefer the version this tool is configured +# for, then whatever is installed under /lib/modules, and SAY which one. +_fw_modver() { + local v + for v in "${VERSION:-}" "$(uname -r)"; do + [ -n "$v" ] && [ -d "/lib/modules/$v" ] && { printf '%s' "$v"; return 0; } + done + ls -1d /lib/modules/*/ 2>/dev/null | sed 's|.*/modules/||; s|/$||' \ + | sort -V | tail -n1 +} + +# Firmware that is NOT in linux-firmware, so "not in the checkout" is the +# right answer and updating the checkout will never help. +_fw_elsewhere() { + cat <<'ELSE' +regulatory.db|wireless-regdb|https://kernel.org/pub/software/network/wireless-regdb/ +regulatory.db.p7s|wireless-regdb|https://kernel.org/pub/software/network/wireless-regdb/ +ELSE +} + +# WHAT WILL BE ASKED FOR -- from sources that also work in a chroot. +# +# dmesg answers "what did the kernel that is running ask for", which in a +# chroot is the host's boot: it reported one missing file for 7.2.0 while the +# kernel of interest was 7.2.4, and said nothing about the wireless at all. +# The tree can be asked directly instead: the modules it installed declare +# their firmware (modinfo -F firmware), and the hardware is right here in +# /sys, so the two together give the list for THIS machine and THIS kernel +# without booting it. +# +# Prints "<source>|<path>" so every line can say where the fact came from. +_fw_wanted() { # _fw_wanted <modules-version> + local ver="${1:-}" a m f cfg + # 1. the boot that is running -- _fw_requests is the one dmesg parser + _fw_requests | sed 's/^/boot|/' + # 2. this tree's modules, for the hardware that is actually present + if [ -n "$ver" ] && [ -d "/lib/modules/$ver" ] \ + && command -v modinfo >/dev/null 2>&1; then + find /sys/devices -name modalias -print 2>/dev/null \ + | xargs -r cat 2>/dev/null | sort -u \ + | while IFS= read -r a; do + [ -n "$a" ] || continue + # modprobe resolves an alias through the TREE's + # modules.alias when told which version to use + modprobe -S "$ver" -R -- "$a" 2>/dev/null || true + done | sort -u \ + | while IFS= read -r m; do + [ -n "$m" ] || continue + # the MODULE's name travels with the request: a module + # declares firmware for every chip it supports, so these + # are candidates, and which module they came from is the + # only thing that makes them readable + modinfo -k "$ver" -F firmware -- "$m" 2>/dev/null \ + | sed "s|^|modules:$m\||" + done + fi + # 3. what the config embeds -- those must be on disk to BUILD, and are + # the one part a running kernel never complains about + for cfg in "$ESP/config-$ver" "/boot/config-$ver" /proc/config.gz; do + [ -r "$cfg" ] || continue + case "$cfg" in + *.gz) zcat "$cfg" 2>/dev/null ;; + *) cat "$cfg" ;; + esac | sed -n 's/^CONFIG_EXTRA_FIRMWARE="\(.*\)"$/\1/p' \ + | tr ' ' '\n' | sed '/^$/d; s/^/config|/' + break + done +} + +# A MODULE DECLARES FIRMWARE FOR EVERY CHIP IT SUPPORTS. +# +# The first version of the modules scan printed 56 lines of "MISSING", each +# with its own remedy: amdgpu blobs for Vega, Sienna Cichlid and Aldebaran, +# iwlwifi blobs for five RF families this laptop does not have. Every one of +# them was a fact -- `modinfo -F firmware amdgpu` really does list them -- +# and the conclusion drawn from them ("56 missing") was wrong: the driver +# loads what its own chip needs and ignores the rest. So a boot failure and +# a module declaration are not the same kind of statement and must not be +# printed the same way. +# WHAT IS THERE, NEXT TO WHAT IS NOT. +# +# `iwlwifi-cc-a0-77.ucode` missing, on a machine whose wifi worked under the +# previous kernel, means one of two things -- the file is on disk under +# another directory, or an OLDER API version of it is installed and the new +# driver will not take it -- and the report said neither. Both are one +# `find` and one glob away, and both change what you do next. +_FW_LOCAL="" +_fw_neighbours() { # _fw_neighbours <requested-path> [<repo>] + # THE EXPLANATION GOES TO THE READER, THE PATH GOES IN A VARIABLE. + # Returning the path on stdout meant the caller's `$(...)` swallowed + # every line of the explanation with it, and the report said only + # "installed" with no word about where the file had been. + local req="$1" src="${2:-}" base stem ext found sib _api + _FW_LOCAL="" + base="$(basename "$req")" + # 1. the same file, somewhere else under /lib/firmware. FIRMWARE_ONLY + # entries keep the repository's path, and a driver asks the loader for + # its own -- so `intel` installs intel/iwlwifi/x.ucode while the + # driver wants x.ucode at the top. + found="$(find /lib/firmware -name "$base" -printf '%P\n' 2>/dev/null \ + | grep -v "^$req$" | head -3)" + if [ -n "$found" ]; then + echo " it IS on disk, at another path:" + printf ' /lib/firmware/%s\n' $found + echo " the loader asks for /lib/firmware/$req and nothing else," + echo " so --fix puts a copy there." + _FW_LOCAL="$(printf '%s' "$found" | head -n1)" + return 0 + fi + # 2. an API-versioned sibling: iwlwifi-cc-a0-72.ucode when -77 is asked + # for. The number is the firmware API the driver demands, so an older + # one on disk is the whole story: this kernel needs a newer blob. + case "$base" in + *-[0-9][0-9].*|*-[0-9][0-9][0-9].*) + ext="${base##*.}" + stem="${base%-*}" + sib="$(ls -1 "/lib/firmware/$(dirname "$req")/$stem"-*."$ext" \ + 2>/dev/null | xargs -r -n1 basename | tr '\n' ' ')" + [ -n "$sib" ] && { + echo " installed already, for the same device but an older" + echo " firmware API: $sib" + _api="${base##*-}"; _api="${_api%.*}" + echo " This kernel's driver requires API $_api and will not" + echo " fall back, so the checkout needs to be newer."; } + if [ -n "$src" ] && [ -d "$src" ]; then + sib="$(_fw_index "$src" | grep -- "\(^\|/\)$stem-[0-9]*\.$ext$" \ + | xargs -r -n1 basename | sort -V | tail -3 | tr '\n' ' ')" + [ -n "$sib" ] && echo " the checkout has: $sib" + fi ;; + esac + return 1 +} + +_firmware_missing() { + local fix="${1:-0}" all="${2:-0}" src req f hits n=0 gone=0 fixed=0 failed=0 + local sugg="" ver source chroot=0 _rr _tr seen="" cand=0 cdir mod + src="$(_fw_repo)" + ver="$(_fw_modver)" + cdir="$(mktemp -d)" + _rr="$(stat -c %d:%i /proc/1/root/. 2>/dev/null || true)" + _tr="$(stat -c %d:%i / 2>/dev/null || true)" + [ -n "$_rr" ] && [ -n "$_tr" ] && [ "$_rr" != "$_tr" ] && chroot=1 + + echo "### firmware this machine will ask for, and has not got" + echo "# modules scanned: ${ver:-(none installed under /lib/modules)}" + echo "# running kernel: $(uname -r)$([ "$chroot" = 1 ] && printf ' (the HOST kernel -- this is a chroot)')" + if [ "$chroot" = 1 ]; then + echo "# in a chroot, 'boot' lines describe the HOST's boot -- possibly" + echo "# another distribution on other hardware, in which case they can" + echo "# neither confirm nor deny anything about this tree. 'modules'" + echo "# and 'config' lines are this tree's." + fi + [ -z "$ver" ] && echo "# no modules to scan, so only the running boot can be asked." + echo + while IFS='|' read -r source req; do + [ -n "${req:-}" ] || continue + case "$req" in */) continue ;; esac + case " $seen " in *" $req "*) continue ;; esac + seen="${seen:-} $req" + n=$((n + 1)) + if [ -e "/lib/firmware/$req" ]; then + # SAY WHAT IS SATISFIED, especially here. A boot line that is + # now present is the answer to "will this work when I boot it", + # and it was skipped silently -- so a run with everything in + # place looked identical to one that had not looked at all. + case "$source" in + modules:*) printf '%s\n' "$req" >> "$cdir/${source#modules:}.ok" ;; + *) echo " ok $req ($source) -- installed now" ;; + esac + continue + fi + # A DECLARATION IS A CANDIDATE, NOT A FAILURE. Collected per module + # and summarised below; only boot and config lines are treated as + # things that must be fixed. + case "$source" in + modules:*) + mod="${source#modules:}" + printf '%s\n' "$req" >> "$cdir/$mod" + cand=$((cand + 1)) + [ "$all" = 1 ] || continue ;; + esac + gone=$((gone + 1)) + echo " MISSING /lib/firmware/$req ($source)" + while IFS='|' read -r _ep _epkg _eurl; do + [ "$_ep" = "$req" ] || continue + echo " not part of linux-firmware: it is released as $_epkg" + echo " $_eurl" + if [ "$fix" = 1 ] && [ "$_epkg" = wireless-regdb ]; then + _install_regdb force && fixed=$((fixed + 1)) + else + echo " install it: lfs-kernel firmware --missing --fix" + echo " (without it cfg80211 falls back to the most" + echo " conservative channels and power -- wifi still works)" + fi + continue 2 + done < <(_fw_elsewhere) + # WHAT IS THERE, before what is not: a copy under another path, or an + # older API version, answers the question without the repository. + _fw_neighbours "$req" "$src" + if [ -n "$_FW_LOCAL" ] && [ -e "/lib/firmware/$_FW_LOCAL" ]; then + if [ "$fix" = 1 ]; then + _fw_place "/lib/firmware" "$req" "$_FW_LOCAL" \ + && fixed=$((fixed + 1)) + fi + continue + fi + if [ -z "$src" ] || [ ! -d "$src" ]; then + echo " no linux-firmware checkout to look in" + echo " (FIRMWARE_FROM=$FIRMWARE_FROM; run lfs-kernel firmware first)" + continue + fi + hits="$(_fw_index "$src" "$(basename "$req")" | head -4)" + if [ -n "$_FW_INDEX_ERR" ]; then + echo " could not read the repository, so where this file" + echo " lives is UNKNOWN -- not absent:" + printf ' %s\n' "$_FW_INDEX_ERR" + failed=$((failed + 1)) + continue + fi + if [ -z "$hits" ]; then + echo " this checkout has no file by that name." + echo " su - p_linux-firmware -c 'git -C $src pull'" + continue + fi + printf ' the repository has it at: %s\n' $hits + for f in $hits; do + if [ "$f" = "$req" ]; then + case "$f" in + */) ;; + */*) sugg="$sugg ${f%/*}" ;; + *) sugg="$sugg $f" ;; + esac + else + echo " no FIRMWARE_ONLY entry can express this: the" + echo " repository keeps it at '$f', the driver asks for" + echo " '$req'. --fix installs it where the loader looks." + fi + done + [ "$fix" = 1 ] && _fw_place "$src" "$req" "$(echo $hits | cut -d" " -f1)" \ + && fixed=$((fixed + 1)) + done < <(_fw_wanted "$ver") + + [ "$gone" = 0 ] && [ "$n" -gt 0 ] \ + && echo " nothing that has actually been asked for is missing." + + # ---- the candidates, per module, in a few lines rather than fifty ---- + if [ "$cand" -gt 0 ] && [ "$all" != 1 ]; then + echo + echo "Declared but not installed -- CANDIDATES, not failures. A module" + echo "declares firmware for every chip it supports; yours loads only" + echo "the ones it needs, and never asks for the others:" + for f in "$cdir"/*; do + [ -f "$f" ] || continue + case "$f" in *.ok) continue ;; esac + mod="$(basename "$f")" + echo " $mod: $(grep -c . "$f") of $(( $(grep -c . "$f") \ + + $( [ -f "$cdir/$mod.ok" ] && grep -c . "$cdir/$mod.ok" || echo 0) )) \ +declared file(s) not installed, e.g. $(head -n1 "$f")" + done + if [ "$chroot" = 1 ]; then + echo " If a driver you need is silent, boot THIS kernel and run this" + echo " again there: the 'boot' list above is the host's, and says" + echo " nothing about the driver you are asking about." + else + echo " If a driver you need is silent, look at what it actually asked" + echo " for -- the 'boot' list above, which is this kernel's." + fi + echo " To install every declared file anyway (large, mostly unused):" + echo " lfs-kernel firmware --missing --fix --all" + fi + + # WHICH FILE A DRIVER PICKS IS DECIDED AT RUN TIME. iwlwifi derives the + # name from the RF and CRF ids it reads off the card, so no amount of + # looking at modules and directories can promise it will find one. Say + # that, rather than implying a green run is a working driver. + if [ "$cand" -gt 0 ] || [ "$gone" -gt 0 ]; then + echo + echo " Only a boot of THIS kernel settles it: a driver derives the" + echo " firmware name from the ids it reads off the card, so what is" + echo " installed can be checked here, and what will be asked for" + echo " cannot." + fi + rm -rf "$cdir" + echo + if [ "$n" = 0 ]; then + echo "Nothing asks for firmware here: no failed load in this boot, and" + echo "no module under /lib/modules${ver:+/$ver} declares any." + return 0 + fi + if [ "$gone" = 0 ]; then + return 0 + fi + if [ "$fix" = 1 ]; then + echo "Placed $fixed of $gone file(s) at the exact path the driver asked for." + if [ -n "$sugg" ]; then + echo "Make it survive the next full run -- in $CONF:" + echo " FIRMWARE_ONLY=\"$FIRMWARE_ONLY$(printf '%s' "$sugg")\"" + fi + [ "$fixed" -gt 0 ] && { + echo "Then load the driver again (or reboot):" + echo " modprobe -r <driver> # iwlwifi, amdgpu, ..."; } + if [ "$failed" -gt 0 ]; then + echo "$failed lookup(s) failed, so nothing is known about those files." + return 1 + fi + [ "$fixed" = "$gone" ] && return 0 + echo "Still missing: $((gone - fixed))." + return 1 + fi + echo "Nothing was changed. Install what linux-firmware has:" + echo " lfs-kernel firmware --missing --fix" + if [ -n "$sugg" ]; then + echo "or add them to FIRMWARE_ONLY in $CONF and re-run lfs-kernel firmware:" + echo " FIRMWARE_ONLY=\"$FIRMWARE_ONLY$(printf '%s' "$sugg")\"" + fi + return 1 +} + +# Install ONE blob at the path the driver actually asked for. The driver's +# path and the repository's path are not the same thing (iwlwifi asks for a +# flat name; the repository may keep it under a directory), and it is the +# driver's that has to exist. +# +# ONE FILE MEANS ONE FILE. The first version widened the sparse checkout, +# and fell back to `sparse-checkout disable` when the add was refused -- in +# cone mode a file path is refused, so the fallback would have materialised +# the whole of linux-firmware: over ten gigabytes, to fix one 2 MB blob. +# `git show HEAD:<path>` reads exactly one blob out of the repository and +# needs no sparse mode at all (in a partial clone it fetches that blob and +# nothing else). +_fw_place() { # <src-repo> <requested-path> <repo-path> + local src="$1" req="$2" rel="$3" acct=p_linux-firmware + # SEPARATE STATEMENT. `local a="$2" d="/lib/firmware/$a"` reads $a before + # this local has assigned it, and `set -u` kills the function: + # lfs-kernel: line ...: req: unbound variable + local dst="/lib/firmware/$req" + id "$acct" >/dev/null 2>&1 || packagemanager add-user linux-firmware || return 1 + # /lib/firmware is this package's directory (the full run hands it over); + # the account has to be able to write the file it is about to own. + install -d -m 755 /lib/firmware 2>/dev/null + [ "$(stat -c %U /lib/firmware 2>/dev/null)" = "$acct" ] \ + || chown "$acct:$acct" /lib/firmware 2>/dev/null + case "$req" in + */*) install -d -o "$acct" -g "$acct" -m 755 "/lib/firmware/$(dirname "$req")" \ + 2>/dev/null || true ;; + esac + if [ -e "$src/$rel" ]; then + local _cerr + _cerr="$(su - "$acct" -c "cp -a '$src/$rel' '$dst'" 2>&1)" || { + echo " !! could not copy it as $acct:" + printf ' %s\n' "${_cerr:-(no message)}"; return 1; } + elif [ -d "$src/.git" ]; then + echo " reading it out of the repository (it is in the index," + echo " not in the sparse checkout)" + local _gerr + # git's own message, not a guess about what went wrong: "does not + # exist in HEAD" and "unable to read sha1 file" (a partial clone with + # no network) need different answers from the reader. + _gerr="$(su - "$acct" -c "git -c 'safe.directory=$src' -C '$src' show 'HEAD:$rel' > '$dst'" 2>&1)" || { + echo " !! git could not produce $rel:" + printf ' %s\n' "${_gerr:-(no message)}" + echo " a partial clone fetches the blob on demand, so" + echo " this needs the network; check with:" + echo " su - $acct -c 'git -C $src fetch --depth 1'" + rm -f "$dst"; return 1; } + else + echo " !! $rel is not in $src" + return 1 + fi + # CONFIRM YOUR OWN EFFECT: a failed `git show` can leave an empty file + # behind, which looks exactly like a successful install. + if [ -s "$dst" ]; then + echo " installed $dst ($(stat -c %s "$dst") bytes)" + packagemanager reload-pkg-list linux-firmware >/dev/null 2>&1 || true + return 0 + fi + echo " !! $dst is empty or absent after the copy" + rm -f "$dst" + return 1 +} + +# ---- firmware without a kernel build -------------------------------------- +if [ "${_FWONLY:-0}" = 1 ]; then + case "${_FWARG:-}" in + --missing|missing) + _firmware_missing "${_FWFIX:-0}" "${_FWALL:-0}"; exit $? ;; + "") + [ "$FIRMWARE_FROM" = none ] \ + && { echo "!! FIRMWARE_FROM=none in $CONF -- nothing to install."; exit 1; } + _install_firmware || exit 1 + echo "# firmware installed; no kernel was built or touched." + exit 0 ;; + *) echo "!! lfs-kernel firmware takes --missing [--fix] or nothing"; exit 1 ;; + esac +fi + +echo "# kernel $VERSION -> $ESP (tree $tree, $JOBS)" + +# -------------------------------------------------------------------------- +# THE COMMANDS. Everything above chooses the version and checks the paths; +# everything below is the build, and is meant to be read. +# -------------------------------------------------------------------------- +install -d "$SRC" +if [ ! -d "$tree" ]; then + [ -f "$tarball" ] || { + echo "# fetching $MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" + wget -q --show-progress -O "$tarball.part" \ + "$MIRROR_TAR/v$major.x/linux-$VERSION.tar.xz" \ + && mv "$tarball.part" "$tarball" \ + || { rm -f "$tarball.part" + echo "!! download failed -- KERNEL_LIST=1 shows what exists"; exit 1; } + } + tar -xf "$tarball" -C "$SRC" || exit 1 + chown -R "$acct:$acct" "$tree" "$tarball" 2>/dev/null || true +fi +[ -d "$tree" ] || { echo "!! $tree is missing"; exit 1; } + +# boot logo (optional) +if [ -n "${BOOTLOGO:-}" ]; then + logo="$SRC/$(basename "$BOOTLOGO")" + [ -f "$logo" ] || { wget -qO "$logo" "$BOOTLOGO" && chown "$acct:$acct" "$logo"; } || logo="" + [ -n "$logo" ] && _as_pkg "cp -v '$logo' drivers/video/logo/logo_linux_clut224.ppm" +fi + +# .config +case "$CONFIG_FROM" in + running) _as_pkg "gunzip < /proc/config.gz > .config" || { + echo "!! /proc/config.gz unreadable -- set CONFIG_FROM=file:<path>"; exit 1; } ;; + file:*) _as_pkg "cp -v '${CONFIG_FROM#file:}' .config" || exit 1 ;; + keep) [ -f "$tree/.config" ] || { echo "!! no .config in $tree"; exit 1; } ;; + *) echo "!! CONFIG_FROM=$CONFIG_FROM not understood"; exit 1 ;; +esac +# ---- refreshing the config for a NEW kernel ------------------------------ +# +# A config from one kernel is never complete for another: every version adds +# symbols the old file has no answer for. Something must decide them, and +# which something matters: +# +# olddefconfig take the DEFAULT for each new symbol, no questions. The +# right choice for an unattended run, and what this tool has +# always done -- but the defaults are upstream's, not yours. +# oldconfig ask about each new symbol, one at a time. Slow, and needs +# someone at the terminal, but nothing is decided silently. +# menuconfig open the editor and decide everything at once. +# none use the .config exactly as it is; `make` will then ask, or +# fail, depending on what is missing. +# +# It also says how many symbols the refresh had to answer, because "47 new +# symbols were set to their defaults" is worth knowing before a boot. +: "${CONFIG_REFRESH:=olddefconfig}" +_cfg_before="$(mktemp)"; cp "$tree/.config" "$_cfg_before" 2>/dev/null || true +case "$CONFIG_REFRESH" in + none) echo "# not refreshing the config (CONFIG_REFRESH=none)" ;; + olddefconfig|oldconfig|menuconfig|*config) + echo "# refreshing the config for $VERSION (make $CONFIG_REFRESH) ..." + _as_pkg "make $CONFIG_REFRESH" || exit 1 ;; + *) echo "!! CONFIG_REFRESH=$CONFIG_REFRESH is not understood"; exit 1 ;; +esac +if [ -s "$_cfg_before" ] && [ "$CONFIG_REFRESH" != none ]; then + _newsym="$(diff <(sort "$_cfg_before") <(sort "$tree/.config") 2>/dev/null \ + | grep -c '^>' || true)" + [ "${_newsym:-0}" -gt 0 ] \ + && echo "# $_newsym config line(s) differ from the config you started from" +fi +rm -f "$_cfg_before" + +# ---- CONFIG_SET: this machine's own answers, applied last ---------------- +# +# The config comes from the running kernel (or a file), then a refresh +# answers the new symbols. An option you decided on has to survive both, so +# it is applied AFTER them, resolved once more (olddefconfig -- a forced +# symbol can pull or drop others), and READ BACK: `scripts/config` writes +# what it is told, and the resolve then silently reverts anything whose +# dependencies are not met. +if [ -n "${CONFIG_SET:-}" ]; then + _cs_args="" + for _cs in $CONFIG_SET; do + _sym="${_cs%%=*}"; _val="${_cs#*=}"; _sym="${_sym#CONFIG_}" + [ "$_sym" != "$_cs" ] || { echo "!! CONFIG_SET entry '$_cs' has no '=' -- SYMBOL=VALUE"; exit 1; } + case "$_val" in + y) _cs_args="$_cs_args --enable $_sym" ;; + m) _cs_args="$_cs_args --module $_sym" ;; + n) _cs_args="$_cs_args --disable $_sym" ;; + [0-9]*) _cs_args="$_cs_args --set-val $_sym $_val" ;; + *) _cs_args="$_cs_args --set-str $_sym ${_val//\"/}" ;; + esac + done + echo "# applying CONFIG_SET: $CONFIG_SET" + _as_pkg "./scripts/config $_cs_args && make olddefconfig" >/dev/null || exit 1 + _cs_bad="" + for _cs in $CONFIG_SET; do + _sym="${_cs%%=*}"; _val="${_cs#*=}"; _sym="CONFIG_${_sym#CONFIG_}" + case "$_val" in + n) grep -qE "^$_sym=(y|m)" "$tree/.config" && _cs_bad="$_cs_bad $_sym=n" ;; + y|m) grep -qx "$_sym=$_val" "$tree/.config" || _cs_bad="$_cs_bad $_sym=$_val" ;; + *) grep -qx "$_sym=$_val" "$tree/.config" \ + || grep -qx "$_sym=\"${_val//\"/}\"" "$tree/.config" \ + || _cs_bad="$_cs_bad $_sym=$_val" ;; + esac + done + if [ -n "$_cs_bad" ]; then + echo "!! CONFIG_SET did not take -- the kernel reverted these:" + for _cs in $_cs_bad; do + echo " $_cs (now: $(grep -E "^(# )?${_cs%%=*}[= ]" "$tree/.config" | head -1))" + done + echo " a dependency is missing, or $VERSION has no such symbol." + echo " Look it up: cd $tree && make menuconfig (then / and the name)" + exit 1 + fi + echo "# all $(echo $CONFIG_SET | wc -w) set, read back from .config" +fi + +# what a Wayland compositor needs; a warning, not a refusal +for opt in CONFIG_DRM CONFIG_DRM_FBDEV_EMULATION CONFIG_INPUT_EVDEV CONFIG_SYSVIPC; do + grep -qE "^$opt=(y|m)" "$tree/.config" || echo "!! $opt is not set (sway needs it)" +done +# Drivers that need firmware are no use without the driver. These are the +# ones a laptop misses most often; the firmware for each is in FIRMWARE_ONLY. +for opt in CONFIG_IWLWIFI:"Intel wireless (AX200 and friends)" \ + CONFIG_IWLMVM:"the AX200's own MVM firmware layer" \ + CONFIG_BT_HCIBTUSB:"USB Bluetooth (the AX200's other half)" \ + CONFIG_DRM_AMDGPU:"AMD graphics"; do + _o="${opt%%:*}"; _why="${opt#*:}" + grep -qE "^$_o=(y|m)" "$tree/.config" \ + || echo "!! $_o is not set -- no $_why" +done + # FIRMWARE TROUBLE SHOULD NOT COST YOU THE KERNEL. # # This ran before the build and exited on failure, so a wrong FIRMWARE_ONLY @@ -735,12 +1398,32 @@ case "$HEADERS" in # what makes the files the headers package's. Same shape as the ESP # copies and the setuid bit. echo "# installing headers into /usr/include (as root, owned by $_hacct) ..." + # WHAT IT INSTALLED, NOT WHERE IT INSTALLED. + # + # `chown -R $_hacct /usr/include` gave this package every header on + # the system. /usr/include/gstreamer-1.0/gst/mse then belonged to + # p_linux-headers, so gst-plugins-bad could not install into its own + # directory, the repair handed it one file per round, and after three + # rounds the bundled-copy heuristic concluded meson was vendoring + # linux-headers. Every step of that was correct reasoning from a + # wrong fact. + # + # So: copy, remember exactly which paths came out of the tree, and + # chown only those. + _hlist="$(mktemp)" + ( cd "$tree/usr/include" && find . -mindepth 1 -printf '%P\n' ) > "$_hlist" cp -r "$tree/usr/include" /usr || { echo "!! could not install the headers into /usr/include" echo " if some of them belong to a third package:" echo " packagemanager reclaim $HEADERS_PKG --run" - exit 1; } - chown -R "$_hacct:$_hacct" /usr/include 2>/dev/null || true + rm -f "$_hlist"; exit 1; } + ( cd /usr/include && tr '\n' '\0' < "$_hlist" \ + | xargs -0 -r chown -h "$_hacct:$_hacct" ) 2>/dev/null || true + # the directory itself stays root's: it is shared by every package + # that ships a header, and one owner for it is what caused this + echo "# $(grep -c . "$_hlist") path(s) now owned by $_hacct;" + echo "# /usr/include itself left as $(stat -c %U /usr/include)" + rm -f "$_hlist" packagemanager reload-pkg-list "$HEADERS_PKG" >/dev/null 2>&1 || true echo "# headers installed and recorded for $_hacct" ;; *) echo "# headers: skipped (HEADERS=$HEADERS)" ;; diff --git a/lfs-phases b/lfs-phases index ced43ca..2e3b0eb 100755 --- a/lfs-phases +++ b/lfs-phases @@ -34,7 +34,7 @@ # scripts are also SOURCED by lfs-helper to read their metadata, and this file # comes along -- it must not shadow anything in the sourcing shell. -PKGUSR_LIB_VERSION="1.14.146" +PKGUSR_LIB_VERSION="1.14.321" _pp_build_id() { local f="${BASH_SOURCE[0]}" @@ -189,12 +189,30 @@ _pp_mirrors_for() { } _pp_fetch() { - local url="$1" try u + # SAVE IT AS THE NAME THE SCRIPT ASKS FOR. + # + # A GitHub archive URL ends in the TAG, not the project: + # .../cisco/openh264/archive/refs/tags/v2.6.0.tar.gz + # so wget wrote v2.6.0.tar.gz, the unpack looked for the declared + # pkg=openh264-2.6.0.tar.gz, and the run said "no + # openh264-2.6.0.tar.gz ... and nowhere to fetch it from" one line under + # "'v2.6.0.tar.gz' saved [60302243]". Both statements were true. + # + # `pkg` is what every later phase uses, and two projects can both + # publish a v2.6.0.tar.gz, so the file has to land under that name. + local url="$1" try u out="${2:-${pkg:-}}" for try in 1 2 3; do while IFS= read -r u; do [ -n "$u" ] || continue [ "$u" = "$url" ] || _pp_say "trying mirror: $u" - wget -4 -c "$u" && return 0 + if [ -n "$out" ]; then + wget -4 -c -O "$out" "$u" && return 0 + # -O leaves an empty file behind on a 404, and -c would then + # "resume" it for ever + [ -s "$out" ] || rm -f "$out" + else + wget -4 -c "$u" && return 0 + fi done <<< "$(_pp_mirrors_for "$url" | awk '!seen[$0]++')" [ "$try" = 3 ] && break _pp_say "every mirror failed (try $try/3) -- retrying in $((try * 15))s" @@ -240,10 +258,32 @@ _pp_unpack() { if [ "${is_git:-}" = 1 ]; then pkg_dir="$name_version" + # GIT MUST NEVER ASK A QUESTION IN A BUILD. + # + # A URL that is not there gets a 401 from GitHub (it will not admit + # a repository is missing), so git asks for a username and the build + # ends with + # fatal: could not read Username for 'https://github.com' + # ...which reads like a credentials problem on a public repository + # and, on a terminal, would simply hang. Refuse the prompt and say + # what it means. + export GIT_TERMINAL_PROMPT=0 GIT_ASKPASS=/bin/true GIT_SSH_COMMAND="ssh -oBatchMode=yes" if [ -d "$pkg_dir/.git" ]; then git -C "$pkg_dir" fetch --all --tags --prune else - rm -rf "$pkg_dir" && git clone --recurse-submodules "$link" "$pkg_dir" + rm -rf "$pkg_dir" + if ! git clone --recurse-submodules "$link" "$pkg_dir"; then + _pp_fail "could not clone $link" + echo " git was refused, and was not allowed to ask for a" >&2 + echo " password (this is a build). On a public URL that" >&2 + echo " almost always means the PATH IS WRONG: GitHub answers" >&2 + echo " 401 for a repository that does not exist rather than" >&2 + echo " admitting it is missing." >&2 + echo " Check it, from here, before editing anything else:" >&2 + echo " git ls-remote $link | head -3" >&2 + echo " Then fix url= in the stack entry." >&2 + return 1 + fi fi # A REF THAT DOES NOT EXIST IS A TYPO IN THE STACK, NOT A BUILD ERROR. # @@ -252,14 +292,60 @@ _pp_unpack() { # which reads like a build failure and sends people to the log. The # repository is right here and knows what it does have; show it, and # say where the wrong value came from. - if [ -n "${git_ref:-}" ] && ! git -C "$pkg_dir" checkout "$git_ref"; then + # A BRANCH IS NOT A POINT IN TIME, AND `checkout` DOES NOT MOVE IT. + # + # On an existing clone the fetch above brings the new commits in, and + # `git checkout master` then says "Already on 'master'" and stays + # exactly where it was. So a package pinned to a branch rebuilt the + # SAME commit for ever: `update <pkg>` fetched, compiled and installed + # a version it already had, and said it had updated it. A tag or a + # commit is a point in time and must NOT move -- those are checked out + # detached, which is what they are. + _pp_ref="${git_ref:-}" + if [ -z "$_pp_ref" ]; then + # no ref: whatever the remote calls its default branch + _pp_ref="$(git -C "$pkg_dir" symbolic-ref --short refs/remotes/origin/HEAD 2>/dev/null)" + _pp_ref="${_pp_ref#origin/}" + [ -n "$_pp_ref" ] && _pp_say "# no ref given -- following origin's default branch ($_pp_ref)" + fi + if [ -n "$_pp_ref" ] \ + && git -C "$pkg_dir" show-ref --verify --quiet "refs/remotes/origin/$_pp_ref"; then + _pp_before="$(git -C "$pkg_dir" rev-parse --short HEAD 2>/dev/null)" + if ! git -C "$pkg_dir" checkout -B "$_pp_ref" "origin/$_pp_ref"; then + _pp_fail "could not put $pkg_dir on branch $_pp_ref" + return 1 + fi + _pp_after="$(git -C "$pkg_dir" rev-parse --short HEAD 2>/dev/null)" + if [ -n "$_pp_before" ] && [ "$_pp_before" != "$_pp_after" ]; then + _pp_say "# $_pp_ref: $_pp_before -> $_pp_after ($(git -C "$pkg_dir" rev-list --count "$_pp_before..$_pp_after" 2>/dev/null) new commit(s))" + else + _pp_say "# $_pp_ref is already at $_pp_after" + fi + git -C "$pkg_dir" submodule update --init --recursive + echo "$pkg_dir" > "$_pp_dir_marker" + _pp_say "checked out into $BUILD_ROOT/$pkg_dir" + return 0 + fi + if [ -n "${git_ref:-}" ] && ! git -C "$pkg_dir" checkout --detach "$git_ref"; then _pp_fail "'$git_ref' is not a tag, branch or commit in $link" echo " The stack entry (or this script) asks for a ref the" >&2 echo " repository does not have. Recent tags:" >&2 - git -C "$pkg_dir" tag --sort=-v:refname 2>/dev/null | head -8 \ - | sed 's/^/ /' >&2 + # AN EMPTY LIST UNDER A HEADING IS A RIDDLE. A fork with no + # tags printed "Recent tags:" and then nothing, which reads like + # the command failed rather than "there are none -- use a + # branch". + _pp_tags="$(git -C "$pkg_dir" tag --sort=-v:refname 2>/dev/null | head -8)" + if [ -n "$_pp_tags" ]; then + printf '%s\n' "$_pp_tags" | sed 's/^/ /' >&2 + else + echo " (none at all -- this repository is not tagged," >&2 + echo " so a branch is the only thing to ask for)" >&2 + fi echo " Branches:" >&2 + # HEAD and the bare remote name are not branches anyone can ask + # for: the list printed `origin` above `origin/main`. git -C "$pkg_dir" branch -r --format='%(refname:short)' 2>/dev/null \ + | grep -vE '^[^/]+$|/HEAD$' | sed 's|^[^/]*/||' \ | head -5 | sed 's/^/ /' >&2 echo " Fix ref= in the stack entry for '${name:-$name_version}'," >&2 echo " or set git_ref= in this script, then run it again." >&2 @@ -287,13 +373,36 @@ _pp_unpack() { && _pp_say "# the build resumes in $_pp_prepared_in" return 0 fi - _pp_stage_file "$pkg" || { [ -n "${link:-}" ] && _pp_fetch "$link"; } - [ -f "$pkg" ] || { _pp_fail "no $pkg in $BUILD_ROOT and nowhere to fetch it from"; return 1; } + _pp_stage_file "$pkg" || { [ -n "${link:-}" ] && _pp_fetch "$link" "$pkg"; } + if [ ! -f "$pkg" ]; then + _pp_fail "no $pkg in $BUILD_ROOT and nowhere to fetch it from" + # SAY WHAT DID ARRIVE. This fired one line under a successful + # download and read as "the fetch failed", when the fetch had + # worked and only the NAME differed. + local _got + _got="$(ls -1t *.tar.* *.tgz *.zip 2>/dev/null | head -3)" + if [ -n "$_got" ]; then + echo " what IS here, newest first:" >&2 + printf ' %s\n' $_got >&2 + echo " if one of those is it, the entry's pkg= does not match the" >&2 + echo " name it arrived under -- a GitHub archive URL ends in the" >&2 + echo " TAG, not the project." >&2 + fi + return 1 + fi _pp_check_md5 || return 1 local li f + # EACH EXTRA UNDER ITS OWN NAME. _pp_fetch saves to its second + # argument, else to $pkg -- and this call passed none, so LLVM's patch + # was fetched with `wget -c -O llvm-project-22.1.8.src.tar.xz`. The + # tarball was bigger, wget answered 416 "already fully retrieved", and + # the build stopped on + # patch: **** Can't open patch file ../llvm-22.1.8-upstream_fix-1.patch + # (An extra LARGER than the tarball would have been appended onto it.) for li in "${additional_links[@]}"; do f="${li##*/}" - _pp_stage_file "$f" || _pp_fetch "$li" || return 1 + _pp_stage_file "$f" || _pp_fetch "$li" "$f" || return 1 + [ -s "$f" ] || { _pp_fail "fetched $li, and there is no $f"; return 1; } done if [ "${pkg_unpack:-yes}" = "no" ]; then _pp_say "fetched $pkg (the page's own commands unpack it)" diff --git a/packagemanager b/packagemanager index 2fc602f..92749ce 100755 --- a/packagemanager +++ b/packagemanager @@ -27,22 +27,23 @@ installs ends up owned by that user, so `list_package <name>` records the exact manifest into /usr/src/<name>/pkg.lst. Collector groups (nimgnu_*) share directories between package-users; u_* users run GUI apps for the human. -Roadmap (built incrementally, one command at a time) ----------------------------------------------------- - info [IMPLEMENTED] package-user info table (name/uid/gid/home/pkg_list) - check [TODO] report install state of a package - search [TODO] find a package in the book -> blfs search - dependencies [TODO] recursive dependency order -> blfs deps/order - install [TODO] resolve deps, build missing, run lfs-helper pm-install - remove [TODO] remove a package-user + its files (via pkg.lst) - reload-pkg-list [TODO] regenerate /usr/src/<name>/pkg.lst - list [TODO] list all package-users - files_with_broken_id [TODO] find files whose uid/gid don't resolve - -Design decisions still open (we'll settle these as we implement the commands): - * install: auto-run lfs-helper pm-install per package, or just print/queue? - * packagemanager_install: retired in 1.12.0 -- folded into lfs-helper pm-install - * fix `check`, implement `remove`; parameterize the hardcoded site values. +The commands +------------ +`packagemanager --help` lists them, grouped by what you are doing, and that +list is generated beside the parsers themselves. + +THERE WAS A SECOND LIST HERE. A hand-kept "Roadmap" marking check, search, +dependencies, install, remove and reload-pkg-list as [TODO] -- all six had +been implemented for releases, and none of the twenty commands added since +appeared at all. A list of commands maintained anywhere but next to the +commands goes stale silently, and this file is the place a reader starts. + +Design decisions still open: + * should the CONFIGURE phase run as root? Every BLFS "Configuring" section + is written for root, and files it creates would be root-owned, which the + file tracking does not expect. + * the `.new` mechanism: three functions move stack files and it still + surprises people. """ import argparse @@ -117,8 +118,67 @@ BASE_DIR = os.environ.get("PKGUSR_BASE") or _base_dir_default() def appuser_home(): """Where application users live -- separate from package users, because - they are accounts a person uses rather than build trees.""" - return load_config().get("appuser_home") or BASE_DIR + they are accounts a person uses rather than build trees. + + DEFAULT /home, NOT the package base. This returned BASE_DIR, so every + application user's home, shared directory and XDG profile link were + built under /usr/src -- `mkdir -p /usr/src/u_claude/Shared`, and a + ~/Shared/Claude link pointing into the package tree. A build tree is + not a home. Configurable as before (`appuser_home` in the config), but + the default is now the one that is right. + """ + return load_config().get("appuser_home") or APPUSER_HOME_DEFAULT + + +# ONE SUBDIRECTORY FOR ALL OF THEM. The user's layout, recorded on the +# machine as appuser_home=/home/shared_user -- and ignored: the account was +# moved to a hard-coded "/home/<user>", so /home filled up with u_* beside +# the real logins. The default is that subdirectory; the config key still +# overrides it. +APPUSER_HOME_DEFAULT = "/home/shared_user" + + +# /home/Shared (1.14.315). 1.14.312 put a hub of LINKS in +# /home/shared_user/Shared and argued against /home: a directory there that +# is not a home looks like one. Once the hub holds the DATA, that argument +# turns round: the shared directories are the one thing in /home/shared_user +# that is not private to an account, and taking them out is what lets every +# account home be 0700. +SHARED_HUB_DEFAULT = "/home/Shared" + + +def shared_hub(): + """Where every shared directory lives (1.14.312; the data itself since + 1.14.315 -- see SHARED_HUB_DEFAULT). + + It was ~/Shared of the main user. Two things were wrong with that: + + * ONE PERSON'S HOME IS NOT A SYSTEM PATH. A program of your own, a + script, a launcher, an app account -- anything that wants "the + shared dirs" had to know whose home to look in. A fixed path is the + same answer for every account. + * The SHARED_* variables were enumerated from $HOME/Shared by EVERY + login -- and in an app account $HOME/Shared is its own data + directory, so u_claude got SHARED_<FILENAME> for each file in it. + + Default /home/Shared: the real directories, one per account + (/home/Shared/Claude, 2770 u_claude:u_claude). Each account's ~/Shared + and the main user's ~/Shared are links into it, so nothing you type + changes. `shared_hub=` in packagemanager.conf moves it. + """ + return (load_config().get("shared_hub") or "").rstrip("/") \ + or SHARED_HUB_DEFAULT + + +def _app_home(u): + """An application user's home: what passwd says, or where they go. + + passwd FIRST, because an account whose home has already been moved + (usermod -d) must not have its shared directory rebuilt somewhere else + -- which is exactly what happened: the homes were moved to /home and the + next `cfg app-users` run recreated Shared under /usr/src. + """ + return _home_of(u) or os.path.join(appuser_home(), u) INSTALL_SCRIPTS_DIR = os.environ.get("PKGUSR_INSTALL_SCRIPTS", "/etc/pkgusr/install_scripts") TMP_DIR = "/tmp/packagemanager" @@ -128,7 +188,7 @@ TMP_DIR = "/tmp/packagemanager" CONFIG_PATH = os.environ.get("PKGUSR_CONFIG", "/etc/pkgusr/packagemanager.conf") # Version stamps. Bump PLAN_VERSION when the cached dry-run plan structure # changes (so stale plans are ignored); TOOL_VERSION is the program version. -TOOL_VERSION = "1.14.146" +TOOL_VERSION = "1.14.321" def _build_id(): @@ -636,7 +696,128 @@ def find_blfs(): return ["blfs"] # rely on $PATH -def _blfs_book_flags(): +# A PACKAGE MAY NEED A DIFFERENT BOOK, AND ONLY A WHOLE PAGE IS COHERENT. +# +# The systemd book's linux-pam page lists systemd as required; the 12.4 SysV +# book's linux-pam page has no Required section at all -- it wants only +# Shadow. Same package, two books, two dependency graphs, and this machine +# is SysV. +# +# NOT a merge. Taking the dependency list from one book and the commands +# from another produces a script neither book has ever tested: the versions +# differ (1.7.1 vs 1.7.2), the configure switches differ, the doc tarball +# differs. A whole page from the older book is something upstream shipped +# and someone ran. Half of each is not. +# +# machine.conf: +# [linux-pam] +# book = /usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html +# +# ...and everything that asks the book about THAT package -- versions, +# dependencies, the generated script -- asks the one named there. +_BOOK_FOR_PKG = {} + +# A STACK CAN READ ITS PACKAGES FROM ANOTHER BOOK (1.14.321). +# +# media.stack: from-book svn +# +# ...and every book entry of THAT file (not of files it includes) is read +# from the development book: versions, dependencies, the generated script. +# It is a property of the package, not of one run -- `update`, `info` and +# `install --reinstall` ask the same book the stack built it from, or an +# update-all would "update" mpv back to the stable book's older page. +# +# Dependencies outside the stack stay on their own book: an installed one is +# left alone (the svn page naming a newer glib is not a reason to rebuild +# glib), a missing one is built from the default book, and only a page the +# default book does not have at all is read from the stack's book. +_STACK_BOOKS = None # anchor (lower) -> selector +_PLAN_BOOK = None # the stack book a running plan's target reads +_BOOK_PATHS = {} # selector -> local file ("" = could not resolve) + + +def _stack_book_map(): + """anchor -> selector, from every installed stack's `from-book`.""" + global _STACK_BOOKS + if _STACK_BOOKS is not None: + return _STACK_BOOKS + _STACK_BOOKS = {} + # a stack run hands its own file's books to the installs it starts + # (it may be a checkout, not the installed copy) + for kv in (os.environ.get("PM_STACK_BOOKS") or "").split(","): + if "=" in kv: + a, sel = kv.split("=", 1) + _STACK_BOOKS[a.strip().lower()] = sel.strip() + d = STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks" + for f in sorted(glob.glob(os.path.join(d, "*.stack"))): + try: + for r in _parse_stack_file(f): + sel = r["opts"].get("from-book") + if sel and r["kind"] == "book": + _STACK_BOOKS.setdefault(r["target"].lower(), sel) + except (ValueError, OSError): + continue + return _STACK_BOOKS + + +def _book_path_for(selector, refresh=False): + """The local file for a book selector, asked of `blfs path` ONCE. + + With refresh, `blfs fetch` first -- a stack run on the svn book wants + today's svn book; every other command reads the cached copy and never + touches the network.""" + if not selector: + return None + if os.path.isfile(selector): + return selector + if refresh or selector not in _BOOK_PATHS: + if refresh: + set_status("refreshing the %s book ..." % selector) + try: + r = subprocess.run(find_blfs() + ["fetch", selector], + capture_output=True, text=True) + finally: + clear_status() + if r.returncode != 0: + printWarning(" could not refresh the %s book; using the " + "cached one" % selector) + r = subprocess.run(find_blfs() + ["path", selector], + capture_output=True, text=True) + out = (r.stdout or "").strip().splitlines() + _BOOK_PATHS[selector] = out[-1] if r.returncode == 0 and out \ + and os.path.isfile(out[-1]) else "" + if not _BOOK_PATHS[selector]: + printWarning(" no %s book on disk (blfs fetch %s)" + % (selector, selector)) + return _BOOK_PATHS[selector] or None + + +def _book_override(anchor): + """The book this package should be read from, or None. + + machine.conf `book =` first, then a stack's `from-book`, then -- inside + a plan for a stack-book package -- that book for a page the default + book does not have at all.""" + if not anchor: + return None + if anchor not in _BOOK_FOR_PKG: + opts = _machine_opts_for(anchor) + sel = opts.get("book") or _stack_book_map().get(anchor.lower()) or "" + _BOOK_FOR_PKG[anchor] = (_book_path_for(sel) or sel) if sel else "" + if _BOOK_FOR_PKG[anchor]: + return _BOOK_FOR_PKG[anchor] + if _PLAN_BOOK and not _map_lookup(_versions_for(""), [anchor]) \ + and _map_lookup(_versions_for(_PLAN_BOOK), [anchor]): + return _PLAN_BOOK + return None + + +def _blfs_book_flags(anchor=None): + over = _book_override(anchor) if anchor else None + if over: + if os.path.isfile(over): + return ["--book-file", over] + return ["--book", over] if os.environ.get("BLFS_BOOK_FILE"): return ["--book-file", os.environ["BLFS_BOOK_FILE"]] if os.environ.get("BLFS_BOOK"): @@ -659,13 +840,31 @@ _BLFS_ACTIVITY = { _FORWARDED_NOTES = set() -def run_blfs(args): +_SAID_BOOK = set() + + +def run_blfs(args, book=None): """Run blfs; return (returncode, stdout, stderr). rc 127 => blfs missing. + book= reads that book file for this call, whatever the package. + Each call is a subprocess, and the first one for a given book INDEXES it, which takes many seconds. Announce what is happening: an install that sits silent for half a minute before its first output looks frozen.""" - cmd = find_blfs() + _blfs_book_flags() + list(args) + # the package a call is ABOUT is its first non-flag argument, which is + # what decides whether an override applies + _anchor = next((a for a in args[1:] if not a.startswith("-")), None) \ + if len(args) > 1 else None + if book: + cmd = find_blfs() + ["--book-file", book] + list(args) + else: + cmd = find_blfs() + _blfs_book_flags(_anchor) + list(args) + # ONCE per package: a plan asks blfs several things about each one + if not book and _anchor and _book_override(_anchor) \ + and _anchor not in _SAID_BOOK: + _SAID_BOOK.add(_anchor) + printInfo(" %s: read from %s" + % (_anchor, os.path.basename(_book_override(_anchor)))) verb = args[0] if args else "" if verb in _BLFS_ACTIVITY: arg = args[1] if len(args) > 1 and not args[1].startswith("-") else "" @@ -706,50 +905,73 @@ def run_blfs(args): pass -_VERSION_MAP = None # anchor -> name_version, the whole book, once +_VERSION_MAPS = {} # book file ("" = default) -> {anchor: nv} +_BOOK_MISSES = set() # names no book had: ask the slow way once -def book_version(anchor): - """Current name-version of a package in the book, or None if unavailable. +def _versions_for(book): + """The whole version map of one book, loaded once per process.""" + if book not in _VERSION_MAPS: + try: + _VERSION_MAPS[book] = book_versions_all(book or None) or {} + except Exception: + _VERSION_MAPS[book] = {} + return _VERSION_MAPS[book] + + +def _map_lookup(vmap, names): + for a in names: + if a in vmap: + return vmap[a] + for a in names: + low = a.lower() + for k, nv in vmap.items(): + if k.lower() == low: + return nv + return None + - ONE PROCESS FOR THE WHOLE BOOK, not one per question. This ran `blfs - debug <anchor>` every time it was called -- a subprocess that opens and - reads the book cache to answer one lookup. A stack preview asks it once - per entry (56 for sway) and the planner asks it for every package it - considers, so a command whose real work is a dictionary lookup spent - minutes launching processes: +def book_version(anchor): + """Current name-version of a package in ITS book, or None. - looking up pipewire in the book <- one process - looking up ... in the book <- another + One version map per book, loaded once (see book_versions_all); a name + no map has falls back to `blfs debug` once, and a miss is remembered -- + `install imv` asked three slow questions to learn the same "no" three + times. The package's book is machine.conf's, else its stack's + from-book, else the default. - `book_versions_all` was already here, used only by `update`. The map is - loaded on first use and answers everything after that; a name the map does - not have (an alias, a URL, a fragment) still falls back to the old call. + THE PREFIX IS NOT PART OF THE BOOK'S NAME: `info webkitgtk` asks with + the account name p_webkitgtk, which no anchor matches. """ - global _VERSION_MAP - if _VERSION_MAP is None: - try: - _VERSION_MAP = book_versions_all() or {} - except Exception: - _VERSION_MAP = {} - hit = _VERSION_MAP.get(anchor) - if hit: - return hit - if _VERSION_MAP: - low = anchor.lower() - for a, nv in _VERSION_MAP.items(): - if a.lower() == low: - return nv - rc, out, _ = run_blfs(["debug", anchor]) - if rc != 0: + _tries = [anchor] + for _pre in (pkgusr_prefix(), user_prefix(), collector_prefix()): + if _pre and anchor.startswith(_pre): + _tries.append(anchor[len(_pre):]) + own = _book_override(_tries[-1]) + books = ([own] if own and os.path.isfile(own) else []) + [""] + if _PLAN_BOOK and _PLAN_BOOK not in books: + books.append(_PLAN_BOOK) + for b in books: + hit = _map_lookup(_versions_for(b), _tries) + if hit: + return hit + if _tries[-1] in _BOOK_MISSES: return None - m = re.search(r"^title\s*:\s*(.+)$", out, re.M) + rc, out, _ = run_blfs(["debug", _tries[-1]]) + m = re.search(r"^title\s*:\s*(.+)$", out, re.M) if rc == 0 else None nv = m.group(1).strip() if m else None - if nv and _VERSION_MAP is not None: - _VERSION_MAP[anchor] = nv # ask once, even for an alias + if nv: + _versions_for(books[0])[anchor] = nv # ask once, even for an alias + else: + _BOOK_MISSES.add(_tries[-1]) return nv +def default_book_version(anchor): + """The version the DEFAULT book has, whatever book the package reads.""" + return _map_lookup(_versions_for(""), [anchor]) + + def book_rdeps(target, no_recommended=False, optional=False): """anchors of packages that recursively depend on `target`, via blfs rdeps.""" argv = ["rdeps", target, "--anchors"] @@ -768,10 +990,79 @@ def book_rdeps(target, no_recommended=False, optional=False): return result -def book_versions_all(): - """Map anchor -> name_version for the whole book, in ONE blfs call (so update - doesn't spawn a process per package).""" - rc, out, _ = run_blfs(["versions"]) +def _versions_cache_path(book=""): + """Where the anchor -> version map is kept between runs: one per book.""" + d = os.path.join("/var", "cache", "pkgusr") + try: + os.makedirs(d, exist_ok=True) + except OSError: + d = "/tmp" + tag = hashlib.md5(book.encode()).hexdigest()[:8] if book else "default" + return os.path.join(d, "book-versions-%s.json" % tag) + + +def _default_book_file(): + """The file the default book is: BLFS_BOOK_FILE, BLFS_BOOK, `blfs path`. + + NOT the newest file on disk: once a stack fetched the svn book it was + the newest, and every version question on the machine would have been + answered from svn.""" + _bf = os.environ.get("BLFS_BOOK_FILE") or "" + if _bf and os.path.isfile(_bf): + return _bf + if not hasattr(_default_book_file, "_p"): + _p = "" + try: + r = subprocess.run(find_blfs() + ( + ["--book", os.environ["BLFS_BOOK"]] + if os.environ.get("BLFS_BOOK") else []) + ["path", "--offline"], + capture_output=True, text=True) + out = (r.stdout or "").strip().splitlines() + if r.returncode == 0 and out and os.path.isfile(out[-1]): + _p = out[-1] + except OSError: + pass + if not _p: # an older blfs without `path` + _books = sorted((f for f in glob.glob("/usr/share/blfs/books/*.html") + if "svn" not in os.path.basename(f)), + key=lambda f: os.stat(f).st_mtime, reverse=True) + _p = _books[0] if _books else "" + _default_book_file._p = _p + return _default_book_file._p + + +def book_versions_all(book=None): + """Map anchor -> name_version for the whole book (default, or `book`). + + ONE CALL WAS NOT ENOUGH: IT IS ONE CALL PER PROCESS. `blfs versions` + re-reads the whole book to build this map, which is eighteen seconds on + a 12 MB book -- and `info` pays it to answer one question. The page + index is cached; this map was not, so every command that wanted a + version rebuilt it from scratch. + + Cached on disk against the book's own size and mtime: the moment the + book file changes the cache is ignored, so a stale map cannot outlive + the book it came from. + """ + # WHICH book: the one asked for, else the default book's file. + _bf = book if book and os.path.isfile(book) else _default_book_file() + _key = None + if _bf: + try: + _st = os.stat(_bf) + _key = "%s:%d:%d" % (_bf, _st.st_size, int(_st.st_mtime)) + except OSError: + _key = None + _cp = _versions_cache_path(_bf) + if _key: + try: + with open(_cp, encoding="utf-8") as f: + _c = json.load(f) + if _c.get("key") == _key and isinstance(_c.get("versions"), dict): + return _c["versions"] + except (OSError, ValueError): + pass + rc, out, _ = run_blfs(["versions"], book=_bf or None) versions = {} if rc == 0: for line in out.splitlines(): @@ -780,6 +1071,19 @@ def book_versions_all(): a, nv = a.strip(), nv.strip() if a and nv and " " not in a: versions[a] = nv + if versions and _key: + # CONFIRM IT CAN BE READ BACK, not just written: a cache that fails + # to load silently turns into the 18 seconds all over again, on + # every command, invisibly. + try: + with open(_cp, "w", encoding="utf-8") as f: + json.dump({"key": _key, "versions": versions}, f) + with open(_cp, encoding="utf-8") as f: + if json.load(f).get("key") != _key: + printWarning(" the version cache did not read back; " + "every command will rebuild it") + except (OSError, ValueError) as e: + printInfo(" (not caching the book version map: %s)" % e) return versions @@ -873,9 +1177,20 @@ def _parse_bash_vars(text, scalars=(), arrays=()): # trailing comment is allowed and ignored. out = {} for key in scalars: - m = re.search(rf'^{key}="(.*?)"[ \t]*(#.*)?$', text, re.M) + # SINGLE QUOTES TOO, AND THE BOOK USES THEM. + # + # This read only key="value". The book-generated scripts write + # name_version='Rustc-1.93.1' + # so name_version was never parsed for any package built from the + # book -- and that is where the installed version comes from. + # `info webkitgtk` printed uid, home, 1727 files and every + # dependency, and not WHICH webkitgtk; `update` filed it under + # "cannot judge"; late-deps reported 28 packages with no recorded + # version. One quote character, three symptoms. + m = re.search(rf'^{key}=(?:"(?P<d>.*?)"|\x27(?P<s>.*?)\x27)[ \\t]*(#.*)?$', text, re.M) if m: - out[key] = m.group(1) + out[key] = m.group("d") if m.group("d") is not None \ + else m.group("s") for key in arrays: m = re.search(rf'^{key}=\((.*?)\)[ \t]*(#.*)?$', text, re.M) if m: @@ -1071,9 +1386,62 @@ class User: @property def version(self): + # THE VERSION IS WHAT FINISHED INSTALLING. + # + # install_last is staged into the home BEFORE the build, so reading + # the version out of it meant a FAILED build advanced it: webkitgtk + # 2.52.5 died at configure and `update` then said "already up to + # date (2.52.5)" with 2.50.5's libraries still on disk. + # + # .pkgusr-installed is written by lfs-helper only after every phase + # succeeded. Where it is absent -- every package built before this + # release -- fall back to install_last, which is the old behaviour + # and no worse than it was. + if self.home: + try: + with open(os.path.join(self.home, ".pkgusr-installed"), + encoding="utf-8", errors="replace") as f: + done = f.read().strip() + if done: + return done + except OSError: + pass return self.meta.get("name_version") +def _resolve_need(name): + """A needs= name -> the book's anchor for it (1.14.317). + + The book files a package under an ANCHOR that is not always its name: + libjpeg-turbo-3.2.0 is `libjpeg`. needs=libjpeg-turbo therefore found + no page, the stack tried to INSTALL it, and stopped on + ERROR: 'libjpeg-turbo' not found + with the package installed all along -- the "did you mean" under it even + named the right anchor. So: a name that is not an anchor, and matches + exactly ONE page's own package name (its version string without the + version), is that page. More than one, or none, and the name is used as + given, which fails the way it always did. + """ + try: + rc, out, _e = run_blfs(["search", name]) + except Exception: + return name + anchors, by_pkg = set(), [] + for ln in (out or "").splitlines(): + m = re.match(r"\s*\[package\]\s+(\S+)\s+(\S+)", ln) + if not m: + continue + anchor, ver = m.group(1), m.group(2) + anchors.add(anchor) + base = re.sub(r"-\d[\w.+~-]*$", "", ver) + if base.lower() == name.lower(): + by_pkg.append(anchor) + if name in anchors or len(set(by_pkg)) != 1: + return name + printInfo(" (%s is the book's `%s`)" % (name, by_pkg[0])) + return by_pkg[0] + + def gather_user(name): """Populate a User from the system user db, the on-disk package dir, and its install_last script. @@ -1127,8 +1495,46 @@ def print_user_details(u): head += f" ({u.version})" print(f"\n{colors.cyan}{head}{colors.normal} " f"[{_STATE_COLOR.get(state, '')}{state}{colors.normal}]") - print(f" {colors.dim}why{colors.normal:<9}: {why}") + # `why` EXISTS TO EXPLAIN A VERDICT THAT NEEDS EXPLAINING. + # + # For an installed package it was printing the routine sentence every + # time -- "recorded files present (the page names no targets to verify + # against)" above the very fields it is talking about -- so the one line + # that ever matters was indistinguishable from noise. Routine reasons + # are dropped; anything unusual is kept and labelled `note`, because for + # an installed package it is a note, not a justification. + _routine = ("all installed_* targets present", + "recorded files present", + "its script names no verifiable target") + _dull = any(r in (why or "") for r in _routine) + if why and not (state == "installed" and _dull): + _label = "note" if state == "installed" else "why" + print(f" {colors.dim}{_label:<12}{colors.normal}: {why}") + # THE VERSION IS THE FIRST THING ANYONE WANTS, and it was only in the + # header -- shown when `u.version` happened to be set, and silent + # otherwise. webkitgtk printed uid, home, 1727 files and every + # dependency, and not which webkitgtk. + # + # `name_version` is in the install script itself (WebKitGTK-2.50.2), so + # when the recorded version is missing there is still an answer; it just + # has to say which answer it is. And the BOOK's version belongs beside + # it: "is this current" is the question behind the question. + # JUST THE NUMBER. The header already says WebKitGTK-2.50.5, so + # repeating the name in the field made the two lines look like two + # different facts. + def _ver_only(nv): + m2 = re.search(r"-([0-9][0-9A-Za-z.\-_+]*)$", nv or "") + return m2.group(1) if m2 else (nv or "") + _nv = m.get("name_version") or "" + _inst = _ver_only(u.version or "") + if not _inst and _nv: + _inst = _ver_only(_nv) + if _inst: + _inst += " (from the install script, not recorded)" + _book = _ver_only(expected or "") rows = [ + ("version", _inst or "not recorded, and the script does not name one"), + ("book", _book if _book and _book not in _inst else None), ("uid/gid", f"{u.uid}/{u.gid}" if u.has_user else None), ("home", u.home), ("link", m.get("link")), @@ -1146,6 +1552,25 @@ def print_user_details(u): if label == "info": lines = [ln.strip() for ln in str(value).replace("\\n", "\n").split("\n") if ln.strip()] + # THE BOOK'S BLURB REPEATS THE DEPENDENCY LISTS, which are three + # rows further down: webkitgtk printed 18 required packages + # twice, in different spellings, and the description they were + # attached to scrolled away. Keep the prose, drop the lines that + # are just a dependency list. + _deps = set() + for _k in ("required", "recommended", "optional"): + for _d in m.get(_k, []): + _deps.add(_d.split("-")[0].lower()) + def _is_dep_list(ln): + parts = [x.strip() for x in ln.replace(" and ", ", ").split(",") + if x.strip()] + if len(parts) < 3: + return False + hit = sum(1 for x in parts + if x.split("-")[0].lower().replace(" ", "-") in _deps + or x.split("-")[0].lower() in _deps) + return hit >= max(2, len(parts) // 2) + lines = [ln for ln in lines if not _is_dep_list(ln)] if not lines: continue print(f" {label:<12}: {lines[0]}") @@ -1351,6 +1776,59 @@ def _prewarm_scripts(plan, script_dir, force_regen=False): printInfo(" %d written" % made) +def _report_stale_requires(names, _verbose=False): + """For a single-package update: which of its requirements are older here. + + The prereq check catches a missing TOOL. It says nothing about a + dependency that is present but older than the version the book's page + was written against -- and that is the real question behind "can I + update webkitgtk without updating glib2": + + webkitgtk 2.52.5 requires GLib-2.88.3; this system has GLib-2.86.4. + + That may build perfectly. It may also fail at configure, or worse, + build and misbehave. Nobody can tell from here, so this reports the + numbers and leaves the decision alone -- refusing would break the one + workflow it exists to serve, which is updating one package at a time. + """ + for name in names[:4]: + u = gather_user(name) + if not u.meta: + continue + rows = [] + for kind in ("required", "recommended"): + for dep in u.meta.get(kind, []): + a = _resolve_dep(dep, _pkg_dirs()) if "_resolve_dep" in globals() \ + else None + want = book_version(a or dep.rsplit("-", 1)[0]) or dep + have = None + if a: + have = gather_user(a).version or \ + (gather_user(a).meta or {}).get("name_version") + if have and want and not _same_version(have, want): + _row = (kind, a or dep, have, want) + if _row not in rows: # two deps can resolve alike + rows.append(_row) + if not rows: + continue + print() + printWarning(" %s's requirements that differ from the book here:" + % name) + _show = rows if _verbose else rows[:12] + for kind, a, have, want in _show: + printWarning(" %-11s %-22s installed %-24s book %s" + % (kind, a, have, want)) + if len(rows) > len(_show): + # A TRUNCATED LIST MUST SAY HOW TO SEE THE REST. "... and 5 + # more" with no way to ask is a dead end. + printInfo(" ... and %d more -- all of them: packagemanager " + "update %s -v" % (len(rows) - len(_show), name)) + printInfo(" This is not a refusal: one of them may be exactly what") + printInfo(" you do not want to touch. It is what the page was") + printInfo(" written against, so if the build fails at configure,") + printInfo(" start with these.") + + def _stop_if_prereqs_missing(args): """A plan that cannot finish should not start. @@ -1692,9 +2170,45 @@ def _read_conf_section(path, anchor): return {} +def _opt_sections(anchor, keys, first_only=False): + """Which sections actually define these keys -- [all], the package, or both. + + Both messages below used to name the package section whatever the truth + was: `machine.conf [rust]: x.py --jobs 13` for a file with no [rust] + section, and `[somepkg] and [all] set weird_option` where only one of + them did. A message that names a thing must be checked against the + thing, and here the thing is two files and two sections. + """ + keys = set(keys) + out = [] + for sec in (anchor, "all"): + for f in (os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), + os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES)): + if keys & set(_read_conf_section(f, sec)): + out.append(sec) + break + # the package section overrides [all], so for a value that was USED + # only the winner is worth naming: "[webkitgtk] and [all]: ninja -j8" + # credited a section whose 13 had just been overridden by 8. + if out and first_only: + break + return out + + def _machine_opts_for(anchor): + # [all] FIRST, THEN THE PACKAGE. + # + # `jobs` had to be written per package, because ninja ignores MAKEFLAGS + # and there was no way to say "never more than 13 on this machine" once. + # An [all] section in either file applies to every package, and a named + # section overrides it -- so [webkitgtk] jobs = 8 still wins over + # [all] jobs = 13. opts = _read_conf_section( - os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), anchor) + os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), "all") + opts.update(_read_conf_section( + os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), "all")) + opts.update(_read_conf_section( + os.environ.get("PM_PACKAGE_FIXES", PACKAGE_FIXES), anchor)) opts.update(_read_conf_section( os.environ.get("PM_MACHINE_CONF", MACHINE_CONF), anchor)) return opts @@ -1967,6 +2481,29 @@ def _check_script_prereqs(anchor, script_path): _warn_missing_build_tools(anchor, script_path) +def _has_command(body, cmd): + r"""Is `cmd` really invoked here -- not mentioned in a comment? + + THE BOOK PUTS IT AFTER `&&`. webkitgtk's page writes + mkdir -vp build && cd build && cmake -D CMAKE_BUILD_TYPE=Release \\ + and the detector used `^\s*cmake`, so it found no cmake line and + reported "machine.conf [webkitgtk] set USE_WOFF2 ... but the script has + no `meson setup` or `cmake` line -- NOT applied". Seven options, + silently dropped, on the one package where they matter most. + `_insert_on_command` had already learnt about `&&`; only the test that + decides whether to call it had not. + + Comment lines do not count: the same page opens with a NOTE about "the + cmake command", which is prose. + """ + for line in body.split("\n"): + if line.lstrip().startswith("#"): + continue + if re.search(r"(?:^|\s|&&|;)%s\b" % re.escape(cmd), line): + return True + return False + + def _insert_on_command(body, cmd, arg): """Add an argument to the first REAL invocation of `cmd`, skipping comments -- a book NOTE that mentions the command by name is prose.""" @@ -1980,9 +2517,68 @@ def _insert_on_command(body, cmd, arg): return "\n".join(out) +def _meson_option_names(anchor): + """The options this package's own source declares, if it is unpacked. + + meson fails the whole build on an unknown option, and a machine.conf + section is written by hand: `[gst10-plugins-bad] alsa = enabled` looks + perfectly reasonable and is wrong, because `alsa` is a + gst-plugins-BASE option. The answer is in the source tree, which is on + disk by the time anything is built. + """ + home = pkgusr_home(anchor) + src = os.path.join(home, "src") + if not os.path.isdir(src): + return None + names = set() + found = False + for d in sorted(os.listdir(src)): + for f in ("meson.options", "meson_options.txt"): + path = os.path.join(src, d, f) + if not os.path.isfile(path): + continue + found = True + try: + body = open(path, encoding="utf-8", errors="replace").read() + except OSError: + continue + names |= set(re.findall(r"""^\s*option\(\s*['"]([^'"]+)['"]""", + body, re.M)) + return names if found else None + + +def _warn_unknown_meson_opts(anchor, opts): + """Drop options the package does not have, and say which they are.""" + known = _meson_option_names(anchor) + if not known: + return opts # not unpacked yet, or not a meson package + # meson's own built-ins are not in meson.options and are always valid + builtin = {"wrap_mode", "buildtype", "prefix", "libdir", "default_library", + "c_args", "cpp_args", "c_link_args", "cpp_link_args", + "b_lto", "b_ndebug", "werror", "optimization", "strip", + "auto_features", "backend", "jobs", "configure_args"} + bad = {k: v for k, v in opts.items() if k not in known and k not in builtin} + if not bad: + return opts + for k in sorted(bad): + printWarning(" machine.conf [%s]: '%s' is not an option of this " + "package -- NOT passed" % (anchor, k)) + # the near misses are what makes this useful: `alsa` under + # [gst10-plugins-bad] belongs under [gst10-plugins-base] + near = sorted(n for n in known if k in n or n in k)[:4] + if near: + printInfo(" it does have: %s" % ", ".join(near)) + printInfo(" meson would have failed the whole build on this" + " (\"Unknown option\").") + printInfo(" the options it does have: grep -h \"^option(\" %s" + % os.path.join(pkgusr_home(anchor), "src", "*", + "meson.options")) + return {k: v for k, v in opts.items() if k not in bad} + + def apply_machine_opts(anchor, script_path): """Merge machine.conf's [anchor] options into the script's meson line.""" - opts = _machine_opts_for(anchor) + opts = _warn_unknown_meson_opts(anchor, _machine_opts_for(anchor)) if not opts or not script_path or not os.path.isfile(script_path): return try: @@ -2001,6 +2597,10 @@ def apply_machine_opts(anchor, script_path): # ERROR: Unknown option: "needs". # One file, two readers: each must ignore what belongs to the other. opts.pop("needs", None) + # Same hazard, same answer: expect= is a post-install feature probe read + # by the stack runner. It belongs in a stack line, but this file is + # where a key that is not a build option does damage, so drop it here too. + opts.pop("expect", None) # ...and REMOVE it if an earlier version already wrote it into the script. # 1.13.69 stopped adding -Dneeds=, but every script generated while the # bug existed still carries it, and merging is idempotent so nothing took @@ -2035,13 +2635,45 @@ def apply_machine_opts(anchor, script_path): # the failure it prevents arrives hours in, as a killed compiler. _jobs = opts.pop("jobs", None) if _jobs: - _nb = re.sub(r"(?m)^(\s*ninja\b(?![^\n]*\s-j))", - r"\1 -j" + str(_jobs).lstrip("-j"), body) - if _nb != body: - body = _nb + _n = str(_jobs).lstrip("-j") + # NINJA IS NOT THE ONLY ONE THAT IGNORES THE ENVIRONMENT. + # + # rust's own page says it: the build "can be mostly limited to a + # specified number of processors by a COMBINATION of adding the + # switch --jobs <N> on each invocation of ./x.py and using the + # CARGO_BUILD_JOBS environment variable". With only the variable + # set, `./x.py build` used every core. + # + # Each pattern refuses to touch a line that already names -j or + # --jobs, so an explicit choice in a script or a stack entry wins. + _subs = ( + # ninja: no environment variable at all + # ...and not only at the start of a line: `cmake .. && ninja` is + # how half the pages are written, and that ninja was untouched + (r"(?m)(^|&&[ \t]*|;[ \t]*)(ninja\b(?![^\n]*(\s-j|\s--jobs)))", + r"\1\2 -j" + _n, "ninja -j%s"), + # ./x.py build|install|test|dist + (r"(?m)^(\s*\.?/?x\.py\s+\w+(?![^\n]*(\s-j|\s--jobs)))", + r"\1 --jobs " + _n, "x.py --jobs %s"), + # cargo build / cbuild / install / cinstall + (r"(?m)^(\s*cargo\s+c?(?:build|install)\b(?![^\n]*(\s-j|\s--jobs)))", + r"\1 -j " + _n, "cargo -j %s"), + ) + _changed = [] + for _pat, _rep, _what in _subs: + _nb = re.sub(_pat, _rep, body) + if _nb != body: + body = _nb + _changed.append(_what % _n) + if _changed: with open(script_path, "w") as f: f.write(body) - printInfo(" machine.conf [%s]: ninja -j%s" % (anchor, _jobs)) + printInfo(" machine.conf %s: %s" + % (" and ".join("[%s]" % w + for w in _opt_sections(anchor, ["jobs"], + first_only=True)) + or "[%s]" % anchor, + ", ".join(_changed))) _wrap = opts.pop("wrap_mode", None) if _wrap: if "meson setup" not in body: @@ -2057,6 +2689,20 @@ def apply_machine_opts(anchor, script_path): with open(script_path, "w") as f: f.write(body) printInfo(" machine.conf [%s]: wrap-mode %s" % (anchor, _wrap)) + # KEYS THAT ADDRESS THE TOOL, NOT THE BUILD. + # + # `[linux-pam] book = .../BLFS-BOOK-12_4-nochunks.html` tells the tool + # WHICH BOOK to read that package from -- and it went straight onto the + # build line: + # meson setup -Dbook=/usr/share/blfs/books/... .. + # ERROR: Unknown option: "book". + # jobs and configure_args were already popped out one at a time; a + # reserved SET means the next such key cannot leak the same way, and + # `_warn_unknown_meson_opts` cannot save us here because at script + # generation time the source is not unpacked yet and it has no + # meson.options to check against. + for _reserved in ("book",): + opts.pop(_reserved, None) _conf_args = opts.pop("configure_args", None) if _conf_args: if "./configure" not in body: @@ -2112,11 +2758,26 @@ def apply_machine_opts(anchor, script_path): # and the person had nowhere to put -D ENABLE_SPELLCHECK=OFF. Both build # systems take -Dkey=value; only the command word differs. (autotools is # already handled above, by configure_args.) - _anchor_cmd = "meson setup" if "meson setup" in body else ( - "cmake" if re.search(r"^\s*cmake\b", body, re.M) else "") + # NOTHING LEFT TO APPLY IS NOT A PROBLEM. + # + # `[all] jobs = 13` was consumed above (x.py --jobs 13, printed and + # correct) and then this warned that rust's options had NOT been + # applied -- with nothing left in opts to apply. + if not opts: + return + _anchor_cmd = ("meson setup" if _has_command(body, "meson setup") + else ("cmake" if _has_command(body, "cmake") else "")) if not _anchor_cmd: - printWarning(f" machine.conf has [{anchor}] but the script has no " - f"`meson setup` or `cmake` line -- options NOT applied") + # ...AND NAME THE SECTION THAT ACTUALLY EXISTS. It said + # machine.conf has [rust] but ... + # about a file with no [rust] section at all: the keys had come from + # [all]. A message that names a thing must be checked against the + # thing. + _where = _opt_sections(anchor, opts) + printWarning(" machine.conf %s set %s for %s, but the script has no " + "`meson setup` or `cmake` line -- NOT applied" + % (" and ".join("[%s]" % w for w in _where) or "(unknown)", + ", ".join(sorted(opts)), anchor)) printWarning(" (an autotools package takes configure_args instead)") return new = body @@ -2270,14 +2931,26 @@ def install_targets(meta): """Collect concrete things the install script says it installs: programs, libraries, directories (from installed_program(s)/-libraries/ -director(y|ies)).""" + # ONE SCALAR, TWO PATHS. The book writes "Installed Directories: + # /usr/include/opus and /usr/share/doc/opus-1.6.1", and both ended up in + # a single `installed_directory=`. The check then looked for a + # directory literally called + # "/usr/include/opus /usr/share/doc/opus-1.6.1" + # which of course does not exist -- so opus read "unvalidated: missing: + # dir /usr/include/opus ..." with /usr/include/opus sitting right there. + # Same for a program or library field with two names in it. + def _paths(v): + return [x for x in str(v or "").split() if x] + progs, libs, dirs = [], [], [] - if meta.get("installed_program"): - progs.append(meta["installed_program"]) - progs += meta.get("installed_programs", []) - libs += meta.get("installed_libraries", []) - if meta.get("installed_directory"): - dirs.append(meta["installed_directory"]) - dirs += meta.get("installed_directories", []) + progs += _paths(meta.get("installed_program")) + for _p2 in meta.get("installed_programs", []): + progs += _paths(_p2) + for _l in meta.get("installed_libraries", []): + libs += _paths(_l) + dirs += _paths(meta.get("installed_directory")) + for _d in meta.get("installed_directories", []): + dirs += _paths(_d) # A STACK ENTRY'S OWN DECLARATION WINS OVER A STALE RECORD. # # If the stack says what proves this package installed, use that: it is @@ -2390,6 +3063,25 @@ def assess(u, expected=None, run_cmd=True): and missing[0] == f"program {_pkgname}"): return "unvalidated", ("its script names no verifiable target" " (rebuilt scripts declare one)") + # A DOC DIRECTORY NOBODY WAS GOING TO BUILD IS NOT A FAULT. + # + # /usr/share/doc/opus-1.6.1 exists only if Doxygen and texlive are + # installed, and this system deliberately has neither -- the + # generator comments those steps out. So the page names a directory + # that was never going to appear, and opus read "unvalidated" + # because of it while every header and library was in place. + # the path COMPONENT, not a /usr prefix: a doc directory is a doc + # directory under any prefix, and anchoring on /usr/share/doc made + # this untestable anywhere else + _docs = [x for x in missing + if x.startswith("dir ") + and ("/share/doc/" in x or "/share/gtk-doc" in x + or "/share/info" in x or "/share/man" in x)] + if _docs and len(_docs) == len(missing): + return "installed", ("docs not built (no doxygen/texlive here): %s" + % ", ".join(d[4:] for d in _docs[:2])) + if _docs: + missing = [x for x in missing if x not in _docs] if missing: # SAY WHERE THE EXPECTATION CAME FROM. "missing: dir # /usr/include/wlr" sent the user to look for a directory; the @@ -2523,7 +3215,7 @@ def cmd_verify(args): printWarning("the book named no file to check):") for n in _act: u = gather_user(n) - st, why = assess(u, versions.get(u.meta.get("name") or n)) + st, why = assess(u, None, run_cmd=False) printWarning(" %-24s %-14s %s" % (n, st, why)) # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. # @@ -2585,15 +3277,30 @@ def cmd_info(args): if st in ("not installed", "unvalidated") and not args.no_book: print_book_info(u.name or u.search) - counts = {} + # THREE BUGS IN TWO LINES, and `info` crashed after printing everything + # it was asked for: + # NameError: name '_states' is not defined + # `_states` was never initialised here (it belongs to verify's loop, + # where this was copied from), `name` does not exist in this scope -- + # `anchor` does -- and the line was there twice. The dict is worth + # having: it is what names the packages that need attention. + counts, _states = {}, {} for u in users: anchor = u.meta.get("name") or u.name or u.search st, _ = assess(u, versions.get(anchor), run_cmd=False) counts[st] = counts.get(st, 0) + 1 - _states[name] = st - _states[name] = st + _states[anchor] = st print(f"\nTotal: {len(users)} " + " ".join(f"{k}: {v}" for k, v in sorted(counts.items()))) + # ...and say which ones, where the eye lands, rather than leaving the + # reader to match a count against the list above. + _act = [n for n, st in _states.items() + if st in ("not installed", "broken")] + if _act: + print() + printWarning("%d of these need attention: %s" + % (len(_act), ", ".join(sorted(_act)))) + printInfo(" what and why: packagemanager verify %s" % " ".join(sorted(_act))) # =========================================================================== # @@ -2612,6 +3319,13 @@ def cmd_info(args): COLLECTOR_GID_MIN = 90000 COLLECTOR_GID_MAX = 99000 +# Application users (u_*) get their OWN range too. They were created with +# the package-user range, so u_claude took 10349 and the next package user +# started after it -- an application account in the middle of the package +# accounts, uid and private gid alike. +APP_UID_MIN = 30000 +APP_UID_MAX = 39999 + def nimgnu_name(name): """The collector group for a package or directory name. @@ -3026,18 +3740,23 @@ def cmd_nimgnu_list(args): def cmd_nimgnu_create(args): g = nimgnu_name(args.name) - if _group_exists(g) and _user_exists(g): - printInfo(f"{g}: already exists (group + user).") + if _group_exists(g): + printInfo(f"{g}: already exists.") return printHeader(f"Create collector {g}") print() - cmd = find_add_package_user() + [g, g, "10000", "20000", g, "10000", "20000"] - printInfo(f"Creating collector group + user: {' '.join(cmd)}") - if _run(cmd): - printSuccess(f"{g} created (group + collector user).") - report_created_user(g, kind="collector user") + # A GROUP, IN THE COLLECTOR RANGE -- no account. This ran + # the package-user helper with the package range, which made a login account + # named after the group (home /usr/src/pkgusr/p_nimgnu_python-modules) + # and a gid in the middle of the package users (10096). Every other + # collector group comes from _create_collector_group at 90000+. + if _create_collector_group(g): + printSuccess(f"{g} created (gid {grp.getgrnam(g).gr_gid if grp else '?'}).") printInfo(f" assign directories to it with: " f"packagemanager add-dir-to-sysgroup {args.name} <dir>") + else: + printError(f"could not create {g}") + return 1 def cmd_nimgnu_add(args): @@ -3210,6 +3929,11 @@ SHARED_DIR_PATTERNS = [ "/usr/share/aclocal", "/usr/share/gtk-doc/html", "/usr/lib/systemd/system", "/usr/lib/udev/rules.d", "/etc/xdg/autostart", "/usr/libexec", + # /etc/profile.d is shared by nature: rust, bash-completion, gettext and + # the book's own config pages each drop ONE file in it. Without this the + # first package to create the directory owns it -- p_libnice here -- and + # every later package has to be granted it one at a time. + "/etc/profile.d", "/usr/share/terminfo/*", ] @@ -3646,7 +4370,10 @@ def install_tag(anchor, book_version_str): if u.state != "installed": return colors.yellow + "not installed" + colors.normal if u.version and book_version_str and u.version != book_version_str: - return colors.cyan + f"installed {u.version} (update available)" + colors.normal + # "the book has moved on", not "this will be updated": nothing in a + # stack run changes an installed package. + return (colors.cyan + f"installed {u.version} (book is newer)" + + colors.normal) return colors.green + f"installed {u.version or ''}".rstrip() + colors.normal @@ -3897,6 +4624,32 @@ def build_install_plan(args): install order (deps first); reason in {new, update, reinstall}. why maps anchor -> (kind, parent) so we can show 'required by <parent>'.""" ignore = {x for x in (args.ignore or "").split(",") if x} + # THE AVOID LIST IS A PROPERTY OF THE MACHINE, NOT OF A STACK RUN. + # + # `install linux-pam --run --recursive` planned **Systemd-261.2** on a + # SysV system. avoid.stack says `avoid systemd` and has since 1.14.224 -- + # but only `stack` ever read it, so a bare install walked the book's + # graph with no exclusions at all and offered to install the one thing + # this machine must never have. + # + # A package named ON THE COMMAND LINE is still built: asking for it + # explicitly is an instruction, and refusing that would be second- + # guessing. It is dependencies that are filtered. + _named = {a for a in (args.packages or [])} + _avoided = _machine_avoids() - _named + if _avoided: + ignore |= _avoided + # SAY IT, because an avoided REQUIRED dependency changes the build. + # linux-pam's page (in the systemd book) lists systemd as required; + # skipping it is the right call on a SysV machine and the reader + # still needs to know it happened. + printInfo(" %d package(s) this machine never builds are excluded " + "from the plan" % len(_avoided)) + printInfo(" (avoid.stack; a package you name yourself is still " + "built). If the plan") + printInfo(" then looks short, one of them was a dependency: " + "packagemanager deps <pkg>") + printInfo(" the list: packagemanager stack avoid --avoided") plan, seen, missing_all, why = [], set(), [], {} # WHAT IS ALREADY DONE, counted as we go. # @@ -3916,6 +4669,12 @@ def build_install_plan(args): if anchor in seen or anchor in ignore: return seen.add(anchor) + # A DEPENDENCY OUTSIDE A STACK BOOK READS ITS OWN BOOK. The order + # came from the svn book, so it carries svn's versions for + # everything; a dependency the stack does not own is judged -- and + # built, if missing -- against the book it always reads. + if _PLAN_BOOK and _book_override(anchor) != _PLAN_BOOK: + nv = default_book_version(anchor) or nv u = gather_user(anchor) up_to_date = (u.state == "installed" and (not u.version or u.version == nv)) @@ -3928,8 +4687,12 @@ def build_install_plan(args): plan.append((anchor, nv, "update" if u.state == "installed" else "new")) + global _PLAN_BOOK for target in args.packages: set_status(f"working out what {target} needs ...") + _PLAN_BOOK = None + _tb = _book_override(target) + _PLAN_BOOK = _tb if _tb and os.path.isfile(_tb) else None if not getattr(args, "recursive", False): # DEFAULT: just this package, no dependency walk nv = book_version(target) @@ -4091,7 +4854,67 @@ def install_local(name, path, run, yes, verb="install", test=False, clean=False) printSuccess(f"Done -- {name} {verb}ed. logs: {_pkg_log_dir(name)}") +def _stack_git_entry(name): + """The git/tar entry for `name` in an installed stack, or None.""" + d = STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks" + for f in sorted(glob.glob(os.path.join(d, "*.stack"))): + try: + for r in _parse_stack_file(f): + if r["kind"] in ("git", "tar") and r["target"] == name: + return r + except (ValueError, OSError): + continue + return None + + +def _install_git_packages(args): + """Build the git packages among args.packages from their own scripts. + + NO BOOK. `install --reinstall imv` asked the book about imv three + times (an index lookup, `blfs debug`, `blfs search`), took a minute to + say "not in the BLFS book", and built nothing -- imv is a git entry of + apps.stack, and its script is in its home. A git package is recognised + from that script (or from a stack's git/tar entry, the first time), + before anything asks the book. --clean and --test go through as they + do for `update`. Returns the names it handled.""" + done = [] + for name in list(args.packages or []): + script = _canonical_script_path(name) + if not script: + p = os.path.join(pkgusr_home(name), "install_%s" % name) + script = p if os.path.isfile(p) else None + is_git = bool(script) and _is_git_package(name) + entry = None if is_git else _stack_git_entry(name) + if not is_git and not entry: + continue + done.append(name) + if entry and not script: + script, _ = _stack_git_script( + entry, os.path.join(TMP_DIR, "stack_files")) + u = gather_user(name) + if u.state == "installed" and not args.reinstall: + printSuccess("%s: installed (a %s package). Rebuild it: " + "--reinstall [--clean]; new commits: update %s" + % (name, entry["kind"] if entry else "git", name)) + continue + printInfo("%s: %s package -- building from %s (no book lookup)" + % (name, entry["kind"] if entry else "git", script)) + if script != (_canonical_script_path(name) or "") and entry: + apply_machine_opts(name, script) + install_local(name, script, args.run, args.yes, "install", + test=getattr(args, "test", False), + clean=getattr(args, "clean", False)) + return done + + def cmd_install(args, _replanned=False): + # ---- git packages: their own script, never the book ---- + if not args.local and args.packages: + _git_done = _install_git_packages(args) + if _git_done: + args.packages = [p for p in args.packages if p not in _git_done] + if not args.packages: + return # ---- local package: install a custom script directly, no BLFS ---- if args.local or (len(args.packages) == 1 and book_version(args.packages[0]) is None @@ -4146,12 +4969,35 @@ def cmd_install(args, _replanned=False): # because libass was never installed and was never going to be. The # plan knows; it just did not say. if not getattr(args, "recursive", False) and len(plan) == 1: + # A SWALLOWED EXCEPTION HERE IS SILENCE WHERE THE WARNING SHOULD BE. + # + # `except (SystemExit, Exception): _missing = []` made a failed + # dependency lookup indistinguishable from "no missing + # dependencies", and the trap this whole block exists to catch went + # uncaught -- numpy planned alone and died on + # BackendUnavailable: Cannot import 'mesonpy' + # with meson_python (a book page, and numpy's Required) never built. + _missing, _why_not = [], "" try: - _deps, _ = blfs_order_anchors(plan[0][0], ignore=ignore) + # the SAME ignore set the plan was built with (build_install_plan + # has its own local; this referenced a name that does not exist + # here, so every non-recursive install printed "NameError: name + # 'ignore' is not defined" instead of its missing dependencies) + _ignore = {x for x in (getattr(args, "ignore", "") or "").split(",") if x} + _deps, _ = blfs_order_anchors(plan[0][0], ignore=_ignore) _missing = [nv for a, nv, _k, parent in _deps if parent and gather_user(a).state != "installed"] - except (SystemExit, Exception): - _missing = [] + except SystemExit as e: + _why_not = "the book lookup exited (%s)" % e + except Exception as e: + _why_not = "%s: %s" % (type(e).__name__, e) + if _why_not: + printWarning(" could not work out what %s depends on -- %s" + % (plan[0][1], _why_not)) + printWarning(" so this plan builds ONLY %s, and a missing" + " dependency will not be" % plan[0][1]) + printWarning(" reported until the build fails on it. Safer:" + " --recursive") if _missing: printWarning(" %d dependenc%s of %s %s NOT installed:" % (len(_missing), "y" if len(_missing) == 1 else "ies", @@ -4338,7 +5184,7 @@ def _pkglist_paths(user): return {p for p in pkglist_entries(pl) if p.startswith("/")} -def _orphan_sweep(user, old_paths, since): +def _orphan_sweep(user, old_paths, since, remove=True): """Remove files that were in the package's OLD manifest but were NOT rewritten by the new version (mtime older than the install start `since`). Only old- manifest files are considered; config paths + the source tree are kept. Does @@ -4347,6 +5193,16 @@ def _orphan_sweep(user, old_paths, since): if old_paths is None: return 0 protected = _PROTECTED_PREFIXES + (BASE_DIR.rstrip("/") + "/",) + # NEVER ANOTHER PACKAGE'S FILE. An old manifest can list a path that + # has since moved to someone else: switching Mesa to glvnd hands + # libGL.so.1 to libglvnd, which installs it BEFORE the Mesa rebuild -- + # so it is old, in Mesa's old list, and not rewritten by Mesa. mtime + # alone would delete the GL library. A package cleans only what it + # still owns. + try: + own_uid = pwd.getpwnam(user).pw_uid if pwd else None + except KeyError: + own_uid = None victims = [] for p in sorted(old_paths): if any(p == pre.rstrip("/") or p.startswith(pre) for pre in protected): @@ -4354,89 +5210,721 @@ def _orphan_sweep(user, old_paths, since): if not os.path.lexists(p): continue try: - if os.path.getmtime(p) < since: + st = os.lstat(p) + if own_uid is not None and st.st_uid != own_uid: + continue + if st.st_mtime < since: victims.append(p) except OSError: continue + # A LIBRARY SOMETHING IS STILL LINKED TO IS NOT AN ORPHAN. + # + # webkitgtk rebuilt from the GTK4 port to the GTK3 one leaves + # libwebkitgtk-6.0.so.4 in the old manifest, unrewritten and owned by this + # package -- every test above says "orphan", and deleting it breaks every + # program built against it, with no warning and no way back but a rebuild. + # So each library about to go is asked the only question that matters: + # does anything else still record its soname? Files of this package do + # not count -- they go with it. + kept = [] + if victims: + libs = [p for p in victims if re.search(r"\.so(\.\d+)*$", p) + and os.path.isfile(p) and not os.path.islink(p)] + victim_set = set(victims) + for p in libs: + s = _soname_of(p) + if not s: + continue + users = [(f, o) for f, o in _needed_by(s, skip_uid=own_uid) + if f not in victim_set] + if users: + pkgs = sorted({o for _f, o in users}) + printWarning(" --clean: KEEPING %s -- %d file(s) in %s still " + "link against %s" + % (p, len(users), ", ".join(pkgs[:4]), s)) + kept.append((p, s, pkgs)) + if kept: + keep_paths = {p for p, _s, _pk in kept} + # the symlinks that point at a kept library stay with it, or the + # soname it is found by disappears and nothing loads anyway + for p in list(victims): + if os.path.islink(p): + t = os.path.realpath(p) + if t in keep_paths: + keep_paths.add(p) + victims = [p for p in victims if p not in keep_paths] + printInfo(" rebuild those packages, then clean again:") + for _p, _s, pkgs in kept: + printInfo(" packagemanager update %s --reinstall --run" + % " ".join(p[2:] if p.startswith("p_") else p + for p in pkgs[:4])) + break + printInfo(" or see the whole picture: packagemanager used-by %s" + % kept[0][1]) + if not remove: + # the caller decides (_clean_after_install parks them first) + return victims if victims: printInfo(f" --clean: removing {len(victims)} old-version file(s) the " f"upgrade no longer installs (configs / source kept)") remove_paths_as_user(user, victims) - else: + elif not kept: printInfo(" --clean: no leftover files from the old version.") return len(victims) def _reapply_install(user): - """Re-run ONLY the install phase (no recompile), as the user, to restore any - current-version files that --clean may have removed (e.g. files installed - with preserved timestamps). Orphans stay removed.""" - inner = f"cd ~ && bash {shlex.quote(f'install_{user}')} install" - cmd = (["su", "-", user, "-c", inner] if _can_su(user) - else ["sh", "-c", inner]) + """Re-run ONLY the install phase (no recompile), as the user, to put back + the current-version files the mtime test mistook for old ones. + + Returns True only when the phase ran and exited 0. + + IT HAD NOT RUN SINCE 1.14.16 (fixed 1.14.311). It called + `bash ~/install_<account>` -- the runtime copy 1.14.16 removed everywhere + else -- with stdout and stderr thrown away and the exit status ignored. + So every --clean deleted the files the mtime test misjudged, the "restore" + failed on `No such file or directory` where nobody could see it, and the + run ended green. webkitgtk lost /usr/include/webkitgtk-6.0/jsc/*.h that + way: CMake's install skips a file whose content is unchanged ("-- + Up-to-date:"), so the headers kept their old mtime and read as orphans. + + Now: the same script and the same door as `packagemanager script + install` (lfs-helper build), its output kept in a log, and its result + returned so the caller can undo the clean when it fails. + """ + home = pkgusr_home(user) + script = (_versioned_script_in(home) + or os.path.join(home, f"install_{user}")) + if not os.path.isfile(script): + printError(f" --clean: no install script in {home} -- cannot reapply") + return False + helper = shutil.which("lfs-helper") + if helper and _can_su(user): + cmd = [helper, "build", user, "--script", script, + "--phase", "install", "--force"] + else: + inner = f"cd ~ && bash {shlex.quote(script)} install" + cmd = (["su", "-", user, "-c", inner] if _can_su(user) + else ["sh", "-c", inner]) + log = os.path.join(_plans_dir(), + re.sub(r"[^A-Za-z0-9_.-]+", "_", user) + ".clean-reapply.log") printInfo(" --clean: reapplying the install step to restore current files...") - subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + printInfo(" $ " + " ".join(cmd[1:] if cmd[0] == helper else cmd)) + try: + with open(log, "w") as lf: + rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT).returncode + except OSError as e: + printError(f" --clean: could not run the install step ({e})") + return False + if rc != 0: + printError(f" --clean: the install step FAILED (exit {rc}) -- log: {log}") + try: + for line in open(log, errors="replace").read().splitlines()[-12:]: + printInfo(" | " + line) + except OSError: + pass + return False + return True -def _clean_after_install(user, old_paths, since): - """Full --clean sequence run AFTER a successful install: remove old-version - orphans, reapply install (to undo any over-removal), then regenerate the - manifest so it reflects the final state.""" - removed = _orphan_sweep(user, old_paths, since) - if removed: - _reapply_install(user) - printInfo(" --clean: regenerating the file manifest...") - write_pkg_list(user, background=False) - return removed +_PARK_SUFFIX = ".pkgusr-clean" -def _is_git_package(name): - """A package whose install script builds from a git repo (is_git=1, or it - clones/pulls) rather than a book tarball.""" - p = (_canonical_script_path(name) - or os.path.join(pkgusr_home(name), f"install_{name}")) - if not p or not os.path.isfile(p): - return False +def _park_ledger(user): + return os.path.join(_plans_dir(), + re.sub(r"[^A-Za-z0-9_.-]+", "_", user) + ".parked") + + +def _unpark_all(user): + """Put every parked file back, from the ledger. Returns how many.""" + ledger = _park_ledger(user) + n = 0 try: - t = open(p, encoding="utf-8", errors="replace").read(8000) + lines = open(ledger).read().splitlines() except OSError: - return False - return bool(re.search(r'is_git="?1"?', t) - or re.search(r'\bgit\s+(clone|pull|fetch)\b', t)) + return 0 + for p in lines: + if not p: + continue + q = p + _PARK_SUFFIX + if not os.path.lexists(q): + continue + if os.path.lexists(p): + # the install put the current version there: that one stays + try: + os.unlink(q) + except OSError: + pass + continue + try: + os.rename(q, p) + n += 1 + except OSError as e: + printError(f" could not put back {p} ({e}) -- it is at {q}") + try: + os.unlink(ledger) + except OSError: + pass + return n -# A plan entry whose "script" is this is built by `lfs`, not by us: the LFS -# book owns it, and lfs already knows how to build one of its packages. -_LFS_STEP = ":lfs-book:" +# =========================================================================== # +# git packages: what they follow, and whether it moved [IMPLEMENTED] +# =========================================================================== # +# +# A book package has a version, and `update` compares it with the book's. A +# git package has a REF, and a ref answers a different question depending on +# what it is: +# a branch -- a moving target. "Is it behind?" is asked of the remote, and +# the answer changes without anyone doing anything. +# a tag -- a point in time. It is never behind; the question is whether +# a NEWER tag exists, and moving to it is a decision. +# a commit -- the same, without even a name to compare. +# Telling them apart is the whole of this. + +_GIT_ENV = dict(os.environ, GIT_TERMINAL_PROMPT="0", GIT_ASKPASS="/bin/true", + GIT_SSH_COMMAND="ssh -oBatchMode=yes") + + +def _script_var(text, var): + """A shell assignment out of an install script: link="..." / git_ref=...""" + m = re.search(r'^\s*%s\s*=\s*(?:"([^"]*)"|\'([^\']*)\'|([^\s#]*))' + % re.escape(var), text or "", re.M) + if not m: + return "" + return (m.group(1) or m.group(2) or m.group(3) or "").strip() -_TOOLCHAIN_RISK = ("glibc", "binutils", "gcc", "libstdcpp", "ld", "linux-headers") + +def _git_clone_dir(name, link=""): + """Where this package's checkout is. lfs-phases builds in $HOME/src, so + that is where to look -- and the remote is what confirms it, because a + home can hold more than one tree.""" + home = pkgusr_home(name) + for base in (os.path.join(home, "src"), home): + if not os.path.isdir(base): + continue + try: + entries = sorted(os.listdir(base)) + except OSError: + continue + cands = [os.path.join(base, d) for d in entries + if os.path.isdir(os.path.join(base, d, ".git"))] + if link: + for c in cands: + r = subprocess.run(["git", "-C", c, "remote", "get-url", "origin"], + capture_output=True, text=True, env=_GIT_ENV) + if r.returncode == 0 and r.stdout.strip().rstrip("/") == link.rstrip("/"): + return c + if len(cands) == 1: + return cands[0] + return "" -def _lfs_stale_list(include_current=False): - """[(name, installed, book)] for packages the LFS book has newer. +def _git(clone, *args, timeout=60): + try: + r = subprocess.run(["git", "-C", clone] + list(args), capture_output=True, + text=True, env=_GIT_ENV, timeout=timeout) + except (OSError, subprocess.SubprocessError): + return "" + return r.stdout.strip() if r.returncode == 0 else "" - With include_current, the packages whose version did NOT move are listed - too (installed == book). After glibc changes, "up to date" and "built - against the current system" stop being the same statement -- so a full - rebuild in book order is a thing a person may legitimately ask for. - Read from `lfs update --porcelain` -- one tab-separated fact per line. - Parsing the human report instead would break the first time a word in it - changed, and this tool has been bitten by exactly that before.""" - lfs_bin = shutil.which("lfs") or ("/usr/bin/lfs" if os.access( - "/usr/bin/lfs", os.X_OK) else None) - if not lfs_bin: - return [] +def _ls_remote(link, pattern=None, timeout=90): + """[(sha, refname)] from the remote, without cloning anything.""" + cmd = ["git", "ls-remote"] + if pattern == "tags": + cmd += ["--tags", "--refs"] + cmd.append(link) + if pattern and pattern != "tags": + cmd.append(pattern) try: - _cmd = [lfs_bin, "update", "--porcelain"] - if include_current: - _cmd.append("--include-current") - r = subprocess.run(_cmd, capture_output=True, text=True, timeout=600) - except Exception: + r = subprocess.run(cmd, capture_output=True, text=True, env=_GIT_ENV, + timeout=timeout) + except (OSError, subprocess.SubprocessError): + return [] + if r.returncode != 0: return [] out = [] - for ln in (r.stdout or "").splitlines(): - parts = ln.split("\t") + for ln in r.stdout.splitlines(): + parts = ln.split(None, 1) + if len(parts) == 2: + out.append((parts[0].strip(), parts[1].strip())) + return out + + +def _vsort_key(tag): + """Sort 1.9.0 before 1.10.0, which a plain string sort does not.""" + return [int(x) if x.isdigit() else x + for x in re.split(r"(\d+)", tag.lstrip("vV"))] + + +def _git_pkg_state(name, remote=True): + """Everything worth saying about one git package. + + Returns a dict, or None when this is not a git package. `remote=False` + asks the disk only -- no network, for the bulk view.""" + script = (_canonical_script_path(name) + or os.path.join(pkgusr_home(name), "install_%s" % name)) + if not script or not os.path.isfile(script): + return None + try: + text = open(script, encoding="utf-8", errors="replace").read() + except OSError: + return None + if not (re.search(r'is_git="?1"?', text) or re.search(r"\bgit\s+clone\b", text)): + return None + link = _script_var(text, "link") + ref = _script_var(text, "git_ref") + clone = _git_clone_dir(name, link) + st = {"name": name, "script": script, "link": link, "ref": ref, + "clone": clone, "kind": "?", "local": "", "remote": "", + "newer_tags": [], "behind": None, "note": ""} + if clone: + st["local"] = _git(clone, "rev-parse", "HEAD") + if not link: + st["note"] = "the script names no repository (link=)" + return st + if not remote: + return st + # WHAT KIND OF REF IS IT? The remote knows; guessing from the string + # ("master" is a branch, "2.18.0" is a tag) is how a tag named `main` + # or a branch named `1.x` goes wrong. + heads = {r.rsplit("/", 1)[-1]: s for s, r in _ls_remote(link, "refs/heads/*")} + tags = [r.rsplit("/", 1)[-1] for _s, r in _ls_remote(link, "tags")] + st["tags"] = tags + if ref and ref in heads: + st["kind"] = "branch" + st["remote"] = heads[ref] + elif ref and ref in tags: + st["kind"] = "tag" + newer = [t for t in tags if _vsort_key(t) > _vsort_key(ref)] + st["newer_tags"] = sorted(newer, key=_vsort_key) + elif not ref: + st["kind"] = "default branch" + for d in ("main", "master"): + if d in heads: + st["remote"] = heads[d] + st["ref"] = d + break + elif ref: + st["kind"] = "commit or missing ref" + st["note"] = "the remote has no branch or tag called %s" % ref + # how far behind, when the clone already has the objects + if st["kind"].endswith("branch") and st["local"] and st["remote"] \ + and st["local"] != st["remote"] and clone: + n = _git(clone, "rev-list", "--count", "%s..%s" % (st["local"], st["remote"])) + st["behind"] = int(n) if n.isdigit() else None + return st + + +def _git_pkg_line(st): + n = st["name"] + ref = st["ref"] or "(default)" + if st["note"] and not st["remote"] and not st["newer_tags"]: + return " %-20s %-14s %s" % (n, ref, st["note"]) + if st["kind"].endswith("branch"): + if not st["remote"]: + return " %-20s %-14s branch -- the remote could not be reached" % (n, ref) + if st["local"][:12] == st["remote"][:12]: + return " %-20s %-14s branch, up to date (%s)" % (n, ref, st["local"][:8]) + behind = ("%d commit(s) behind" % st["behind"]) if st["behind"] \ + else "moved" + return " %-20s %-14s branch, %s: %s -> %s" % ( + n, ref, behind, (st["local"] or "?")[:8], st["remote"][:8]) + if st["kind"] == "tag": + if not st["newer_tags"]: + return " %-20s %-14s tag, nothing newer released" % (n, ref) + nt = st["newer_tags"] + return " %-20s %-14s tag -- %d newer: %s" % ( + n, ref, len(nt), ", ".join(nt[-4:])) + return " %-20s %-14s %s %s" % (n, ref, st["kind"], st["note"]) + + +def cmd_git_status(args): + """What every git package follows, and whether it has moved.""" + names = args.packages or list_all_users() + states = [] + for n in sorted(set(names)): + st = _git_pkg_state(n, remote=not getattr(args, "local_only", False)) + if st: + states.append(st) + if not states: + printInfo("no git-based packages here" + if not args.packages else + "none of those are git-based packages") + return 0 + printHeader("git packages") + print() + print(" %-20s %-14s %s" % ("package", "ref", "state")) + print(" " + "-" * 62) + moved, taggy = [], [] + for st in states: + print(_git_pkg_line(st)) + if st["kind"].endswith("branch") and st["remote"] \ + and st["local"][:12] != st["remote"][:12]: + moved.append(st["name"]) + if st["newer_tags"]: + taggy.append(st) + print() + if moved: + printInfo(" follow the branch (fetch, rebuild, install):") + printInfo(" packagemanager update %s --run" % " ".join(moved)) + for st in taggy: + printInfo(" %s is pinned to %s; to move it:" % (st["name"], st["ref"])) + printInfo(" packagemanager git-ref %s %s" + % (st["name"], st["newer_tags"][-1])) + if not moved and not taggy: + printInfo(" nothing to do: every git package is where its ref says") + return 0 + + +def cmd_git_ref(args): + """Show what a git package could follow, or move it to another ref.""" + name = args.package + st = _git_pkg_state(name) + if st is None: + printError("%s is not a git-based package" % name) + return 1 + if not st["link"]: + printError("%s: its script names no repository" % name) + return 1 + if not args.ref: + tags = sorted(st.get("tags") or [], key=_vsort_key) + heads = [r.rsplit("/", 1)[-1] for _s, r in _ls_remote(st["link"], "refs/heads/*")] + printHeader("%s -- %s" % (name, st["link"])) + print() + print(" now: %s (%s)" % (st["ref"] or "(default)", st["kind"])) + print() + print(" branches: " + (", ".join(heads[:8]) or "(none)")) + print(" tags, newest last:") + for t in tags[-12:]: + print(" %s%s" % (t, " <- current" if t == st["ref"] else "")) + if not tags: + print(" (none -- this repository is not tagged)") + print() + printInfo(" pick one: packagemanager git-ref %s <ref>" % name) + return 0 + want = args.ref + known = set(st.get("tags") or []) | { + r.rsplit("/", 1)[-1] for _s, r in _ls_remote(st["link"], "refs/heads/*")} + if want not in known and not re.fullmatch(r"[0-9a-f]{7,40}", want): + printError("%s: the remote has no tag or branch called %s" % (name, want)) + printInfo(" what it has: packagemanager git-ref %s" % name) + return 1 + # THE SCRIPT IS ONE OF TWO PLACES THE REF LIVES. A stack entry writes it + # into the generated script, so editing only the script is undone by the + # next stack run -- say so, and name the file. + try: + text = open(st["script"], encoding="utf-8", errors="replace").read() + except OSError as e: + printError("%s: %s" % (st["script"], e)) + return 1 + new, n = re.subn(r'^(\s*git_ref\s*=\s*).*$', r'\1"%s"' % want, text, + count=1, flags=re.M) + if not n: + printError("%s has no git_ref= line to change" % st["script"]) + return 1 + if not getattr(args, "run", False): + printInfo(" would set git_ref=\"%s\" in %s (was \"%s\")" + % (want, st["script"], st["ref"])) + printInfo(" dry run -- add --run") + return 0 + bak = st["script"] + ".bak-" + time.strftime("%Y%m%d-%H%M%S") + try: + shutil.copy2(st["script"], bak) + open(st["script"], "w").write(new) + except OSError as e: + printError("could not write %s: %s" % (st["script"], e)) + return 1 + printSuccess("%s: git_ref is now %s (was %s; backup %s)" + % (name, want, st["ref"] or "(default)", os.path.basename(bak))) + src = _STACK_TARGET_SOURCE.get(name) + if src: + printWarning(" %s also carries this ref: a stack run regenerates the" + " script and puts the old one back." % src) + printInfo(" change the entry there too: ref=%s" % want) + printInfo(" then build it: packagemanager update %s --run" % name) + return 0 + + +# =========================================================================== # +# who still links against this? [IMPLEMENTED] +# =========================================================================== # +# +# A package-user system records who INSTALLED every file. It records nothing +# about who USES one -- and a library is used by things that were built +# against it long ago. webkitgtk rebuilt from the GTK4 port to the GTK3 one +# leaves libwebkitgtk-6.0.so.4 behind with nothing to rebuild it, and +# `--clean` is then one keystroke from deleting a library other programs are +# still linked to. ldd answers it for ONE file; this answers it for a +# library. + +_ELF_SCAN_DIRS = ("/usr/bin", "/usr/sbin", "/usr/lib", "/usr/libexec", + "/bin", "/sbin", "/lib", "/opt") +_ELF_CONFIRM_MAX = 600 # readelf calls; a grep hit is cheap, this is not + + +def _soname_of(path): + """The SONAME an ELF library announces -- libwebkitgtk-6.0.so.4, not the + file name libwebkitgtk-6.0.so.4.13.8. That is the name other binaries + record, so it is the name to search for.""" + if not os.path.isfile(path): + return None + try: + r = subprocess.run(["readelf", "-d", path], capture_output=True, + text=True, timeout=30) + except (OSError, subprocess.SubprocessError): + r = None + if r and r.returncode == 0: + m = re.search(r"\(SONAME\)\s+Library soname: \[([^\]]+)\]", r.stdout) + if m: + return m.group(1) + b = os.path.basename(path) + m = re.match(r"^(lib.+?\.so(?:\.\d+)?)(?:\.|$)", b) + return m.group(1) if m else None + + +def _needed_by(soname, skip_uid=None, dirs=None): + """Every ELF that records this soname in DT_NEEDED: [(path, owner)]. + + grep first -- a soname is a literal string in .dynstr, and grep over + /usr/lib is seconds -- then readelf only on the hits, because a string can + also appear in a script, a .la file or another library's rpath. + """ + # RESOLVED HERE, NOT IN THE SIGNATURE. A default argument is bound at + # import, so `dirs=_ELF_SCAN_DIRS` could never be overridden -- including + # by a test, which is how this function first appeared to find nothing. + live = [d for d in (dirs or _ELF_SCAN_DIRS) if os.path.isdir(d)] + if not live: + return [] + try: + r = subprocess.run(["grep", "-rlZ", "--binary-files=binary", "-F", + "--", soname] + live, + capture_output=True, timeout=600) + except (OSError, subprocess.SubprocessError): + return [] + _all = [c.decode("utf-8", "replace") for c in r.stdout.split(b"\0") if c] + cands = _all[:_ELF_CONFIRM_MAX] + if len(_all) > _ELF_CONFIRM_MAX: + # A TRUNCATED ANSWER MUST NOT READ AS A COMPLETE ONE. + printWarning(" %s: %d candidates, only the first %d were confirmed" + % (soname, len(_all), _ELF_CONFIRM_MAX)) + out = [] + for p in cands: + if os.path.islink(p) or not os.path.isfile(p): + continue + if os.path.basename(p).startswith(soname.split(".so")[0] + ".so"): + continue # the library itself, or its siblings + try: + st = os.lstat(p) + if skip_uid is not None and st.st_uid == skip_uid: + continue # its own package may rebuild freely + d = subprocess.run(["readelf", "-d", p], capture_output=True, + text=True, timeout=30) + except (OSError, subprocess.SubprocessError): + continue + if d.returncode != 0: + continue + if re.search(r"\(NEEDED\).*\[%s\]" % re.escape(soname), d.stdout): + owner = "?" + try: + owner = pwd.getpwuid(st.st_uid).pw_name if pwd else "?" + except KeyError: + owner = str(st.st_uid) + out.append((p, owner)) + return out + + +def cmd_used_by(args): + """What still links against this library, or this package's libraries.""" + target = args.name + sonames = [] + if re.search(r"\.so(\.|$)", target): + sonames = [target if os.sep not in target + else (_soname_of(target) or os.path.basename(target))] + else: + # a package: every shared library in its manifest + libs = [p for p in _pkglist_paths(target) or [] + if re.search(r"/lib[^/]*\.so(\.\d+)*$", p) + and os.path.isfile(p) and not os.path.islink(p)] + for p in sorted(libs): + s = _soname_of(p) + if s and s not in sonames: + sonames.append(s) + if not sonames: + printInfo("%s installs no shared library (or has no manifest)" % target) + return 0 + printHeader("What links against %s" % ", ".join(sonames)) + print() + total = 0 + for s in sonames: + users = _needed_by(s) + by_pkg = {} + for p, owner in users: + by_pkg.setdefault(owner, []).append(p) + if not users: + print(" %-34s nothing links against it" % s) + continue + total += len(users) + print(" %s -- %d file(s), %d package(s):" % (s, len(users), len(by_pkg))) + for owner in sorted(by_pkg): + ex = ", ".join(os.path.basename(x) for x in sorted(by_pkg[owner])[:3]) + more = "" if len(by_pkg[owner]) <= 3 else " ..." + print(" %-22s %s%s" % (owner, ex, more)) + print() + if total: + printInfo(" removing a library these were built against breaks them" + " until they are rebuilt:") + printInfo(" packagemanager update <package> --reinstall --run") + else: + printInfo(" nothing needs these; they are safe to remove") + return 0 + + +def _clean_after_install(user, old_paths, since): + """Full --clean sequence run AFTER a successful install: remove old-version + orphans, reapply install (to undo any over-removal), then regenerate the + manifest so it reflects the final state. + + PARK, REINSTALL, THEN JUDGE (1.14.311). "Older than the install" is + not "no longer installed": CMake skips a file whose content has not + changed, meson and `install -p` keep timestamps, so a current file can + look old. This used to delete first and trust a reapply to repair the + damage -- a reapply that had not run since 1.14.16, so the damage stayed + (webkitgtk's jsc/*.h, and every build against webkitgtk-6.0 broke). + + Now nothing is deleted on the mtime test's word alone. The candidates + are renamed aside (same directory, so it is one rename each and nothing + is copied), the install phase runs, and only then: + * a path the install put back IS current -- the parked copy goes; + * a path it did not put back is a real orphan -- that one goes; + * if the install step FAILS, every parked file is put back, and the + clean is reported as not done. Nothing is half-cleaned. + """ + # an earlier clean interrupted between park and judge: finish undoing it + if os.path.exists(_park_ledger(user)): + n = _unpark_all(user) + printWarning(f" --clean: an interrupted clean had parked files -- " + f"{n} put back first") + + victims = _orphan_sweep(user, old_paths, since, remove=False) + if not victims: + printInfo(" --clean: no leftover files from the old version.") + printInfo(" --clean: regenerating the file manifest...") + write_pkg_list(user, background=False) + return 0 + + files = [p for p in victims if os.path.islink(p) or not os.path.isdir(p)] + dirs = [p for p in victims if p not in set(files)] + parked = [] + with open(_park_ledger(user), "w") as led: + for p in files: + try: + os.rename(p, p + _PARK_SUFFIX) + parked.append(p) + led.write(p + "\n") + led.flush() + except OSError as e: + printWarning(f" could not set {p} aside ({e}) -- left in place") + printInfo(f" --clean: {len(parked)} file(s) look older than this install " + f"-- set aside while the install step confirms which are gone") + + if not _reapply_install(user): + n = _unpark_all(user) + printError(f" --clean: NOT DONE -- put {n} file(s) back as they were.") + printInfo(" The upgrade itself is installed; only the clean-up was undone.") + printInfo(" Fix the install step, then: packagemanager update " + f"{user[2:] if user.startswith('p_') else user} --reinstall --clean --run") + write_pkg_list(user, background=False) + return 0 + + still = [p for p in parked if os.path.lexists(p)] + gone = [p for p in parked if not os.path.lexists(p)] + # both kinds of parked copy go: for `still` the install wrote the + # current one, for `gone` nothing replaced it -- that is the orphan + remove_paths_as_user(user, [p + _PARK_SUFFIX for p in parked] + dirs) + try: + os.unlink(_park_ledger(user)) + except OSError: + pass + if still: + printInfo(f" --clean: {len(still)} of them are still installed by this " + f"version (unchanged, so the install left their old date) -- kept") + if gone: + printInfo(f" --clean: removed {len(gone)} file(s) the new version no " + f"longer installs") + for p in gone[:10]: + printInfo(" " + p) + if len(gone) > 10: + printInfo(" ... and %d more" % (len(gone) - 10)) + else: + printInfo(" --clean: nothing was an orphan after all") + printInfo(" --clean: regenerating the file manifest...") + write_pkg_list(user, background=False) + return len(gone) + + +def _is_phased_script(text): + """Does this install script hand its phases to the runner? + + The runner is what fetches, unpacks, resumes in the build directory and + re-runs one phase -- so "phased" means "ends in pkgusr_run", not "defines + some particular function". Three spellings, because the header and the + library name are each written by a different generation of the tools. + """ + t = text or "" + return bool(re.search(r"\bpkgusr_run\b", t) + or re.search(r"PKGUSR_LIB|lfs-phases", t) + or re.search(r"^###\s*phases:", t, re.M)) + + +def _is_git_package(name): + """A package whose install script builds from a git repo (is_git=1, or it + clones/pulls) rather than a book tarball.""" + p = (_canonical_script_path(name) + or os.path.join(pkgusr_home(name), f"install_{name}")) + if not p or not os.path.isfile(p): + return False + try: + t = open(p, encoding="utf-8", errors="replace").read(8000) + except OSError: + return False + return bool(re.search(r'is_git="?1"?', t) + or re.search(r'\bgit\s+(clone|pull|fetch)\b', t)) + + +# A plan entry whose "script" is this is built by `lfs`, not by us: the LFS +# book owns it, and lfs already knows how to build one of its packages. +_LFS_STEP = ":lfs-book:" + +_TOOLCHAIN_RISK = ("glibc", "binutils", "gcc", "libstdcpp", "ld", "linux-headers") + + +def _lfs_stale_list(include_current=False): + """[(name, installed, book)] for packages the LFS book has newer. + + With include_current, the packages whose version did NOT move are listed + too (installed == book). After glibc changes, "up to date" and "built + against the current system" stop being the same statement -- so a full + rebuild in book order is a thing a person may legitimately ask for. + + Read from `lfs update --porcelain` -- one tab-separated fact per line. + Parsing the human report instead would break the first time a word in it + changed, and this tool has been bitten by exactly that before.""" + lfs_bin = shutil.which("lfs") or ("/usr/bin/lfs" if os.access( + "/usr/bin/lfs", os.X_OK) else None) + if not lfs_bin: + return [] + try: + _cmd = [lfs_bin, "update", "--porcelain"] + if include_current: + _cmd.append("--include-current") + r = subprocess.run(_cmd, capture_output=True, text=True, timeout=600) + except Exception: + return [] + out = [] + for ln in (r.stdout or "").splitlines(): + parts = ln.split("\t") if len(parts) == 3 and parts[0]: out.append((parts[0], parts[1], parts[2])) return out @@ -4501,9 +5989,13 @@ def _report_no_book(no_book, consult_lfs, no_record=(), unjudged=(), print() if no_record: printInfo(f" {len(no_record)} installed, but no recorded version " - f"(built before packagemanager, or its record was lost):") - printInfo(" " + ", ".join(sorted(no_record)[:12]) - + (" ..." if len(no_record) > 12 else "")) + f"here -- what the book carries is shown beside each:") + for _nr in sorted(no_record)[:12]: + printInfo(" " + _nr) + if len(no_record) > 12: + printInfo(" ... and %d more" % (len(no_record) - 12)) + printInfo(" the installed version, if the script still names it:" + " packagemanager info <name>") if unjudged: printInfo(f" {len(unjudged)} with no completed-install record here " f"(the LFS build tracks these in its own manifests):") @@ -4576,7 +6068,78 @@ def _systemctl_users(steps): return out +def _machine_avoids(): + """Everything this machine never builds, from avoid.stack. + + `avoid` entries were only consulted by `stack`, so `install` and + `update` walked the book's dependency graph with nothing excluded. The + exclusions describe the MACHINE -- that is why they were moved into + their own file in 1.14.224 -- so every plan should honour them. + """ + out = set() + d = STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks" + for f in sorted(glob.glob(os.path.join(d, "*.stack"))): + try: + for r in _parse_stack_file(f): + if r["kind"] == "avoid": + out.add(r["target"]) + except (ValueError, OSError): + continue + return out + + +def _stack_targets(name): + """The packages a stack builds -- book/git/tar entries, not checks.""" + path = name if os.path.sep in name else os.path.join( + STACK_DIR if "STACK_DIR" in globals() else "/etc/pkgusr/stacks", + name if name.endswith(".stack") else name + ".stack") + if not os.path.isfile(path): + printError("no such stack: %s (%s)" % (name, path)) + return None + try: + rows = _parse_stack_file(path) + except ValueError as e: + printError(str(e)) + return None + # a stack on another book: refresh it once, and make sure the entries + # read it even when this file is not the installed copy + _sb = {r["target"]: r["opts"]["from-book"] for r in rows + if r["kind"] == "book" and r["opts"].get("from-book")} + if _sb: + _stack_book_map().update({k.lower(): v for k, v in _sb.items()}) + _BOOK_FOR_PKG.clear() + os.environ["PM_STACK_BOOKS"] = ",".join( + "%s=%s" % kv for kv in sorted(_sb.items())) + for _sel in sorted(set(_sb.values())): + _book_path_for(_sel, refresh=True) + printInfo(" %d package(s) of this stack read the %s book" + % (list(_sb.values()).count(_sel), _sel)) + return [r["target"] for r in rows + if r["kind"] in ("book", "git", "tar")] + + def cmd_update(args): + # "UPDATE THE MEDIA CHAIN" WITHOUT TYPING EIGHTEEN NAMES. + # + # `stack media --run` builds what is MISSING and never updates what is + # installed -- correct, and not what someone means by "update the media + # stack". The other end, `update webkitgtk`, is one package. Between + # them there was nothing but a list of names to type out. + if getattr(args, "stack", None): + _tg = _stack_targets(args.stack) + if _tg is None: + return 2 + if args.packages: + printInfo(" --stack %s and named packages: using the named ones" + % args.stack) + else: + args.packages = _tg + printInfo(" --stack %s: %d package(s) considered -- %s" + % (args.stack, len(_tg), ", ".join(_tg[:8]) + + (" ..." if len(_tg) > 8 else ""))) + printInfo(" only those the book has moved past will appear in " + "the plan.") + printInfo(" pick from them instead of taking all: --select") # --local: update a single package from a custom/local install script # (e.g. a git-based package). if getattr(args, "local", None): @@ -4602,10 +6165,61 @@ def cmd_update(args): printWarning(f"{name}: not a git package and not in the book -- " f"skipping (check the name, or use --local).") continue - if "unpack_pkg" not in open(script, errors="replace").read(): - printWarning(f"{name}: git script isn't phased -- it won't fetch/" - f"rebuild cleanly. Regenerate it with:\n" - f" packagemanager template {name} --git <repo-url>") + # IS IT PHASED? ASK THE RIGHT QUESTION. + # + # This looked for `unpack_pkg`, which is the RARE OVERRIDE a script + # defines to replace the built-in unpack entirely. A correct, + # current script does not define it -- the runner's own unpack is + # what clones git -- so every properly phased package was told its + # script "isn't phased", with advice (`template --git`) that + # overwrites the file and takes any hand-tuned build_pkg with it: + # meson setup build ... -D c_link_args=-licuuc + # A phased script is one that ENDS IN THE RUNNER. That is the + # thing that makes fetch/rebuild work, and it is what to test. + _body = open(script, errors="replace").read() + if not _is_phased_script(_body): + printWarning(f"{name}: this script does not end in the phase " + f"runner, so `update` cannot fetch and rebuild it " + f"cleanly.") + printInfo(" a phased script ends with:") + printInfo(' . "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"') + printInfo(" and puts its commands in build_pkg / install_pkg.") + printInfo(f" `packagemanager template {name} --git <url>` writes" + f" a fresh one -- to a NEW file first, then move your") + printInfo(" build commands across: it does not keep them.") + # SAY WHETHER THERE IS ANYTHING TO UPDATE TO. + # + # "git package -- updating from its repo" is true of a package + # pinned to the newest tag, where the rebuild produces the same + # commit it already has: the command works, does exactly what it + # says, and changes nothing -- and there is no way to tell that + # from the output. The remote knows; ask it. + _st = None if getattr(args, "no_git_check", False) \ + else _git_pkg_state(name) + if _st: + if _st["kind"].endswith("branch") and _st["remote"]: + if _st["local"][:12] == _st["remote"][:12]: + printInfo(" %s: %s is at %s, the same commit as here " + "-- this rebuilds what is already installed" + % (name, _st["ref"], _st["remote"][:8])) + else: + printInfo(" %s: %s moved %s -> %s%s -- this brings it in" + % (name, _st["ref"], (_st["local"] or "?")[:8], + _st["remote"][:8], + (" (%d commit(s))" % _st["behind"]) + if _st["behind"] else "")) + elif _st["kind"] == "tag": + if _st["newer_tags"]: + printInfo(" %s: pinned to tag %s; %s is out" + % (name, _st["ref"], _st["newer_tags"][-1])) + printInfo(" a rebuild stays on %s. To move it:" + % _st["ref"]) + printInfo(" packagemanager git-ref %s %s --run" + % (name, _st["newer_tags"][-1])) + else: + printInfo(" %s: pinned to tag %s, which is the newest " + "released -- this rebuilds the same commit" + % (name, _st["ref"])) printInfo(f"{name}: git package -- updating from its repo.") install_local(name, script, args.run, args.yes, "update", test=getattr(args, "test", False), @@ -4624,6 +6238,7 @@ def cmd_update(args): cached = use_stored_plan("update", plan_key, assume_yes=args.yes) if args.run else None no_book, no_record, unjudged, uptodate = [], [], [], [] + git_tracked = [] _checked = 0 if cached is not None: @@ -4656,6 +6271,10 @@ def cmd_update(args): vlookup = book_versions_all() names = list_all_users() + # a walk from a stack-book target carries that book's versions for + # every dependency; the ones it does not own read their own book + _stack_walk = getattr(args, "recursive", False) and any( + _book_override(t) for t in (args.packages or [])) _bk = _current_blfs_book() _checked = len(names) printInfo(f"Checking {len(names)} package(s) against the book" @@ -4671,6 +6290,13 @@ def cmd_update(args): u = gather_user(name) anchor = u.meta.get("name") or name bv = vlookup.get(anchor) + # EACH PACKAGE AGAINST ITS OWN BOOK: a stack's from-book (or + # machine.conf's book=) -- the default map would "update" an + # svn-built mpv back to the stable page. + if _book_override(anchor): + bv = book_version(anchor) or bv + elif _stack_walk: + bv = default_book_version(anchor) or bv if u.state != "installed": if name in named or anchor in named: not_installed.append(name) @@ -4691,7 +6317,33 @@ def cmd_update(args): # same test as the comparison, where it came out falsy -- # and the package was filed as up to date. Not knowing is # not the same as being current. - no_record.append(name) + # + # BUT --reinstall IS NOT A QUESTION ABOUT VERSIONS. It says + # "build this again at the book's version", and the answer + # does not depend on what is recorded: with the version + # missing, `update curl --reinstall` reported "nothing to + # update, 1 it cannot judge" and did nothing, 28 times in a + # row, for packages whose records predate the tool. + if getattr(args, "reinstall", False): + plan.append((anchor, name, "(reinstall)", bv)) + else: + # ...AND SAY WHAT THE BOOK HAS. "no recorded version" + # answers half the question: the reader wants to know + # what they would get, and the plan already knows it. + no_record.append("%s (book: %s)" % (name, bv or "?")) + continue + # A GIT-TRACKED PACKAGE HAS NO VERSION TO COMPARE. + # + # gst-plugins-rs is a `git` stack entry following upstream, and + # its name_version is `gst-plugins-rs-git`. Against the book's + # 1.28.6 that read as "older", so an update plan offered to + # replace a cargo-built checkout with the book's page -- a + # different build entirely, silently. `-git` is not a version + # and cannot be behind one. + _is_git_tracked = (str(u.version or "").endswith("-git") + or (u.meta or {}).get("is_git") in ("1", 1)) + if _is_git_tracked: + git_tracked.append((name, u.version, bv)) continue outdated = not _same_version(bv, u.version) if outdated: @@ -4711,6 +6363,49 @@ def cmd_update(args): printWarning(f"{name} is NOT installed (no completed install " f"found) -- use: packagemanager install {name}") + if git_tracked: + # "LEFT ALONE" IS NOT AN ANSWER TO "IS IT OUT OF DATE?" + # + # This said `-git` cannot be behind a book version, which is true + # and beside the point: a package that follows a BRANCH is behind + # whenever the branch moves, and the remote can be asked in a + # second. A package pinned to a TAG is not behind at all -- the + # question there is whether a newer tag exists, and taking it is a + # decision, not an update. Both are answered here rather than + # left to the reader. + print() + printInfo(" %d package(s) follow GIT, not the book:" + % len(git_tracked)) + _moved, _tagged = [], [] + for _n, _iv, _bv in git_tracked: + _st = _git_pkg_state(_n) if not getattr(args, "no_git_check", + False) else None + if _st: + print(_git_pkg_line(_st)) + if _st["kind"].endswith("branch") and _st["remote"] \ + and _st["local"][:12] != _st["remote"][:12]: + _moved.append(_n) + if _st["newer_tags"]: + _tagged.append(_st) + else: + printInfo(" %-22s tracked as %-20s (the book has %s)" + % (_n, _iv, _bv or "no page")) + if _moved: + printInfo(" behind their branch -- fetch, rebuild, install:") + printInfo(" packagemanager update %s --run" % " ".join(_moved)) + for _st in _tagged: + printInfo(" %s is pinned to %s, and %s is out:" + % (_st["name"], _st["ref"], _st["newer_tags"][-1])) + printInfo(" packagemanager git-ref %s %s --run" + % (_st["name"], _st["newer_tags"][-1])) + if not _moved and not _tagged: + printInfo(" every one of them is where its ref says it" + " should be") + printInfo(" the whole picture: packagemanager git-status") + printInfo(" (taking the BOOK's page instead would be a" + " different build:") + printInfo(" `packagemanager update <pkg> --reinstall --run`.)") + # named packages that are already current: offer to re-apply for anchor, name, iv, bv in uptodate: if name not in named and anchor not in named: @@ -4719,6 +6414,15 @@ def cmd_update(args): printInfo(f"{name} is already up to date ({bv}); skipping " f"(--no-reinstall).") continue + # A DRY RUN ASKS NOTHING. `update --stack media` prompted four + # times -- once per already-current package -- before printing a + # plan it was never going to carry out. A preview that needs + # answers is not a preview. + if not getattr(args, "run", False): + printInfo(f" {name} is already current ({bv}) -- not in the " + f"plan. To rebuild it anyway:") + printInfo(f" packagemanager update {name} --reinstall --run") + continue if args.yes or ask_yes_no(f"{name} is already up to date ({bv}). " f"Re-apply the update anyway?"): plan.append((anchor, name, "(reinstall)", bv)) @@ -4922,6 +6626,27 @@ def cmd_update(args): "--clean" if getattr(args, "clean", False) else "", "-e" if args.ignore_recommended else "", "--optional" if args.optional else "") if f] + ["--run"]) + _report_stale_requires([st[1] for st in steps], + _verbose=getattr(args, "verbose", False)) + # WHERE THE WHOLE LIST IS. Asking about one package tells you + # nothing about the rest, and the command for "everything the + # book has moved on from" is this one with no name after it. + # WHAT WILL ACTUALLY CHANGE, in one line. The requirement + # report above lists a dozen packages and reads like a plan; it + # is not one, and saying so beats hoping the header is noticed. + printInfo(" this plan updates %d package(s): %s" + % (len(steps), ", ".join(st[1] for st in steps[:10]) + + (" ..." if len(steps) > 10 else ""))) + printInfo(" nothing else is touched -- the requirements above" + " are a comparison, not a list of work.") + if args.packages: + printInfo(" other ways to choose:") + printInfo(" packagemanager update #" + " everything the book has moved past") + printInfo(" packagemanager update --stack media #" + " just that stack's packages") + printInfo(" packagemanager update --stack media --select" + " --run # pick from them") printInfo(f"Dry run -- this exact command runs THIS plan ({len(steps)} " f"package(s)):\n {exact}") printInfo(f"(cached under key '{plan_key}', so --run reuses/resumes it).") @@ -5143,91 +6868,677 @@ def cmd_plan(args): "clear all: packagemanager plan clear") -def cmd_errors(args): - """Show which packages failed in the LAST install/update run, with their log - locations. With --all, show every package's result.""" - log = read_run_log() - if not log: - printInfo("No install/update run recorded yet.") - return - entries = log.get("entries", []) - fails = [e for e in entries if not e["ok"]] - oks = [e for e in entries if e["ok"]] - printHeader(f"Last run: {log.get('kind')} {log.get('key', '')}") - print(f"\n {_ago(log.get('started', 0))} -- " - f"{len(oks)} ok, {colors.red}{len(fails)} failed{colors.normal}, " - f"{len(entries)} total\n") - show = entries if getattr(args, "all", False) else fails - if not show: - printSuccess("No errors in the last run.") - return - for e in show: - mark = (f"{colors.green}ok{colors.normal}" if e["ok"] - else f"{colors.red}FAILED (exit {e['exit']}){colors.normal}") - print(f" {e['pkg']:<26} {e['action']:<8} {mark}") - print(f" log: {e['log_dir']}") - if fails: - newest = max(fails, key=lambda e: e["time"]) - printInfo(f"\n tip: view the newest failing log with:\n" - f" ls -t {newest['log_dir']} | head -1 # then less that file") - -def set_status(msg): - """Show a persistent one-line status at the bottom (progress across many - packages). No-op when stdout isn't a terminal so it can't corrupt logs.""" - if sys.stdout.isatty(): - sys.stdout.write("\r\033[K" + msg[:120]) - sys.stdout.flush() +# --------------------------------------------------------------------------- +# WHO WAS BUILT BEFORE WHAT. +# +# Every generated script records the page's own required/recommended/optional +# lists, and every package's pkg.lst is written when it finished installing. +# Those two facts together answer the question that cost this system its ALSA +# and its WebRTC: was anything a package can USE installed after that package +# was built? Nothing else has to be inferred, guessed or remembered. +# --------------------------------------------------------------------------- + +def _pkg_dirs(): + root = os.path.join(BASE_DIR, load_config().get("pkgusr_subdir", "pkgusr")) + if not os.path.isdir(root): + root = BASE_DIR + out = {} + for d in sorted(os.listdir(root)): + full = os.path.join(root, d) + if not os.path.isdir(full): + continue + if not os.path.isfile(os.path.join(full, "install_last")): + continue + out[re.sub(r"^" + re.escape(pkgusr_prefix()), "", d)] = full + return out -def clear_status(): - if sys.stdout.isatty(): - sys.stdout.write("\r\033[K") - sys.stdout.flush() +def _built_when(home): + """When this package last finished installing. + pkg.lst is written after the install, so its mtime is the install time. + install_last is the fallback: generated just before the run. + """ + for f in ("pkg.lst", "install_last"): + p = os.path.join(home, f) + try: + return os.path.getmtime(p), f + except OSError: + continue + return None, "" -def _versioned_script_in(home): - """The editable install_<name>-<version> in a package user's home. - That is the file a person is told to edit, and the one the copy that - actually runs is made from -- so a failure has to be able to name it. - """ +def _script_deps(home): + """required/recommended/optional, as the page declared them.""" + out = {} try: - for f in sorted(os.listdir(home)): - if not f.startswith("install_") or f.endswith((".edited", ".bak")): - continue - # install_wget-1.25.0 -- a version after the last dash - if re.search(r"-\d[\w.]*$", f): - return os.path.join(home, f) + body = open(os.path.join(home, "install_last"), + encoding="utf-8", errors="replace").read() except OSError: - pass - return None + return out + for kind in ("required", "recommended", "optional"): + m = re.search(r"^%s=\((.*?)\)$" % kind, body, re.M | re.S) + if not m: + continue + out[kind] = re.findall(r"'([^']*)'", m.group(1)) + return out -def _unpacked_source_dir(home): - """Where the package unpacked, when there is exactly one candidate.""" - src = os.path.join(home, "src") - try: - dirs = [d for d in os.listdir(src) if os.path.isdir(os.path.join(src, d))] - except OSError: - return None - return os.path.join(src, dirs[0]) if len(dirs) == 1 else None +def _norm(s): + return re.sub(r"[^a-z0-9]", "", s.lower()) -def _pkg_log_dir(user): - return os.path.join(pkgusr_home(user), "log") +_DEP_IDX = {} -def _maybe_remove_old_sources(user, keep_name_version, keep_sources=False): - """After a successful (re)install, remove OLD unpacked source dirs and stale - downloaded archives in the package user's home, keeping the current version. - Removes automatically; pass keep_sources=True to skip.""" - if keep_sources: - return - home = pkgusr_home(user) - keep = keep_name_version - victims = [] - try: +def _dep_index(installed): + """title -> package, built from the packages themselves. + + A dep is written as the page's TITLE ('gst-plugins-base-1.28.1', + 'FreeType-2.14.1'); a package directory is the page's id + ('gst10-plugins-base', 'freetype2'). Normalising the strings cannot + bridge that, and 119 of them did not resolve -- which quietly broke the + REBUILD ORDER, because a dependency edge that does not resolve is an edge + that does not exist: gst10-plugins-bad sorted before gst10-plugins-base, + and fontconfig before freetype2. + + Every script already records `name_version` -- the tarball base, which is + the title -- so the mapping is in the data and does not have to be + guessed. + """ + key = tuple(sorted(installed)) + if _DEP_IDX.get("key") == key: + return _DEP_IDX["idx"] + idx = {} + for name, home in installed.items(): + idx.setdefault(_norm(name), name) + try: + body = open(os.path.join(home, "install_last"), + encoding="utf-8", errors="replace").read(4096) + except OSError: + continue + m = re.search(r"^name_version='([^']*)'", body, re.M) + if not m: + continue + base = re.sub(r"-[0-9][0-9a-z.]*$", "", m.group(1)) + idx.setdefault(_norm(base), name) + _DEP_IDX.update(key=key, idx=idx) + return idx + + +def _resolve_dep(dep, installed): + """A dep is 'Little-CMS-2.18'; a package dir is 'lcms2'. + + THE MAJOR VERSION IS PART OF THE NAME, sometimes. gtk3 and gtk4 both + record name_version 'GTK-<v>', so the base name 'GTK' matched whichever + was indexed first -- and `GTK-4.20.3` resolved to **gtk3**. webkitgtk + requires both, so it listed gtk3 twice and gtk4 not at all; the same + collision hides libsoup-3.x behind libsoup, sdl3 behind sdl, qt6 behind + qt. + + So try base+major BEFORE base: `GTK-4.20.3` -> gtk4, `libsoup-3.6.6` -> + libsoup3, and `Opus-1.6.1` -> opus1 does not exist, which falls through + to opus as before. + """ + idx = _dep_index(installed) + base = re.sub(r"-[0-9][0-9a-z.]*$", "", dep) + m = re.search(r"-([0-9]+)", dep[len(base):]) if len(dep) > len(base) else None + major = m.group(1) if m else "" + cands = [] + if major: + cands += [base + major, base.replace("-", "") + major, + base.split("-")[0] + major] + cands += [base, base.replace("-", ""), base.split("-")[0]] + for cand in cands: + n = _norm(cand) + if n and n in idx: + return idx[n] + return None + + +# DOCS AND TESTS ARE NOT FEATURES. +# +# graphviz appeared against wayland, llvm, ruby, libinput, libexif and +# libical; docutils against glib2 and libdrm. Rebuilding six packages to +# regenerate documentation nobody reads is not the point of this audit, and +# burying the real findings under it is how a report gets ignored. +_DOC_DEPS = {"graphviz", "docutils", "doxygen", "sphinx", "texlive", "asciidoc", + "asciidoctor", "dejagnu", "valgrind", "gtk-doc", "xmlto", "ruby", + "fop", "docbook-xsl", "docbook", "libxslt-docs", "dblatex"} +# ...and these are build-time conveniences, not linked features +_TOOL_DEPS = {"git", "cmake", "cython", "fcron", "meson", "ninja", "nasm", + "yasm", "expect", "which", "mako", "setuptools", "wheel", + "hatchling", "pytest", "gi-docgen", "gobject-introspection-docs", + "unifdef", "gperf"} + + +# INSTALL DOES NOT REBUILD. `packagemanager install <pkg> --run +# --regenerate` on an installed package answers "Nothing to do -- everything +# is installed and up to date" -- correctly, and 28 times in a row, so the +# whole rebuild pass did nothing and said so in a way that looked like +# success. The verb that rebuilds at the same version is `update +# --reinstall`. +def _rebuild_cmd(name): + return ["packagemanager", "update", name, "--reinstall", "--regenerate", + "--run", "--yes"] + + +def _required_graph(installed, kinds=("required",)): + """package -> the packages it depends on, from its own script.""" + g = {} + for name, home in installed.items(): + deps, d = set(), _script_deps(home) + for kind in kinds: + for dep in d.get(kind, []): + got = _resolve_dep(dep, installed) + if got and got != name: + deps.add(got) + g[name] = deps + return g + + +def _depends_on(g, a, b, seen=None): + """does a require b, directly or through anything?""" + if seen is None: + seen = set() + if a in seen: + return False + seen.add(a) + for d in g.get(a, ()): + if d == b or _depends_on(g, d, b, seen): + return True + return False + + +# A TIMESTAMP IS EVIDENCE; A PROBE IS AN ANSWER. +# +# Two packages built in the same session cannot be ordered by pkg.lst +# mtimes, and that is exactly where the interesting cases are (ffmpeg and +# alsa-lib, zero days apart). The stacks already carry `expect=` probes that +# read the built artifact -- run them. +def cmd_probe(args): + import glob as _glob # noqa: also used by the no-probes branch below + stacks = sorted(_glob.glob("/etc/pkgusr/stacks/*.stack")) + if not stacks: + printError("no stack files in /etc/pkgusr/stacks") + return 1 + n = ok = bad = 0 + for path in stacks: + for e in _parse_stack_file(path): + probe = e["opts"].get("expect") + if not probe: + continue + if args.package and e["target"] not in args.package: + continue + n += 1 + r = subprocess.run(["/bin/bash", "-c", probe], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + stack_name = os.path.basename(path).replace(".stack", "") + if r.returncode == 0: + ok += 1 + printSuccess("%-22s %s" % (e["target"], probe)) + continue + bad += 1 + printError("%-22s %s" % (e["target"], probe)) + # A PROBE THAT FAILS BECAUSE THE PACKAGE IS NOT THERE IS NOT THE + # SAME FINDING. gst10-libav had never been built -- it was + # missing from the stack until now -- and the report said "the + # feature is NOT in the installed package", which sends the + # reader looking for a build option. + home = pkgusr_home(e["target"]) + installed = os.path.exists(os.path.join(home, "pkg.lst")) + if not installed: + printInfo(" that package is NOT INSTALLED here -- nothing" + " was built without the feature,") + printInfo(" there is nothing built at all. Build it:") + printInfo(" packagemanager stack %s --only %s --run" + % (stack_name, e["target"])) + else: + printInfo(" the package is installed and the feature is" + " not in it. Rebuild:") + # not `stack --only`: that runs install, which no-ops on an + # installed package + printInfo(" %s" % " ".join(_rebuild_cmd(e["target"]))) + # ...and what the probe itself said, because a probe can be wrong + # about a path as easily as a package can be wrong about a feature + first = (r.stdout or b"").decode("utf-8", "replace").strip() + if first: + printInfo(" the check said: %s" % first.splitlines()[0][:120]) + if n == 0: + printInfo("no expect= probes in the stacks" + + (" for %s" % ", ".join(args.package) if args.package else "")) + # SAY WHY THERE ARE NONE. A stack file you have edited is never + # overwritten -- the new one lands beside it as .stack.new -- so an + # installed tool can carry probes that no installed STACK has, and + # "no probes" reads as "nothing to check" rather than "your stacks + # predate them". + waiting = [] + for newf in sorted(_glob.glob("/etc/pkgusr/stacks/*.stack.new")): + try: + if "expect=" in open(newf, encoding="utf-8", + errors="replace").read(): + waiting.append(newf) + except OSError: + continue + if waiting: + print("") + printWarning("%d newer stack file(s) beside yours DO carry probes:" + % len(waiting)) + for newf in waiting: + base = os.path.basename(newf).replace(".stack.new", "") + printInfo(" %s" % newf) + printInfo(" take it: packagemanager stack %s --take-new" + % base) + printInfo(" Your copy was edited, so make install left it alone" + " and put the new one beside it.") + else: + printInfo(" Probes live in the stack files as expect=\"...\"." + " These stacks have none,") + printInfo(" which is what an older stack looks like:" + " compare with the shipped one.") + return 0 + print("") + if bad: + printWarning("%d of %d probe(s) failed -- see each line: a package that" + " is not installed and one" % (bad, n)) + printWarning(" built without a feature are different problems.") + return 1 + printSuccess("all %d probe(s) pass: the features they check are present." % ok) + return 0 + + +def cmd_late_deps(args): + installed = _pkg_dirs() + if not installed: + printError("no package users with an install_last under %s" % BASE_DIR) + return 1 + kinds = [k.strip() for k in (args.kind or "recommended,optional").split(",") + if k.strip()] + # WHY DID IT DECIDE THAT? The order is computed from edges, and an edge + # that does not resolve is invisible -- so when a position looks wrong, + # the useful thing is the graph it actually used for that package. + if getattr(args, "why", False): + for name in (args.package or sorted(installed)): + if name not in installed: + printWarning("%s is not installed here" % name) + continue + when, src = _built_when(installed[name]) + print("") + printHeader(name) + print(" built: %s (%s)" + % (time.strftime("%Y-%m-%d %H:%M", + time.localtime(when)) if when else "unknown", src)) + d = _script_deps(installed[name]) + for kind in ("required", "recommended", "optional"): + for dep in d.get(kind, []): + got = _resolve_dep(dep, installed) + if got is None: + print(" %-12s %-28s -> no installed package of that name" + % (kind, dep)) + continue + dw, _ = _built_when(installed[got]) + if dw and when: + diff = dw - when + rel = ("built %+d day(s) relative to it" % (diff // 86400) + if abs(diff) >= 86400 else + "built %+d hour(s) relative to it" % (diff // 3600)) + else: + rel = "no date" + print(" %-12s %-28s -> %s, %s" % (kind, dep, got, rel)) + return 0 + only = set(args.package or []) + problems, unresolved, nodate = [], set(), [] + for name, home in installed.items(): + if only and name not in only: + continue + when, src = _built_when(home) + if when is None: + nodate.append(name) + continue + late = [] + for kind in kinds: + for dep in _script_deps(home).get(kind, []): + got = _resolve_dep(dep, installed) + if got is None: + unresolved.add(dep) + continue + if got == name: + continue + dwhen, _ = _built_when(installed[got]) + if dwhen is None: + continue + # SLACK THAT HIDES THE CASE IS TOO MUCH SLACK. + # + # A day was chosen because pkg.lst is rewritten by + # reload-pkg-list and by --clean, so packages from one run + # differ by minutes. But `ffmpeg ... alsa-lib +0 day(s)` was + # then filed as fine, and alsa-lib a few HOURS after ffmpeg + # is the original bug exactly. An hour covers the rewrite + # noise; anything more is a gap. + if dwhen - when > 3600: + late.append((kind, got, dep, dwhen - when)) + if late: + problems.append((name, when, src, late)) + + # THREE DIFFERENT ANSWERS, and the report was giving one. + # + # ORDERABLE the dependency does not need this package, so it can simply + # be built first -- a stack line moves and the problem is + # gone for the next machine. + # CIRCULAR the dependency REQUIRES this package (cairo needs glib2, + # mesa needs libxcb). No order exists. The only fix is a + # second pass: build, build the dependency, rebuild. + # COSMETIC documentation and build tools. Nothing to gain. + g = _required_graph(installed) + # RECOMMENDED IS A DEPENDENCY TOO, for the purpose of asking whether an + # order exists. With `required` alone the report said + # in the stack: put mesa before libxcb + # and mesa cannot precede libxcb -- it needs it. BLFS lists that edge as + # recommended, not required, and cairo needs glib2 the same way. Advice + # that cannot be taken is worse than no advice. + g_order = _required_graph(installed, ("required", "recommended")) + orderable, circular, cosmetic = [], [], [] + for name, when, src, late in problems: + o, c, d = [], [], [] + for item in late: + dep_pkg = item[1] + if dep_pkg in _DOC_DEPS or dep_pkg in _TOOL_DEPS: + d.append(item) + elif _depends_on(g_order, dep_pkg, name): + c.append(item) + else: + o.append(item) + if o: + orderable.append((name, when, src, o)) + if c: + circular.append((name, when, src, c)) + if d: + cosmetic.append((name, when, src, d)) + + def _show(rows, header, note): + if not rows: + return + print("") + printHeader(header) + printInfo(" " + note) + for name, when, src, late in sorted(rows, key=lambda r: -len(r[3])): + print("") + printWarning("%s (built %s)" + % (name, time.strftime("%Y-%m-%d", time.localtime(when)))) + for kind, got, _dep, gap in late: + when_s = ("%d day(s)" % (gap // 86400) if gap >= 86400 + else "%d hour(s)" % (gap // 3600)) + print(" %-12s %-22s installed %s later%s" + % (kind, got, when_s, + " <- same session: only a probe can settle it" + if gap < 21600 else "")) + # SAY WHAT TO CHANGE, not just what is wrong. The orderable ones + # are a stack edit; without the line the reader has to work out + # which file and which position. + if rows is orderable: + print(" in the stack: put %s before %s" + % (", ".join(sorted({g for _k, g, _d, _x in late})), name)) + + if not problems: + printSuccess("no package was built before something it can use.") + _show(orderable, "built too early -- and an order exists", + "move these dependencies earlier in the stack, then rebuild these once.") + _show(circular, "built too early -- no order exists", + "the dependency requires this package, so it cannot go first." + " A second pass is the only fix.") + if args.include_docs: + _show(cosmetic, "documentation and build tools only", + "rebuilding gains you documentation, not features.") + elif cosmetic: + print("") + printInfo("%d package(s) differ only in documentation or build tools" + " (--include-docs to see them)" % len(cosmetic)) + + # THE REBUILD ORDER IS NOT THE REPORT ORDER. Rebuilding a library after + # the thing that links it means doing it twice. + todo = {r[0] for r in orderable} | {r[0] for r in circular} + if args.include_docs: + todo |= {r[0] for r in cosmetic} + if todo: + # ORDER BY EVERY EDGE THAT DECIDES A FEATURE, not just the required + # ones: mesa RECOMMENDS libxcb, so libxcb has to be rebuilt first or + # mesa is rebuilt against the old one. Cycles are expected here + # (cairo and glib2 recommend each other) and are broken below. + # ORDER THE CLUSTERS, NOT THE NODES. + # + # Picking one node out of a cycle and then going back to the ready + # scan strands the rest of the cluster at the end: freetype2 and + # harfbuzz (which recommend/require each other) came out LAST, after + # fontconfig and cairo -- both of which require freetype2. So the + # cycle has to be treated as one unit. + # + # Tarjan gives the strongly connected components; the components form + # a DAG by construction, so they sort cleanly. Inside a component + # there is no right answer, and the same rule as before applies: + # least dependent, most depended on. + nodes = set(todo) + # THE OPTIONAL EDGE IS THE WHOLE POINT OF THIS PASS. + # + # The order graph was required+recommended, so mpg123 was rebuilt + # BEFORE pulseaudio -- and mpg123's missing feature IS pulseaudio. + # Ordering a rebuild by the edges that do not decide features, while + # ignoring the ones that do, cannot be right. + # + # Optional edges make the graph much more cyclic; that is honest + # rather than inconvenient, and SCCs handle it. Inside a component + # the STRONG edges (required, recommended) still decide, because + # those are the ones that break a build rather than a feature. + g_all = _required_graph(installed, + ("required", "recommended", "optional")) + adj = {n: (g_all.get(n, set()) & nodes) for n in nodes} + strong = {n: (g_order.get(n, set()) & nodes) for n in nodes} + index, low, on_stack, stack, sccs, counter = {}, {}, set(), [], [], [0] + + def _strong(v): + # iterative: a 400-package graph would recurse past the limit + work = [(v, iter(sorted(adj[v])))] + index[v] = low[v] = counter[0]; counter[0] += 1 + stack.append(v); on_stack.add(v) + while work: + node, it = work[-1] + advanced = False + for w in it: + if w not in index: + index[w] = low[w] = counter[0]; counter[0] += 1 + stack.append(w); on_stack.add(w) + work.append((w, iter(sorted(adj[w])))) + advanced = True + break + if w in on_stack: + low[node] = min(low[node], index[w]) + if advanced: + continue + work.pop() + if work: + low[work[-1][0]] = min(low[work[-1][0]], low[node]) + if low[node] == index[node]: + comp = [] + while True: + w = stack.pop(); on_stack.discard(w); comp.append(w) + if w == node: + break + sccs.append(comp) + + for n in sorted(nodes): + if n not in index: + _strong(n) + + # Tarjan emits each component only after everything it depends on, + # so this is already dependency order. + comp_of = {n: i for i, c in enumerate(sccs) for n in c} + ordered, cycles = [], [] + + def _inside(c): + cs = set(c) + return sorted(c, key=lambda n: (len(strong[n] & cs), + len(adj[n] & cs), + -sum(1 for m in cs if n in strong[m]), + -sum(1 for m in cs if n in adj[m]), + n)) + + for comp in sccs: + if len(comp) > 1: + cycles.extend(_inside(comp)) + ordered.extend(_inside(comp)) + else: + ordered.extend(comp) + + print("") + printHeader("rebuild, in this order (%d)" % len(ordered)) + printInfo(" dependencies first, so nothing is rebuilt twice.") + for n in ordered: + print(" %s" % " ".join(_rebuild_cmd(n))) + if cycles: + print("") + printInfo(" %d of these are in dependency cycles, where no order" + " is completely right:" % len(cycles)) + printInfo(" " + ", ".join(cycles)) + printInfo(" each was placed by fewest unmet dependencies, then by" + " how many wait for it.") + printInfo(" If a feature is still missing afterwards, run" + " late-deps again: a cycle") + printInfo(" can need two passes, and the second one is short.") + if args.rebuild: + print("") + if not args.run: + printInfo("that is the plan; add --run to do it.") + return 1 + for i, n in enumerate(ordered, 1): + printHeader("rebuild %d/%d: %s" % (i, len(ordered), n)) + rc = subprocess.run(_rebuild_cmd(n)).returncode + if rc != 0: + printError("%s failed (exit %d) -- stopping here, with" + " %d left." % (n, rc, len(ordered) - i)) + printInfo(" fix it, then run late-deps --rebuild --run again:" + " what succeeded is recorded and will not repeat.") + return 1 + printSuccess("rebuilt %d package(s)." % len(ordered)) + return 0 + if unresolved and args.verbose: + print("") + printInfo("could not match to an installed package (so not checked): %s" + % ", ".join(sorted(unresolved))) + elif unresolved: + print("") + printInfo("%d dependency name(s) match no INSTALLED package" + " (--verbose lists them)." % len(unresolved)) + printInfo(" Mostly that means they are not installed, which is" + " nothing to fix -- a package") + printInfo(" that is not there cannot have been built after anything." + " A name you know IS") + printInfo(" installed appearing there is worth reporting: it is an" + " edge this cannot see.") + if nodate: + printInfo("%d package(s) have no pkg.lst and no install_last date" + % len(nodate)) + return 1 if problems else 0 + +def cmd_errors(args): + """Show which packages failed in the LAST install/update run, with their log + locations. With --all, show every package's result.""" + log = read_run_log() + if not log: + printInfo("No install/update run recorded yet.") + return + entries = log.get("entries", []) + fails = [e for e in entries if not e["ok"]] + oks = [e for e in entries if e["ok"]] + printHeader(f"Last run: {log.get('kind')} {log.get('key', '')}") + print(f"\n {_ago(log.get('started', 0))} -- " + f"{len(oks)} ok, {colors.red}{len(fails)} failed{colors.normal}, " + f"{len(entries)} total\n") + show = entries if getattr(args, "all", False) else fails + if not show: + printSuccess("No errors in the last run.") + return + for e in show: + mark = (f"{colors.green}ok{colors.normal}" if e["ok"] + else f"{colors.red}FAILED (exit {e['exit']}){colors.normal}") + print(f" {e['pkg']:<26} {e['action']:<8} {mark}") + print(f" log: {e['log_dir']}") + if fails: + newest = max(fails, key=lambda e: e["time"]) + printInfo(f"\n tip: view the newest failing log with:\n" + f" ls -t {newest['log_dir']} | head -1 # then less that file") + + +def set_status(msg): + """Show a persistent one-line status at the bottom (progress across many + packages). No-op when stdout isn't a terminal so it can't corrupt logs.""" + if sys.stdout.isatty(): + sys.stdout.write("\r\033[K" + msg[:120]) + sys.stdout.flush() + + +def clear_status(): + if sys.stdout.isatty(): + sys.stdout.write("\r\033[K") + sys.stdout.flush() + + +def _versioned_script_in(home): + """The editable install_<name>-<version> in a package user's home. + + That is the file a person is told to edit, and the one the copy that + actually runs is made from -- so a failure has to be able to name it. + """ + # THE ONE THAT WAS LAST INSTALLED, NOT THE FIRST IN THE ALPHABET. + # + # A home keeps every version's script: p_mesa has install_Mesa-25.3.5 + # and install_Mesa-26.1.7. sorted() returned 25.3.5, so `script install + # mesa` ran the OLD version's install over a 26.1.7 build ("9" also sorts + # after "10"). install_last names the version last staged; take that + # file. Without it, the newest by modification time. + try: + cands = [f for f in os.listdir(home) + if f.startswith("install_") and not f.endswith((".edited", ".bak")) + and re.search(r"-\d[\w.]*$", f)] + except OSError: + return None + if not cands: + return None + try: + m = re.search(r"^name_version=['\"]([^'\"]+)['\"]", + open(os.path.join(home, "install_last"), + encoding="utf-8", errors="replace").read(), re.M) + if m and ("install_" + m.group(1)) in cands: + return os.path.join(home, "install_" + m.group(1)) + except OSError: + pass + return max((os.path.join(home, f) for f in cands), key=os.path.getmtime) + + +def _unpacked_source_dir(home): + """Where the package unpacked, when there is exactly one candidate.""" + src = os.path.join(home, "src") + try: + dirs = [d for d in os.listdir(src) if os.path.isdir(os.path.join(src, d))] + except OSError: + return None + return os.path.join(src, dirs[0]) if len(dirs) == 1 else None + + +def _pkg_log_dir(user): + return os.path.join(pkgusr_home(user), "log") + + +def _maybe_remove_old_sources(user, keep_name_version, keep_sources=False): + """After a successful (re)install, remove OLD unpacked source dirs and stale + downloaded archives in the package user's home, keeping the current version. + Removes automatically; pass keep_sources=True to skip.""" + if keep_sources: + return + home = pkgusr_home(user) + keep = keep_name_version + victims = [] + try: for fn in os.listdir(home): p = os.path.join(home, fn) if fn.startswith(".") or fn.startswith("install_") or fn == "log": @@ -6688,6 +8999,13 @@ def _cmd_config_interactive(args): if args.main_user: cfg["main_user"] = args.main_user changed = True + if getattr(args, "shared_hub", None): + _h = os.path.abspath(args.shared_hub).rstrip("/") + if cfg.get("shared_hub") != _h: + cfg["shared_hub"] = _h + changed = True + printInfo(" the links move the next time the accounts are set up:") + printInfo(" packagemanager stack services --only app-users --run") # Nothing given on the command line and something is unset? Ask. Leaving # a setting blank and carrying on means it surfaces much later as a failed # install with no obvious cause. @@ -6721,6 +9039,8 @@ def _cmd_config_interactive(args): print(f" user prefix : {cfg['user_prefix']}_ (shared users)") print(f" main user : {cfg.get('main_user') or '(not set)'} " f"(your human login account)") + print(f" shared hub : {shared_hub()} " + f"(links to every shared directory)") print(f" editor : {editor()}") print(f" difftool : {difftool()}") @@ -6766,8 +9086,8 @@ player cannot read your documents, your keys, or another application's data. Creating the account is the easy part; what makes it usable is the rest: - * your home directory stays yours -- the account's home is 0750, so the - application cannot read it + * your home directory stays yours, and the account's stays its own: the + account's home is 0700 * your login joins the account's group, so a shared directory works * `su - u_<name>` from your account without a password, where the hint's /etc/pam.d/su_u_user is installed (skipped where it is not: su then @@ -7394,14 +9714,16 @@ def _bootstrap_next_steps(st): def cmd_user_create(args): + if _resolve_shared_level(args) is None: + return 2 u = _user_prefixed(args.name) if _user_exists(u): printInfo(f"{u}: already exists.") else: printHeader(f"Create shared user {u}") print() - cmd = find_add_package_user() + [u, u, "10000", "20000", - u, "10000", "20000"] + cmd = find_add_package_user() + [u, u, str(APP_UID_MIN), str(APP_UID_MAX), + u, str(APP_UID_MIN), str(APP_UID_MAX)] printInfo(f"Creating shared user: {' '.join(cmd)}") if not _run(cmd): return @@ -7410,11 +9732,83 @@ def cmd_user_create(args): if args.groups: _add_groups(u, [g for g in re.split(r"[,\s]+", args.groups) if g]) + # AN APPLICATION USER IS NOT A PACKAGE USER, AND ITS HOME IS NOT IN + # /usr/src. + # + # `add_package_user` is what creates the account -- it handles the uid + # range and the matching group -- and it puts the home where PACKAGE + # homes go: + # /usr/src/u_claude, /usr/src/u_browser-mini, ... + # A package user's home is a build tree. An application user's home is + # a home: it holds a browser profile, a .config, a Downloads directory. + # It belongs in /home, which is also where the shared-directory links + # and the XDG skeleton expect it. + _relocate_app_home(u) + # An account on its own is not usable: you cannot become it, its home is # world-readable, and nothing launches the program. Do the rest. _setup_application_user(u, args) +def _relocate_app_home(u): + """Move an application user's home from the package tree to /home. + + Only from the package-user base, only when the target does not already + exist, and with `usermod -m` so the contents come along. Anything else + is left exactly as it is: a home someone has already set up elsewhere + is not this function's business. + """ + cur = _home_of(u) or "" + root = appuser_home() + want = os.path.join(root, u) + if not cur or os.path.realpath(cur) == os.path.realpath(want): + return + # Moved from where these tools put it before, and nowhere else: the + # package tree, or /home/<user> (the old hard-coded place). A home + # someone chose elsewhere is not this function's business. + base = os.path.realpath(BASE_DIR) + rc = os.path.realpath(cur) + if not (rc.startswith(base + os.sep) + or rc == os.path.realpath(os.path.join("/home", u))): + printInfo(" %s's home is %s -- left alone (not a place these tools" + " put it)" % (u, cur)) + return + if os.path.exists(want): + printWarning(" %s already exists -- not moving %s's home there" + % (want, u)) + printInfo(" check it, then: usermod -d %s -m %s" % (want, u)) + return + if not os.path.isdir(root): + _run(["install", "-d", "-o", "root", "-g", "root", "-m", "0755", root]) + printInfo(" %s's home is %s -- application users live in %s" + % (u, cur, root)) + if _run(["usermod", "-d", want, "-m", u]): + _run(["chmod", "0750", want]) + printSuccess(" moved to %s (0750)" % want) + _repoint_shared_link(u, cur, want) + else: + printWarning(" could not move it (is %s logged in or running" + " something?); by hand:" % u) + printInfo(" usermod -d %s -m %s && chmod 0750 %s" % (want, u, want)) + + +def _repoint_shared_link(u, old_home, new_home): + """~/Shared/<Name> of the main user pointed into the old home.""" + hub = shared_hub() + if not hub or not os.path.isdir(hub): + return + for n in os.listdir(hub): + p = os.path.join(hub, n) + if not os.path.islink(p): + continue + t = os.readlink(p) + if t.startswith(old_home.rstrip("/") + "/"): + nt = new_home.rstrip("/") + t[len(old_home.rstrip("/")):] + os.unlink(p) + os.symlink(nt, p) + printInfo(" %s -> %s" % (p, nt)) + + def _xdg_runtime_dir_of(user): """The main user's XDG runtime directory, however this system names it.""" try: @@ -7431,37 +9825,50 @@ def _xdg_runtime_dir_of(user): return None +def xdg_runtime_group(): + """The one group that opens the main user's session to shared accounts.""" + return collector_prefix() + "xdg-runtime" + + def _join_xdg_runtime_group(main, u): - """Put an application user in the group that owns the session. + """Put an application user -- and the main user -- in the session group. A "shared user" is one that may reach the main user's XDG runtime - directory -- that is where the display socket, the session bus and the - audio socket live, so without it a graphical program starts and then - cannot draw anything. - - The group is read from the directory rather than assumed: it is set up - once for the main user, and every shared account afterwards simply joins - whatever is there.""" + directory: the display socket (and pipewire's) live there, so without it + a graphical program starts and then cannot draw anything. + + THE GROUP IS NAMED, NOT READ OFF THE DIRECTORY. This used to take + whatever group /tmp/xdg-<main> carried -- which is the main user's own + private group -- refuse, and say "set up a runtime group first", which + nothing did. <prefix>_xdg-runtime is created here (collector range), the + main user joins it (a user may only chgrp to a group it is in) and so does + every shared account. The directory is recreated at every boot, so its + group is applied by the session itself: /usr/bin/xdg-runtime-share and + /etc/profile.d/xdg-runtime-share.sh, written by `cfg app-users`. + """ + g = xdg_runtime_group() + if not _create_collector_group(g): + printWarning(f" could not create the session group {g} -- {u} will " + f"not reach the session") + return + for who in (main, u): + _run(["usermod", "-a", "-G", g, who]) + printInfo(f" {main} and {u} are in '{g}' (the session group)") rt = _xdg_runtime_dir_of(main) if not rt: - printWarning(f" no XDG runtime directory found for {main} -- " - f"{u} will not reach the session") - printInfo(" (looked for /run/user/<uid> and /tmp/xdg-<user>)") + printInfo(f" no runtime directory for {main} right now -- the session " + f"applies the group when it starts") return try: import grp - grp_name = grp.getgrgid(os.stat(rt).st_gid).gr_name + cur = grp.getgrgid(os.stat(rt).st_gid).gr_name except (KeyError, ImportError, OSError): - printWarning(f" cannot read the group on {rt}") - return - if grp_name in (main, "root"): - printWarning(f" {rt} is group '{grp_name}', which is not a shared " - f"group -- {u} was not added") - printInfo(" Set up a runtime group for the session first, then " - "re-run this.") - return - _run(["usermod", "-a", "-G", grp_name, u]) - printInfo(f" {u} joined '{grp_name}' (the group on {rt})") + cur = "" + if cur != g: + # root may do it now; the session does it on every later login + _run(["chgrp", g, rt]) + _run(["chmod", "0750", rt]) + printInfo(f" {rt}: group {g}, 0750 (was {cur or '?'})") XDG_SKEL = "/etc/pkgusr/skel-u_xdg/.bash_profile" @@ -7505,7 +9912,9 @@ def _link_xdg_profile(main, u): printWarning(f" could not write {XDG_SKEL}: {e}") return - home = pkgusr_home(u) + # the ACCOUNT's home, not a package home: this linked + # /usr/src/u_claude/.bash_profile + home = _app_home(u) target = os.path.join(home, ".bash_profile") if os.path.islink(target) and os.path.realpath(target) == \ os.path.realpath(XDG_SKEL): @@ -7527,12 +9936,12 @@ def _setup_application_user(u, args): Creating the account is the easy part. What makes it useful: * your account may `su - <user>` without a password (via PAM) - * the home is 0750, so other users cannot read it + * the home is 0700, so no other account can read it * your account joins the user's group, for the shared directory * an optional shared directory both accounts can write * an optional launcher in ~/bin so the program runs as that user """ - home = os.path.join(appuser_home(), u) + home = _app_home(u) main = main_user() if not main: printWarning("\nNo main user is set, so this account cannot be reached " @@ -7541,10 +9950,16 @@ def _setup_application_user(u, args): printWarning(" packagemanager config --main-user <your-login>") return - # 1. the home is private + # 1. the home is private -- 0700, NOT 0750 (1.14.315) + # + # 0750 let the account's GROUP in, and your login is in that group (for + # the shared directory). So "a browser cannot read your documents" held, + # and the reverse did not: you could read its profile and cookies, and + # so could anything running as you. The shared directory lives in the + # hub now, so nothing needs to pass through the home. if os.path.isdir(home): - _run(["chmod", "0750", home]) - printInfo(f" home {home} is 0750 (private to {u} and its group)") + _run(["chmod", "0700", home]) + printInfo(f" home {home} is 0700 (private to {u})") # 2. your account joins the user's group -- this is what makes a shared # directory work, and what lets PAM identify you as entitled @@ -7563,37 +9978,82 @@ def _setup_application_user(u, args): _link_xdg_profile(main, u) # 5. a shared directory, when asked for + # + # THE DIRECTORY LIVES IN THE HUB, THE LINK LIVES IN THE HOME (1.14.315). + # It was the other way round: the data in <account home>/Shared and a + # link to it in the hub -- so reaching it meant passing through the + # account's home, and the home had to be group-readable (0750) for + # that. Your login is in the account's group, so it could read the + # browser profile, the cookies, everything. Now the data is + # /home/Shared/<Name> (2770 u_<name>:u_<name>), the home is 0700, and + # the account's own ~/Shared is a link into the hub -- which keeps + # `--download-dir ~/Shared` in app-users.conf meaning the same thing. if getattr(args, "share_dir", False): - share = os.path.join(home, "Shared") - _run(["mkdir", "-p", share]) - _run(["chown", f"{u}:{u}", share]) - _run(["chmod", "2770", share]) # setgid: new files keep the group - link = os.path.join(_home_of(main) or "", f"{args.name}-shared") - if link and not os.path.exists(link): - _run(["ln", "-s", share, link]) - printInfo(f" shared directory: {share}") - if link: - printInfo(f" reachable from your account as {link}") + _hub = _ensure_shared_hub(main) + share = os.path.join(_hub, args.name[:1].upper() + args.name[1:]) + if os.path.islink(share): + printWarning(f" {share} is a link (to {os.readlink(share)}) -- the " + f"layout before 1.14.315; left alone") + printInfo(" move the data in, as the 1.14.315 notes say, then " + "run this again") + else: + _run(["mkdir", "-p", share]) + _run(["chown", f"{u}:{u}", share]) + _run(["chmod", "2770", share]) # setgid: new files keep the group + printInfo(f" shared directory: {share} (2770 {u}:{u})") + # ...and the account's ~/Shared points at it + _in_home = os.path.join(home, "Shared") + if os.path.islink(_in_home): + if os.path.realpath(_in_home) != os.path.realpath(share): + printWarning(f" {_in_home} points at " + f"{os.readlink(_in_home)} -- left alone") + elif os.path.isdir(_in_home): + if not os.listdir(_in_home): + os.rmdir(_in_home) + _run(["ln", "-s", share, _in_home]) + _run(["chown", "-h", f"{u}:{u}", _in_home]) + else: + # DATA IS NEVER MOVED BY THIS. It was asked for as a + # hand step; the command is printed, not run. + printWarning(f" {_in_home} still holds files -- not " + f"moved (that is yours to do)") + printInfo(f" mv {_in_home}/* {_in_home}/.[!.]* {share}/ " + f"&& rmdir {_in_home} && ln -s {share} {_in_home}") + elif not os.path.lexists(_in_home): + _run(["ln", "-s", share, _in_home]) + _run(["chown", "-h", f"{u}:{u}", _in_home]) # 6. a launcher and a menu entry, when the program is really there app = getattr(args, "app", None) or getattr(args, "name", None) \ or u[len(user_prefix()):] import shutil as _sh + # AN APP CAN BE A COMMAND LINE. `--app "/bin/browser-mini + # https://claude.ai/new"` is the whole point of a per-account browser, + # and `which` on that string finds nothing -- so the launcher and the + # menu entry were silently skipped for exactly the accounts that needed + # them most. Look up the FIRST word; run the whole line. + _app_argv = app.split() + _app_cmd = _app_argv[0] if _app_argv else app + _app_base = os.path.basename(_app_cmd) want_launcher = getattr(args, "launcher", False) want_desktop = getattr(args, "desktop", False) if want_launcher or want_desktop: # Their original checked `which $app` first and skipped both when it # was missing -- a launcher for a program that is not installed is # just a broken command in your PATH. - if not _sh.which(app): - printWarning(f" no '{app}' on PATH -- no launcher or menu entry") + if not (_sh.which(_app_cmd) or os.path.isfile(_app_cmd)): + printWarning(f" no '{_app_cmd}' on PATH -- no launcher or menu " + f"entry") printInfo(f" (install it first, then: packagemanager user " - f"setup {app} --launcher --desktop)") + f"setup {args.name} --launcher --desktop)") else: + # the launcher is NAMED after the account, not after the command: + # two accounts can run the same browser with different arguments, + # and ~/bin/browser-mini cannot be both. if want_launcher: - _write_launcher(main, u, app) + _write_launcher(main, u, app, name=args.name) if want_desktop: - _write_desktop_entry(main, u, app) + _write_desktop_entry(main, u, app, name=args.name) def _home_of(user): @@ -7636,11 +10096,63 @@ def _allow_su_via_pam(u, main): try: with open(pam, "w") as f: f.writelines(out) + # IT IS A SCRIPT pam EXECUTES. Rewriting it must not drop the exec + # bit, and a file written 644 makes `su` ask for a password with + # nothing in the logs to say why. + _m = os.stat(pam).st_mode & 0o777 + if not _m & 0o111: + os.chmod(pam, 0o755) + printInfo(f" {pam} was not executable ({_m:o}) -- set to 755, " + f"pam_exec.so runs it") printInfo(f" {main} may now `su - {u}` without a password") + # ...AND pam ONLY READS IT IF /etc/pam.d/su SAYS SO. + _su = "/etc/pam.d/su" + _line = "auth sufficient pam_exec.so quiet %s" % pam + try: + _has = os.path.isfile(_su) and "su_u_user" in open(_su).read() + except OSError: + _has = False + if not _has: + printWarning(f" {_su} does not call it, so su will still ask for" + f" a password.") + printInfo(" add, above the other auth lines:") + printInfo(" %s" % _line) + printInfo(" `packagemanager stack services --only app-users " + "--run` does it for you.") except OSError as e: printWarning(f" could not write {pam}: {e}") +def _resolve_shared_level(args): + """--shared-slim / --shared-full on top of the original --shared. + + `--shared` joins the group on the XDG runtime directory -- the display, + the session bus and audio sockets live there, so without it a graphical + program cannot draw. Audio and video are SEPARATE groups, and were only + reachable by spelling them out in --groups. + + An account that runs one browser window does not need a microphone. So + the level is named: slim is the desktop alone, full adds audio and video. + """ + slim = getattr(args, "shared_slim", False) + full = getattr(args, "shared_full", False) + if slim and full: + printError("--shared-slim and --shared-full are opposites; pick one") + return None + if slim or full: + args.shared = True + if full: + have = [g for g in (getattr(args, "groups", "") or "").split(",") if g] + for g in ("audio", "video"): + if g not in have: + have.append(g) + args.groups = ",".join(have) + printInfo(" --shared-full: desktop + groups %s" % ", ".join(have)) + elif slim: + printInfo(" --shared-slim: desktop only (no audio, no video)") + return args + + def cmd_user_setup(args): """Apply the application-user setup to an account that already exists. @@ -7648,6 +10160,8 @@ def cmd_user_setup(args): earlier -- or made by hand -- can be brought up to the same state. Each step is idempotent, so running it again is safe and only fills in what is missing.""" + if _resolve_shared_level(args) is None: + return 2 u = _user_prefixed(args.name) if not _user_exists(u): printError(f"{u}: no such user.") @@ -7655,61 +10169,389 @@ def cmd_user_setup(args): sys.exit(1) printHeader(f"Set up {u}") print() + # an account made before its home had a place is brought there too + _relocate_app_home(u) if args.groups: _add_groups(u, [g for g in re.split(r"[,\s]+", args.groups) if g]) _setup_application_user(u, args) -def _write_desktop_entry(main, u, app): - """A menu entry that launches the program as the application user. +# BROWSERS THAT TAKE --app-id. A window's app id is what sway matches rules +# and the taskbar groups on; without one every account's browser is the same +# "browser-mini" window. A line that does not give one gets its launcher name. +_APP_ID_PROGRAMS = ("browser-mini", "browser-big") + +# WHAT YOU TYPE, AND WHAT HAPPENS WHEN YOU TYPE NOTHING. +# +# {} the word is the arguments / a search term +# {https://site/} ...and this page when there are no arguments +# +# The braces are the placeholder; anything INSIDE them is the fallback. It +# cannot be spelled `{|url}`: `|` is app-users.conf's field separator. +_TEMPLATE_RE = re.compile(r"\{[^{}]*\}") + + +def _is_template(word): + """Does this word carry a {} placeholder (with or without a fallback)?""" + return bool(_TEMPLATE_RE.search(word)) + + +def _launcher_words(u, app, who): + """The command line as words, with ~/ made the ACCOUNT's home, and an + --app-id added for a browser that has none.""" + import shlex as _shlex + try: + words = _shlex.split(app) + except ValueError: + words = app.split() + home = _app_home(u) or "" + words = [(home + w[1:]) if (w.startswith("~/") and home) else w for w in words] + if words and os.path.basename(words[0]) in _APP_ID_PROGRAMS \ + and "--app-id" not in words: + words[1:1] = ["--app-id", who] + return words + + +def _recolor_svg(path, color): + """Paint a one-colour SVG in `color`. + + A brand icon is a single path with one fill (or none at all, inheriting + black). simpleicons.org used to do this server-side -- /openai/38BDF8 -- + but only for the icons it still carries, and it dropped `openai` and + `grok`. Doing it here works on any monochrome SVG, wherever it came + from. `fill="none"` is left alone: that is a hole, not a colour. A + multi-colour logo comes out flat, which is why this only happens when a + colour is asked for. + """ + try: + t = open(path, encoding="utf-8", errors="replace").read() + except OSError: + return False + if "<svg" not in t: + return False + new = re.sub(r'fill="(?!none\b)[^"]*"', 'fill="%s"' % color, t) + new = re.sub(r'fill:\s*(?!none\b)[^;"\']+', 'fill:%s' % color, new) + if not re.search(r'<svg\b[^>]*\bfill=', new): + new = re.sub(r'<svg\b', '<svg fill="%s"' % color, new, count=1) + if new == t: + return False + try: + open(path, "w").write(new) + except OSError: + return False + return True + + +def _fetch_icon(main, who, icon): + """The menu icon, fetched once into ~/.local/share/icons/lfs-apps. + + icon: an http(s) URL -> that file; a word -> the simpleicons.org slug of + that name; empty -> the launcher name as the slug; "-" -> no icon. A + trailing #RRGGBB paints a monochrome SVG that colour after it is + fetched. A failed download is not an error: the entry falls back to the + program's own icon name. + """ + if icon == "-": + return None + home = _home_of(main) + if not home: + return None + spec = icon or "" + # #RRGGBB AT THE END IS A COLOUR, NOT PART OF THE URL. No URL these + # icons come from uses a fragment, and a colour has to be expressible for + # a plain slug too (`openai#38BDF8`), not just for a URL. + color = "" + m = re.search(r"#([0-9A-Fa-f]{3}|[0-9A-Fa-f]{6})$", spec) + if m: + color = "#" + m.group(1) + spec = spec[:m.start()] + url = spec if re.match(r"^https?://", spec) else \ + "https://cdn.simpleicons.org/%s" % ((spec or who).lower()) + ext = os.path.splitext(url.split("?")[0])[1].lower() + ext = ext if ext in (".svg", ".png", ".xpm") else ".svg" + d = os.path.join(home, ".local", "share", "icons", "lfs-apps") + path = os.path.join(d, who + ext) + stamp = path + ".from" + # the stamp records the WHOLE spec, so changing only the colour refetches + want = url + color + if os.path.isfile(path) and os.path.getsize(path) > 0: + try: + if open(stamp).read().strip() == want: + return path + except OSError: + pass + os.makedirs(d, exist_ok=True) + tmp = path + ".part" + r = subprocess.run(["curl", "-fsSL", "--max-time", "20", url, "-o", tmp], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + if r.returncode != 0 or not os.path.isfile(tmp) or os.path.getsize(tmp) == 0: + try: + os.unlink(tmp) + except OSError: + pass + printWarning(f" no icon for {who} from {url} -- the menu uses the program's") + if "cdn.simpleicons.org" in url: + printInfo(" (simpleicons.org has dropped several brands, `openai` " + "and `grok` among them -- give the icon column a URL)") + return None + os.replace(tmp, path) + if color and ext == ".svg": + printInfo(f" icon {who}: painted {color}" if _recolor_svg(path, color) + else f" icon {who}: {color} changed nothing (not a one-colour SVG)") + open(stamp, "w").write(want + "\n") + for f in (d, path, stamp): + _run(["chown", f"{main}:{main}", f]) + printInfo(f" icon: {path} ({url})") + return path + - Written next to the wrapper, so the program appears in the menu and starts - under the right account when clicked -- not just from a terminal.""" +def _write_desktop_entry(main, u, app, name=None, icon=""): + """A menu entry that runs the launcher -- named after the LAUNCHER, so + one account can have several (google and duckduckgo run as u_browser).""" home = _home_of(main) if not home: return + who = os.path.basename(name or app.split()[0]) appdir = os.path.join(home, ".local", "share", "applications") os.makedirs(appdir, exist_ok=True) - path = os.path.join(appdir, f"{app}.desktop") - if os.path.exists(path) and not getattr(_write_desktop_entry, "_force", False): - printInfo(f" {path} already exists -- left alone") - return - launcher = os.path.join(home, "bin", app) - with open(path, "w") as f: - f.write( - "[Desktop Entry]\n" - f"Name={app.capitalize()}\n" + path = os.path.join(appdir, f"{who}.desktop") + launcher = os.path.join(home, "bin", who) + words = _launcher_words(u, app, who) + prog = os.path.basename(words[0]) if words else who + icon_path = _fetch_icon(main, who, icon) + takes_args = any(_is_template(w) for w in words) + # A MENU NEEDS A CATEGORY. An entry with none is "Other" at best and + # missing at worst, depending on the menu -- and these are the entries a + # person actually clicks. Taken from the program, since that is what is + # known here. + if prog in _APP_ID_PROGRAMS: + cats = "Network;WebBrowser;" + elif prog in ("mpv", "vlc", "ffplay"): + cats = "AudioVideo;Video;Player;" + else: + cats = "Utility;" + text = ("[Desktop Entry]\n" + f"Name={who.capitalize()}\n" f"Comment=Runs as the application user {u}\n" - f"Exec={launcher} %U\n" + f"Exec={launcher}{' %U' if takes_args else ''}\n" "Terminal=false\n" "Type=Application\n" - f"Icon={app}\n" - f"StartupWMClass={app.capitalize()}\n") + f"Categories={cats}\n" + f"Icon={icon_path or prog}\n" + f"StartupWMClass={who if prog in _APP_ID_PROGRAMS else prog.capitalize()}\n") + if os.path.exists(path): + try: + cur = open(path, encoding="utf-8", errors="replace").read() + except OSError: + cur = "" + if cur == text: + return + if "Comment=Runs as the application user " not in cur: + printInfo(f" {path} was not written by these tools -- left alone") + return + printInfo(f" menu entry {path} was stale -- rewritten") + with open(path, "w") as f: + f.write(text) os.chmod(path, 0o644) _run(["chown", f"{main}:{main}", path]) printInfo(f" menu entry: {path}") -def _write_launcher(main, u, app): - """A wrapper in ~/bin that runs <app> as the application user.""" +_LAUNCHER_TEMPLATE = r'''#!/bin/bash +# Runs @APP@ as the application user @USER@. +# written by packagemanager user launcher -- rewritten when stale +T=(@ARR@) +enc() { # URL-encode, byte by byte; a space becomes + + local LC_ALL=C s="$*" o="" c i + for ((i = 0; i < ${#s}; i++)); do + c="${s:i:1}" + case "$c" in + [a-zA-Z0-9._~-]) o+="$c" ;; + " ") o+="+" ;; + *) printf -v c "%%%02X" "'$c"; o+="$c" ;; + esac + done + printf "%s" "$o" +} +cmd=() +for w in "${T[@]}"; do + if [ "$w" = "{}" ]; then + cmd+=("$@") # the arguments, one by one + elif [[ "$w" == *"{"*"}"* ]]; then + pre="${w%%\{*}"; rest="${w#*\{}" # ...{ inner }... + inner="${rest%%\}*}"; post="${rest#*\}}" + if [ $# -gt 0 ]; then + cmd+=("$pre$(enc "$*")$post") # what you typed, URL-encoded + elif [ -n "$inner" ]; then + cmd+=("$inner") # nothing typed: the default page + else + p="$pre"; case "$p" in *\?*) p="${p%\?*}" ;; esac + cmd+=("$p") # nothing typed, no default: cut at ? + fi + else + cmd+=("$w") + fi +done +# A MENU ENTRY HAS NO TERMINAL, AND su WANTS ONE. +# +# From a terminal this is the whole story: su authenticates through +# /etc/pam.d/su_u_user and execs the program. Started from a .desktop entry, +# a hotkey or a launcher menu there is no tty at all -- su fails, its message +# goes to a stderr nobody is reading, and the entry looks like it "does +# nothing". script(1) gives it a pty and nothing else; the typescript goes +# to /dev/null and -e returns the program's own exit status. +if [ -t 0 ] && [ -t 1 ]; then + exec su - @QUSER@ -c 'exec "$0" "$@"' -- "${cmd[@]}" +fi +script -qec "$(printf '%q ' su - @QUSER@ -c 'exec "$0" "$@"' -- "${cmd[@]}")" /dev/null +rc=$? +# ...and say so where it can be seen, since there is no terminal to print to +if [ "$rc" != 0 ] && command -v notify-send >/dev/null 2>&1; then + notify-send "@WHO@" "did not start (exit $rc) -- run @WHO@ in a terminal to see why" +fi +exit $rc +''' + + +def _write_launcher(main, u, app, name=None): + """A wrapper in ~/bin, named after the LAUNCHER, that runs the command + line as the application account. + + {} IN THE COMMAND LINE IS WHAT YOU TYPE. + a word that IS {} -> the arguments, one by one ("browser-mini {}") + {} inside a word -> the arguments as a URL-encoded search term + ("https://www.youtube.com/results?search_query={}") + and with no arguments the word is cut before + its ? (or before {}): the site's front page + {https://site/} in one -> the same, but with nothing typed the word + becomes exactly what is inside the braces. + Cutting at the ? gives /search/ -- a search + page for a search nobody made. This gives + the page you actually want to land on, and + it is per line: reddit's is the front page + today and may be your own profile tomorrow. + no {} at all -> arguments are NOT passed: `claude foo` must + not hand browser-mini a second address + + THE ARGUMENTS GO THROUGH AS ARGUMENTS. The old writer built + su -c "app $(printf %q-each "$@")", which with no arguments produced one + EMPTY argument ("a second address:"). Here the final list is built in + the launcher and handed to su after --: bash gets $0 and "$@" exactly. + """ home = _home_of(main) if not home: return bindir = os.path.join(home, "bin") os.makedirs(bindir, exist_ok=True) - path = os.path.join(bindir, app) + who = os.path.basename(name or app.split()[0]) + path = os.path.join(bindir, who) + import shlex as _shlex + words = _launcher_words(u, app, who) + arr = " ".join(_shlex.quote(w) for w in words) + # A PLAIN TEMPLATE, NOT %-FORMATTING. This body is bash full of `%02X` + # and `${w%%...}`, and every one of them had to be written `%%%%` here. + # One miscount and the launcher is subtly wrong, which is how the empty + # second address survived so long. Three named slots, .replace(), done. + text = _LAUNCHER_TEMPLATE.replace("@APP@", app).replace("@USER@", u) \ + .replace("@QUSER@", _shlex.quote(u)) \ + .replace("@WHO@", who) \ + .replace("@ARR@", arr) if os.path.exists(path): - printInfo(f" {path} already exists -- left alone") - return + try: + cur = open(path, encoding="utf-8", errors="replace").read() + except OSError: + cur = "" + # OURS AND STALE IS REWRITTEN; ANYTHING ELSE IS YOURS. + if cur == text: + printInfo(f" launcher {path} is current") + return + if not re.search(r"^# Runs .* as the application user \S+\.$", cur, re.M): + # SAY WHAT TO DO ABOUT IT. A launcher from an older version of + # these tools is not recognised either, so "left alone" quietly + # kept the OLD behaviour -- no default page, no --app-id -- on + # exactly the launchers the conf had just changed. + printInfo(f" {path} exists and was not written by these tools -- left alone") + printInfo(f" to take the generated one: rm {path} and run this again") + return + _was = re.search(r"su - (\S+)", cur) + printInfo(f" launcher {path} was stale (ran as " + f"{_was.group(1) if _was else '?'}) -- rewritten") with open(path, "w") as f: - f.write('#!/bin/bash\n' - '# Runs %s as the application user %s.\n' - 'exec su - %s -c "%s $(printf \'%%q \' "$@")"\n' % (app, u, u, app)) + f.write(text) os.chmod(path, 0o755) _run(["chown", f"{main}:{main}", path]) printInfo(f" launcher: {path}") +def cmd_user_launcher(args): + """One launcher (and menu entry) for an EXISTING application account. + + Separate from `user setup` because an account can have several: the + `browser` account runs browser-mini, browser-big, google and duckduckgo. + """ + u = _user_prefixed(args.account) + if not _user_exists(u): + printError(f"{u}: no such account -- `user create {args.account}` first") + return 1 + main = main_user() + if not main: + printError("no main user set (packagemanager config --main-user <login>)") + return 1 + who = args.name + import shutil as _sh + words = _launcher_words(u, args.app, who) + if not words or not (_sh.which(words[0]) or os.path.isfile(words[0])): + printWarning(f" {who}: no '{words[0] if words else args.app}' on PATH " + f"-- no launcher or menu entry") + return 0 + _write_launcher(main, u, args.app, name=who) + if not args.no_desktop: + _write_desktop_entry(main, u, args.app, name=who, icon=args.icon or "") + return 0 + + +def _ours_for(path, marker): + try: + return marker in open(path, encoding="utf-8", errors="replace").read() + except OSError: + return False + + +def cmd_user_remove_launcher(args): + """Remove a launcher, its menu entry and icon -- only files these tools + wrote (a hand-made ~/bin/<name> is never touched).""" + main = main_user() + home = _home_of(main) if main else None + if not home: + printError("no main user set") + return 1 + who = os.path.basename(args.name) + files = [] + lp = os.path.join(home, "bin", who) + if os.path.isfile(lp) and _ours_for(lp, "as the application user"): + files.append(lp) + dp = os.path.join(home, ".local", "share", "applications", who + ".desktop") + if os.path.isfile(dp) and _ours_for(dp, "Comment=Runs as the application user "): + files.append(dp) + idir = os.path.join(home, ".local", "share", "icons", "lfs-apps") + if os.path.isdir(idir): + files += [os.path.join(idir, f) for f in os.listdir(idir) + if f.startswith(who + ".")] + if not files: + printInfo(f" {who}: no launcher, menu entry or icon of these tools") + return 0 + for f in files: + printInfo((" remove " if args.run else " would remove ") + f) + if args.run: + for f in files: + os.unlink(f) + else: + printInfo(" dry run -- add --run") + printInfo(f" and take its line out of /etc/pkgusr/stacks/app-users.conf, " + f"or the next `cfg app-users` writes it again") + return 0 + + def cmd_user_allow(args): user = args.user up = user_prefix() @@ -7725,50 +10567,834 @@ def cmd_user_allow(args): _add_groups(user, args.groups) -def cmd_user_disallow(args): - for g in args.groups: - if _in_group(args.user, g): - if _run(["gpasswd", "-d", args.user, g]): - printSuccess(f"Removed {args.user} from {g}.") - else: - printInfo(f"{args.user} is not in {g}.") +def cmd_user_disallow(args): + for g in args.groups: + if _in_group(args.user, g): + if _run(["gpasswd", "-d", args.user, g]): + printSuccess(f"Removed {args.user} from {g}.") + else: + printInfo(f"{args.user} is not in {g}.") + + +# WHAT AN ACCOUNT'S HOME HOLDS WHEN IT HAS NEVER RUN ANYTHING. +# `user create` puts a .bash_profile symlink there (and keeps whatever was +# there as .bash_profile.pkgusr), plus an empty Shared if the line asked for +# one. Nothing else is ours. +_HOME_SCAFFOLD = {".bash_profile", ".bash_profile.pkgusr", ".bashrc", + ".bash_logout", ".bash_history", ".profile", ".dircolors", + "pkg.lst"} + + +def _home_is_untouched(home): + """Is there anything in this home worth keeping? + + The archive exists because a home holds a browser profile: logins, + cookies, history, downloads -- months of state that `userdel -r` would + take with it and nothing could give back. A home that was created and + never used holds none of that, and a tarball of it is 200 bytes of + nothing in /root, one per account, fourteen more on the next purge. + """ + try: + entries = os.listdir(home) + except OSError: + return False + for n in entries: + if n in _HOME_SCAFFOLD: + continue + p = os.path.join(home, n) + if n == "Shared" and os.path.isdir(p) and not os.path.islink(p) \ + and not os.listdir(p): + continue + return False + return True + + +def _shared_hub(main_home=""): + """Where the per-account links live -- the system hub, not a home + (1.14.312; the argument is kept for the callers and ignored).""" + return shared_hub() + + +def _ensure_shared_hub(main): + """Create the hub, and make the main user's ~/Shared a link to it. + + MOVING, NOT COPYING, AND ONLY OUR LINKS. A ~/Shared that is already a + directory has the links earlier releases made in it; each one moves to + the hub (same name, same target). Anything else in it -- a file, a + directory, a link of your own that clashes with one in the hub -- is + yours, so the directory stays, and says so. Only an EMPTY ~/Shared is + replaced by the link. + """ + hub = shared_hub() + if not os.path.isdir(hub): + _run(["mkdir", "-p", hub]) + # root owns it, your login's group may add a link by hand (0775): the + # hub is a system path that YOU curate, and nobody else may + grp_name = main if (main and _group_exists(main)) else "root" + _run(["chown", f"root:{grp_name}", hub]) + _run(["chmod", "0775" if grp_name != "root" else "0755", hub]) + mh = _home_of(main) if main else None + if not mh: + return hub + home_link = os.path.join(mh, "Shared") + if os.path.islink(home_link): + if os.path.realpath(home_link) != os.path.realpath(hub): + printWarning(f" {home_link} points at {os.readlink(home_link)}, " + f"not the hub {hub} -- left alone") + return hub + if os.path.isdir(home_link): + moved, kept = 0, [] + for n in sorted(os.listdir(home_link)): + p = os.path.join(home_link, n) + dest = os.path.join(hub, n) + if not os.path.islink(p): + kept.append(n) + continue + t = os.readlink(p) + if os.path.lexists(dest): + if os.path.islink(dest) and os.readlink(dest) == t: + os.unlink(p) # already moved on an earlier run + continue + kept.append(n) + continue + os.symlink(t, dest) + os.lchown(dest, 0, 0) + os.unlink(p) + moved += 1 + if moved: + printInfo(f" moved {moved} link(s) from {home_link} to {hub}") + if kept: + printWarning(f" {home_link} still holds {', '.join(kept[:5])}" + f"{' ...' if len(kept) > 5 else ''} -- yours, so it " + f"stays a directory") + printInfo(f" the shared links are in {hub}; move those by hand, " + f"then run this again") + return hub + try: + os.rmdir(home_link) + except OSError: + return hub + os.symlink(hub, home_link) + try: + _pw = pwd.getpwnam(main) + os.lchown(home_link, _pw.pw_uid, _pw.pw_gid) + except (KeyError, OSError): + pass + printInfo(f" {home_link} -> {hub}") + return hub + + +def _remove_empty_hub(mh): + """The hub goes when nothing is left in it -- and ~/Shared, the link to + it, goes with it, or it dangles.""" + hub = shared_hub() + if not (hub and os.path.isdir(hub) and not os.listdir(hub)): + return + try: + os.rmdir(hub) + printInfo(f" removed {hub} (empty)") + except OSError: + return + hl = os.path.join(mh, "Shared") if mh else "" + if hl and os.path.islink(hl) and not os.path.exists(hl): + try: + os.unlink(hl) + printInfo(f" removed {hl} (it pointed at the hub)") + except OSError: + pass + + +def _stale_shared_links(main_home, home=""): + """Links in ~/Shared that point into `home`, plus every link there that + leads nowhere at all. A dangling link is what keeps a SHARED_* variable + alive after the account behind it is gone.""" + hub = _shared_hub(main_home) + out = [] + if not hub or not os.path.isdir(hub): + return out + for f in sorted(os.listdir(hub)): + fp = os.path.join(hub, f) + if not os.path.islink(fp): + continue + if not os.path.exists(fp): # dangling: nothing to keep + out.append(fp) + elif home and os.readlink(fp).startswith(home.rstrip("/") + "/"): + out.append(fp) + return out + + +def _shared_vars_note(paths): + """Say what an open shell still thinks, and how to correct it. + + /etc/profile.d/shared-dirs.sh runs at LOGIN. Removing the link removes + tomorrow's variable; the shell you are typing in keeps today's until you + log in again -- so `cd $SHARED_BROWSER_MINI` still completes and still + fails, which reads like the removal did not work.""" + names = [os.path.basename(p).upper().replace("-", "_") for p in paths] + if not names: + return + printInfo(" shells that are already open still export " + + ", ".join("SHARED_" + n for n in names)) + printInfo(" they are read at login: log in again, or right now:") + printInfo(" unset " + " ".join("SHARED_" + n for n in names)) + + +def cmd_user_delete(args): + """Remove an application account SAFELY. + + Only an application account: the user prefix, and a home under the + application-user root or the package tree. Refused while it runs + anything. Without --no-archive its home is packed first into + /root/<account>-home-<date>.tar.gz (0600). Then: its launchers, menu + entries and icons (the ones these tools wrote that run as it), the + ~/Shared/<Name> link, its line in /etc/pam.d/su_u_user, the account, its + group, and the home (kept with --keep-home). + """ + u = _user_prefixed(args.name) + if not u.startswith(user_prefix()): + printError(f"{u}: not an application account ({user_prefix()}...) -- refused") + return 1 + exists = _user_exists(u) + home = (pwd.getpwnam(u).pw_dir if (exists and pwd) else "") or _app_home(u) + roots = [os.path.realpath(appuser_home()), os.path.realpath(BASE_DIR)] + if home and not any(os.path.realpath(home).startswith(r.rstrip("/") + "/") for r in roots): + printError(f"{u}: home {home} is not under {' or '.join(roots)} -- refused") + return 1 + main = main_user() + mh = _home_of(main) if main else None + # what goes + launchers = [] + if mh: + for d, marker in ((os.path.join(mh, "bin"), f"as the application user {u}."), + (os.path.join(mh, ".local", "share", "applications"), + f"Comment=Runs as the application user {u}\n")): + if os.path.isdir(d): + for f in sorted(os.listdir(d)): + fp = os.path.join(d, f) + if os.path.isfile(fp) and _ours_for(fp, marker): + launchers.append(fp) + icons = [] + if mh: + idir = os.path.join(mh, ".local", "share", "icons", "lfs-apps") + names = {os.path.splitext(os.path.basename(f))[0] for f in launchers} + if os.path.isdir(idir): + icons = [os.path.join(idir, f) for f in sorted(os.listdir(idir)) + if f.split(".")[0] in names] + # THE LINK, AND ANY LINK THAT NO LONGER LEADS ANYWHERE. + # + # A link was only removed when it pointed inside THIS home, so a home + # moved after the link was made (usermod -d, or the move to + # /home/shared_user) left ~/Shared/<Name> behind, dangling. Nothing + # then removed it -- and /etc/profile.d/shared-dirs.sh enumerates that + # directory, which is why $SHARED_BROWSER_MINI outlived the account. + links = _stale_shared_links(mh, home) + printHeader(f"Remove application account {u}") + print(f" account : {u}" + ("" if exists else " (already gone)")) + if home: + print(f" home : {home}" + (" (kept: --keep-home)" if args.keep_home else "")) + if os.path.isdir(home) and not args.no_archive: + if _home_is_untouched(home): + print(" archive : none -- nothing in this home but the files " + "these tools put there") + else: + print(f" archive : /root/{u}-home-<date>.tar.gz first (0600)") + for f in launchers + icons + links: + print(f" remove : {f}") + print(" pam : its line in /etc/pam.d/su_u_user") + if exists: + busy = subprocess.run(["pgrep", "-u", u], stdout=subprocess.DEVNULL).returncode == 0 + if busy: + printError(f" {u} is running something -- close it (pkill -u {u}) and try again") + return 1 + if not args.run: + printInfo("\n dry run -- nothing removed. Remove: " + f"packagemanager user delete {args.name} --run") + printInfo(" and take its lines out of /etc/pkgusr/stacks/app-users.conf, " + "or the next `cfg app-users` creates it again") + return 0 + if not args.yes and not ask_yes_no(f"\nReally remove {u}?"): + printWarning("Aborted.") + return 1 + if home and os.path.isdir(home) and not args.no_archive \ + and not _home_is_untouched(home): + arch = f"/root/{u}-home-{time.strftime('%Y%m%d-%H%M%S')}.tar.gz" + old_umask = os.umask(0o077) + ok_arch = _run(["tar", "-C", os.path.dirname(home), "-czf", arch, + os.path.basename(home)]) + os.umask(old_umask) + if not ok_arch: + printError(f" could not archive {home} -- nothing removed") + return 1 + _run(["chmod", "600", arch]) + printInfo(f" archived {home} -> {arch}") + for f in launchers + icons + links: + try: + os.unlink(f) + printInfo(f" removed {f}") + except OSError as e: + printWarning(f" {f}: {e}") + pam = "/etc/pam.d/su_u_user" + if os.path.isfile(pam): + try: + txt = open(pam).read() + new = re.sub(r"^\s*%s\)[^\n]*\n" % re.escape(u), "", txt, flags=re.M) + if new != txt: + open(pam, "w").write(new) + printInfo(f" removed {u} from {pam}") + except OSError as e: + printWarning(f" {pam}: {e}") + if exists: + _run(["userdel", u]) + if _group_exists(u): + _run(["groupdel", u]) + if home and os.path.isdir(home) and not args.keep_home: + try: + shutil.rmtree(home) + printInfo(f" removed {home}") + except OSError as e: + printError(f" rmtree {home}: {e}") + # ITS SHARED DIRECTORY IS NOT IN ITS HOME ANY MORE (1.14.315), so the + # archive above does not carry it. Empty: removed. With files: kept, + # and said -- those are files you put there, not the account's. + _bare = u[len(user_prefix()):] + _sd = os.path.join(shared_hub(), _bare[:1].upper() + _bare[1:]) + if os.path.isdir(_sd) and not os.path.islink(_sd): + if not os.listdir(_sd): + try: + os.rmdir(_sd) + printInfo(f" removed {_sd} (empty)") + except OSError: + pass + else: + printWarning(f" kept {_sd} -- it has files in it; look, then:" + f" rm -r {_sd}") + # the hub is ours only while it holds our links: an empty hub is + # left-over furniture, one with something else in it is yours. + _remove_empty_hub(mh) + printSuccess(f"{u} removed.") + _shared_vars_note(links) + printInfo(" take its lines out of /etc/pkgusr/stacks/app-users.conf, " + "or the next `cfg app-users` creates it again") + return 0 + + +# =========================================================================== # +# command: user purge -- undo the whole app-users setup [IMPLEMENTED] +# =========================================================================== # +# +# `user delete` removes ONE account. What it cannot remove is everything the +# accounts share, because the next account still needs it: the session group, +# the two profile fragments, /usr/bin/xdg-runtime-share, the pam file and the +# line in /etc/pam.d/su that calls it. Left behind, those are what makes a +# "deleted" setup still visible -- a group you are in, variables in every new +# shell, an auth line pointing at a file that is gone. +# +# NOTHING HERE TOUCHES A FILE THESE TOOLS DID NOT WRITE. Each file is read +# first and only removed if it carries the marker `cfg app-users` writes. +# The home archives in /root are never removed: they are the undo of the undo. + +APP_USERS_MARKER = "# written by stacks/app-users.sh -- safe to delete, regenerated" +APP_USERS_CONF = "/etc/pkgusr/stacks/app-users.conf" + + +def _conf_accounts(path): + """The account names in app-users.conf: column 1, both file formats.""" + out = [] + try: + lines = open(path, encoding="utf-8", errors="replace").read().splitlines() + except OSError: + return out + for raw in lines: + s = raw.strip() + if not s or s.startswith("#"): + continue + a = s.split("|")[0].strip() + if a in ("", "name", "user", "account") or re.search(r"[^a-z0-9_-]", a): + continue + if a not in out: + out.append(a) + return out + + +def _ours(path, marker): + """A file that exists and says these tools wrote it.""" + if not os.path.isfile(path): + return False + try: + return marker in open(path, encoding="utf-8", errors="replace").read() + except OSError: + return False + + +def cmd_user_purge(args): + """Undo the application-account setup: the accounts, then what they share. + + Dry run unless --run. Homes are archived to /root first, exactly as + `user delete` does, and those archives stay. + """ + conf = args.conf or APP_USERS_CONF + main = main_user() + mh = _home_of(main) if main else "" + accounts = _conf_accounts(conf) + if args.accounts: + accounts = [a for a in re.split(r"[,\s]+", args.accounts) if a] + # ACCOUNTS THE FILE NO LONGER MENTIONS ARE STILL OURS TO REMOVE. That is + # exactly the browser-mini/browser-big case: the line was rewritten, the + # account stayed. Found in passwd, not by listing a directory, so an + # account whose home is already gone is found too. + strays = [] + if pwd: + root = os.path.realpath(appuser_home()) + for e in pwd.getpwall(): + if not e.pw_name.startswith(user_prefix()): + continue + a = e.pw_name[len(user_prefix()):] + if a in accounts or a in strays: + continue + if os.path.realpath(e.pw_dir or "").startswith(root.rstrip("/") + "/"): + strays.append(a) + strays.sort() + todo = [a for a in accounts + strays if _user_exists(_user_prefixed(a))] + + printHeader("Undo the application-account setup") + print() + print(" accounts : " + (", ".join(user_prefix() + a for a in todo) or "none left")) + shared = [] + for p in ("/etc/profile.d/shared-dirs.sh", "/etc/profile.d/xdg-runtime-share.sh", + "/usr/bin/xdg-runtime-share"): + if _ours(p, APP_USERS_MARKER): + shared.append(p) + elif os.path.exists(p): + printWarning(f" {p} is not ours -- left alone") + pam_file = "/etc/pam.d/su_u_user" + pam_su = "/etc/pam.d/su" + # an ACTIVE auth line, not a comment that happens to name the file + pam_line = bool(os.path.isfile(pam_su) and re.search( + r"^\s*auth\s+sufficient\s+pam_exec\.so.*su_u_user", + open(pam_su, encoding="utf-8", errors="replace").read(), re.M)) + for p in shared: + print(f" remove : {p}") + if os.path.isfile(pam_file): + print(f" remove : {pam_file}") + if pam_line: + print(f" edit : {pam_su} (drop the pam_exec line; backup kept)") + g = xdg_runtime_group() + if _group_exists(g): + print(f" group : {g} (removed; {main or 'you'} leaves it)") + print(" kept : /root/" + user_prefix() + "*-home-*.tar.gz (the archives)") + print(f" kept : {conf} -- delete the lines you no longer want, or the") + print(" next `cfg app-users` builds all of this again") + + if not args.run: + printInfo("\n dry run -- nothing removed. Undo it: " + "packagemanager user purge --run") + return 0 + if not args.yes and not ask_yes_no("\nReally undo all of it?"): + printWarning("Aborted.") + return 1 + + # 1. the accounts, through the command that already knows how + import argparse as _ap + gone = 0 + for a in todo: + rc = cmd_user_delete(_ap.Namespace( + name=a, run=True, yes=True, + keep_home=args.keep_home, no_archive=args.no_archive)) + if rc: + printWarning(f" {a}: not removed -- stopping before the shared parts") + printInfo(" (close what it is running, then run this again)") + return 1 + gone += 1 + + # 2. anything left in ~/Shared that leads nowhere, then the hub itself + dead = _stale_shared_links(mh) + for p in dead: + try: + os.unlink(p) + printInfo(f" removed {p}") + except OSError as e: + printWarning(f" {p}: {e}") + _remove_empty_hub(mh) + idir = os.path.join(mh, ".local", "share", "icons", "lfs-apps") if mh else "" + if idir and os.path.isdir(idir) and not os.listdir(idir): + try: + os.rmdir(idir) + printInfo(f" removed {idir} (empty)") + except OSError: + pass + + # 3. the files every account shared + for p in shared: + try: + os.unlink(p) + printInfo(f" removed {p}") + except OSError as e: + printWarning(f" {p}: {e}") + if os.path.isfile(pam_file): + try: + os.unlink(pam_file) + printInfo(f" removed {pam_file}") + except OSError as e: + printWarning(f" {pam_file}: {e}") + # THE LINE IN /etc/pam.d/su GOES LAST AND WITH A BACKUP. It is the one + # edit here that can cost you `su` if it goes wrong. + if pam_line: + bak = f"{pam_su}.bak-{time.strftime('%Y%m%d-%H%M%S')}" + try: + txt = open(pam_su).read() + new = "\n".join(l for l in txt.splitlines() + if not re.match(r"^\s*auth\s+sufficient\s+pam_exec\.so.*su_u_user", + l)) + "\n" + if new != txt: + shutil.copy2(pam_su, bak) + open(pam_su, "w").write(new) + printInfo(f" {pam_su}: pam_exec line removed (backup {bak})") + except OSError as e: + printWarning(f" {pam_su}: {e}") + + # 3b. the directory the accounts lived in, if it is now empty + root = appuser_home() + if os.path.isdir(root) and not os.listdir(root): + try: + os.rmdir(root) + printInfo(f" removed {root} (empty)") + except OSError: + pass + + # 4. the session group -- last, when nothing is in it any more + if _group_exists(g): + _run(["groupdel", g]) + printInfo(f" removed group {g}") + + printSuccess(f"undone: {gone} account(s) and everything they shared.") + _shared_vars_note(dead) + printInfo(" your session still has the old group until you log in again") + # THE ARCHIVES ARE THE ONE THING THIS DOES NOT REMOVE, so say what they + # are and how to be rid of them. They only exist for a home that had + # something in it -- a browser profile, its logins and its downloads. + import glob as _glob + arch = sorted(_glob.glob(f"/root/{user_prefix()}*-home-*.tar.gz")) + if arch: + printInfo(f" {len(arch)} home archive(s) kept in /root -- browser profiles, " + f"nothing else can give them back") + printInfo(f" look: tar tzf {arch[-1]} | head") + printInfo(f" or drop: rm /root/{user_prefix()}*-home-*.tar.gz") + if mh and os.path.isfile(os.path.join(mh, ".config", "sway", "config")): + printInfo(" and `exec xdg-runtime-share` is still in your sway config " + "-- yours to remove") + return 0 + + +# =========================================================================== # +# command: service -- the SysV services, without remembering the layout +# =========================================================================== # +# +# A SysV service is two things in two places: /etc/rc.d/init.d/<name>, which +# starts it, and the S/K symlinks in /etc/rc.d/rc<N>.d, which decide whether +# boot does. Everything here is `ls`, `ln -s` and `rm` on those -- no daemon, +# no database, nothing this tool has to keep in step with the system. +# +# START IS NOT ENABLE. systemd taught everyone that they are different, and +# SysV has always had them separate; `service sshd start` runs the script now +# whatever the links say, and `enable` only touches the links. + +SVC_STATE_DIR = "/var/lib/pkgusr/services" +SVC_ACTIONS = ("start", "stop", "restart", "reload", "force-reload", "status", + "enable", "disable", "list") + + +def _svc_dirs(): + """(init.d, the directory holding rc<N>.d) -- LFS's layout, then the + plain one, because only one of them exists on a given machine.""" + for initd, rcroot in (("/etc/rc.d/init.d", "/etc/rc.d"), ("/etc/init.d", "/etc")): + if os.path.isdir(initd): + return initd, rcroot + return None, None + + +# NOT EVERY FILE IN init.d IS A SERVICE. `functions` is a library the others +# source, `rc` is what init runs, `template` is a skeleton to copy. Listing +# them invites starting them. +_SVC_NOT_SERVICES = {"functions", "rc", "rcS", "template", "README", "Makefile"} + + +def _svc_all(initd): + out = [] + try: + names = sorted(os.listdir(initd)) + except OSError: + return out + for n in names: + if n in _SVC_NOT_SERVICES or n.endswith((".bak", "~", ".orig", ".new")): + continue + p = os.path.join(initd, n) + if os.path.isfile(p) and os.access(p, os.X_OK): + out.append(n) + return out + + +def _svc_links(rcroot, name): + """Every rc<N>.d link for this service: [(level, 'S'|'K', num, path)].""" + out = [] + try: + entries = sorted(os.listdir(rcroot)) + except OSError: + return out + for d in entries: + m = re.match(r"^rc([0-6S])\.d$", d) + if not m: + continue + full = os.path.join(rcroot, d) + try: + for f in sorted(os.listdir(full)): + lm = re.match(r"^([SK])(\d{2})" + re.escape(name) + r"$", f) + if lm: + out.append((m.group(1), lm.group(1), lm.group(2), + os.path.join(full, f))) + except OSError: + continue + return out + + +def _svc_default_runlevel(): + try: + for line in open("/etc/inittab"): + m = re.match(r"^\s*id:([0-6]):initdefault:", line) + if m: + return m.group(1) + except OSError: + pass + return "3" + + +def _svc_header_levels(initd, name): + """Default-Start / Default-Stop from the script's INIT INFO block. + + The book's scripts carry it, so the runlevels a service belongs in are + written down on the machine -- no need to invent them here.""" + start, stop = "", "" + try: + for line in open(os.path.join(initd, name), errors="replace"): + m = re.match(r"^#\s*Default-Start:\s*(.*)$", line) + if m: + start = "".join(re.findall(r"[0-6S]", m.group(1))) + m = re.match(r"^#\s*Default-Stop:\s*(.*)$", line) + if m: + stop = "".join(re.findall(r"[0-6S]", m.group(1))) + if line.startswith("### END INIT INFO"): + break + except OSError: + pass + return start, stop + + +def _svc_enabled_levels(rcroot, name): + return sorted({lv for lv, kind, _n, _p in _svc_links(rcroot, name) if kind == "S"}) + + +def _svc_running(initd, name, quick=True): + """'yes' / 'no' / '?' -- asked of the script itself. + + The LSB convention is 0 for running and 1-4 for not, which every script + from the book follows through statusproc. A script with no `status` says + so by failing differently, and '?' is an honest answer.""" + p = os.path.join(initd, name) + try: + r = subprocess.run([p, "status"], stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, timeout=10 if quick else 60) + except (OSError, subprocess.SubprocessError): + return "?", "" + out = (r.stdout or b"").decode("utf-8", "replace").strip() + if r.returncode == 0: + return "yes", out + if r.returncode in (1, 2, 3, 4): + return "no", out + return "?", out + +def _svc_state_file(name): + return os.path.join(SVC_STATE_DIR, name + ".links") -def cmd_user_delete(args): - u = _user_prefixed(args.name) - home = pkgusr_home(u) - has_home = os.path.isdir(home) - if not _user_exists(u) and not has_home: - printInfo(f"{u}: does not exist.") - return - if _user_exists(u): - pw_dir = pwd.getpwnam(u).pw_dir if pwd else "" - if not pw_dir.startswith(BASE_DIR.rstrip("/") + "/"): - printError(f"{u}: user home {pw_dir} is not under {BASE_DIR}; " - f"refusing to delete.") - return - printHeader(f"Delete shared user {u}") - print() - if _user_exists(u): - print(f" user : {u} (uid {pwd.getpwnam(u).pw_uid})") - if has_home: - print(f" home : {home}") - if not args.run: - printInfo("\nDry run -- re-run with --run to delete.") - return - if not args.yes and not ask_yes_no(f"\nReally delete {u}?"): - printWarning("Aborted.") - return - if _user_exists(u): - _run(["userdel", u]) - if _group_exists(u): - _run(["groupdel", u]) - if has_home and os.path.isdir(home): + +def cmd_service(args): + initd, rcroot = _svc_dirs() + if not initd: + printError("no /etc/rc.d/init.d (nor /etc/init.d) -- no SysV services here") + return 1 + # `service sshd start` and `service start sshd` both read naturally; + # whichever word is an action is the action. + name, action = args.name, args.action + if name in SVC_ACTIONS and (action is None or action not in SVC_ACTIONS): + name, action = action, name + if not name and not action: + action = "list" + if name and not action: + action = "status" + if action not in SVC_ACTIONS: + printError("%s: not one of %s" % (action, ", ".join(SVC_ACTIONS))) + return 2 + + if action == "list": + return _svc_cmd_list(initd, rcroot, args) + + if not name: + printError("which service? packagemanager service list") + return 2 + script = os.path.join(initd, name) + if not os.path.isfile(script): + printError("%s: no such service" % name) + near = [s for s in _svc_all(initd) if name in s] + if near: + printInfo(" did you mean: %s" % ", ".join(near)) + printInfo(" what there is: packagemanager service list") + return 1 + + if action in ("start", "stop", "restart", "reload", "force-reload", "status"): + # STARTING IS NOT ENABLING, and it does not ask the links for + # permission: this is the whole point of being able to run sshd for an + # afternoon without it coming back at the next boot. + if action == "start" and not _svc_enabled_levels(rcroot, name): + printInfo(" %s is not enabled at boot -- starting it anyway" % name) + if not os.access(script, os.X_OK): + printWarning(" %s is not executable -- fixing that first" % script) + _run(["chmod", "755", script]) + return subprocess.run([script, action]).returncode + + # ---- enable / disable: the links, and only the links ------------------- + links = _svc_links(rcroot, name) + if action == "disable": + gone = [p for _lv, kind, _n, p in links if kind == "S"] + if not gone: + printInfo("%s is already not started at boot" % name) + return 0 + # REMEMBER EXACTLY WHAT WAS THERE. The priority number is the only + # thing that cannot be worked out again -- it lives in the + # blfs-bootscripts Makefile, which may be long gone by then. try: - shutil.rmtree(home) + os.makedirs(SVC_STATE_DIR, exist_ok=True) + with open(_svc_state_file(name), "w") as f: + f.write("\n".join(gone) + "\n") except OSError as e: - printError(f" rmtree {home}: {e}") - printSuccess(f"{u} deleted.") + printWarning(" could not record the links (%s) -- enable will " + "rebuild them from the script's header" % e) + for p in gone: + try: + os.unlink(p) + printInfo(" removed %s" % p) + except OSError as e: + printWarning(" %s: %s" % (p, e)) + printSuccess("%s: disabled (it is not stopped -- `service %s stop` does that)" + % (name, name)) + return 0 + + # enable + if _svc_enabled_levels(rcroot, name): + printInfo("%s is already started at boot (runlevels %s)" + % (name, ",".join(_svc_enabled_levels(rcroot, name)))) + return 0 + # 1. the exact links we removed, if we removed them + restored = [] + try: + saved = [l.strip() for l in open(_svc_state_file(name)) if l.strip()] + except OSError: + saved = [] + for p in saved: + d = os.path.dirname(p) + if not os.path.isdir(d): + continue + try: + os.symlink("../init.d/" + name, p) + restored.append(p) + except OSError as e: + printWarning(" %s: %s" % (p, e)) + if restored: + for p in restored: + printInfo(" %s -> ../init.d/%s" % (p, name)) + try: + os.unlink(_svc_state_file(name)) + except OSError: + pass + printSuccess("%s: enabled (restored as it was)" % name) + return 0 + # 2. otherwise build them: the number from a link that survived, the + # runlevels from the script's own header. + num = args.priority or next((n for _lv, _k, n, _p in links), None) or "50" + start, stop = _svc_header_levels(initd, name) + if args.levels: + start = "".join(re.findall(r"[0-6S]", args.levels)) + start = start or "2345" + stop = stop or "016" + made = [] + for lv in start: + d = os.path.join(rcroot, "rc%s.d" % lv) + if not os.path.isdir(d): + continue + p = os.path.join(d, "S%s%s" % (num, name)) + if not os.path.lexists(p): + try: + os.symlink("../init.d/" + name, p) + made.append(p) + except OSError as e: + printWarning(" %s: %s" % (p, e)) + for lv in stop: + d = os.path.join(rcroot, "rc%s.d" % lv) + if not os.path.isdir(d): + continue + p = os.path.join(d, "K%s%s" % (num, name)) + if not os.path.lexists(p): + try: + os.symlink("../init.d/" + name, p) + made.append(p) + except OSError as e: + printWarning(" %s: %s" % (p, e)) + if not made: + printWarning("%s: nothing to link (no rc<N>.d directories?)" % name) + return 1 + for p in made: + printInfo(" %s -> ../init.d/%s" % (p, name)) + printSuccess("%s: enabled at runlevels %s (priority %s)" % (name, start, num)) + printInfo(" it is not started yet: packagemanager service %s start" % name) + return 0 + + +def _svc_cmd_list(initd, rcroot, args): + names = _svc_all(initd) + if not names: + printInfo("no services in %s" % initd) + return 0 + default_lv = _svc_default_runlevel() + printHeader("services (init scripts in %s, boot links in %s/rc<N>.d)" + % (initd, rcroot)) + print(" default runlevel: %s\n" % default_lv) + print(" %-22s %-10s %-9s %s" % ("service", "at boot", "running", "runlevels")) + print(" " + "-" * 60) + for n in names: + lv = _svc_enabled_levels(rcroot, n) + boot = "yes" if default_lv in lv else ("other" if lv else "no") + run = "-" + if not args.quick: + run, _out = _svc_running(initd, n) + print(" %-22s %-10s %-9s %s" % (n, boot, run, ",".join(lv) or "-")) + print() + printInfo(" packagemanager service <name> start|stop|restart|status") + printInfo(" packagemanager service <name> enable|disable (boot only)") + if args.quick: + printInfo(" (--quick: 'running' not asked; drop it to ask each script)") + # THE SERVICES THAT ARE NOT INIT SCRIPTS. /lib/services/* are what `ifup` + # hands an interface to; they never appear in rc<N>.d and cannot be + # started, and not saying so is how "dhcpcd is installed but nothing runs + # it" happens. + for sd in ("/lib/services", "/usr/lib/services"): + if os.path.isdir(sd): + have = sorted(os.listdir(sd)) + if have: + print() + printInfo(" network services in %s (run by ifup, never on their " + "own): %s" % (sd, ", ".join(have))) + printInfo(" an interface uses one by naming it: SERVICE=\"<name>\"" + " in /etc/sysconfig/ifconfig.<iface>") + break + return 0 # =========================================================================== # @@ -8202,7 +11828,7 @@ def cmd_script_status(args): printWarning("the book named no file to check):") for n in _act: u = gather_user(n) - st, why = assess(u, versions.get(u.meta.get("name") or n)) + st, why = assess(u, None, run_cmd=False) printWarning(" %-24s %-14s %s" % (n, st, why)) # ONLY WHAT THIS COMMAND CAN ACTUALLY BUILD. # @@ -8602,7 +12228,109 @@ def _warn_env_companions(vals): % (other, _ENV_SUGGESTIONS.get(other, "..."))) +SYSTEM_ENV_FILE = "/etc/profile.d/pkgusr-system.sh" +SYSTEM_ENV_MARKER = ("# written by `packagemanager env --system` -- " + "regenerate, do not edit") + + +def _system_env_values(): + """This system's own facts, for anything that needs to ask. + + Scripts, launchers and one-off commands keep rediscovering the same few + things -- who the human is, what prefixes this machine uses, where the + shared directories are -- and each rediscovery is a different guess: + `cat /etc/pkgusr/desktop-user` in one place, a hard-coded `u_` in + another, `$HOME/Shared` in a third. + + NOT the PKGUSR_ namespace. The tools READ PKGUSR_BASE, PKGUSR_CONFIG + and PKGUSR_LIB as overrides, so exporting that namespace system-wide + would reconfigure every tool in every shell -- a login that quietly + points the package manager at another tree. PKGUSR_SYS_ is inert: + nothing consumes it, so writing it cannot change what anything does. + """ + main = main_user() or "" + vals = { + "PKGUSR_SYS_MAIN_USER": main, + "PKGUSR_SYS_USER_PREFIX": user_prefix(), + "PKGUSR_SYS_PKG_PREFIX": pkgusr_prefix(), + "PKGUSR_SYS_COLLECTOR_PREFIX": collector_prefix(), + } + home = _home_of(main) if main else None + if home: + vals["PKGUSR_SYS_HOME"] = home + # the hub, whoever asks -- not a path inside one person's home + vals["PKGUSR_SYS_SHARED"] = shared_hub() + return vals + + +def cmd_env_system(args): + """Write, or show, the system-fact variables as a profile fragment.""" + vals = _system_env_values() + missing = [k for k, v in vals.items() if not v] + body = [SYSTEM_ENV_MARKER, + "# This system's own facts. Read them; do not set them here --", + "# `packagemanager config` is where they live and this file is", + "# generated from it.", + ""] + for k in sorted(vals): + if vals[k]: + body.append('export %s="%s"' % (k, vals[k])) + text = "\n".join(body) + "\n" + + if not getattr(args, "run", False): + print(text, end="") + printInfo(" (dry run: --run writes %s)" % SYSTEM_ENV_FILE) + if missing: + printWarning(" not known yet, so not exported: %s" + % ", ".join(missing)) + printInfo(" packagemanager config --main-user <your-login>") + return 0 + + d = os.path.dirname(SYSTEM_ENV_FILE) + if not os.path.isdir(d): + printError("no %s -- nowhere to put a login-shell fragment" % d) + printInfo(" the values are above; export them from your own shell") + return 1 + # NEVER OVERWRITE A FILE WE DID NOT WRITE. + if os.path.exists(SYSTEM_ENV_FILE): + try: + had = open(SYSTEM_ENV_FILE, encoding="utf-8", + errors="replace").read() + except OSError: + had = "" + if SYSTEM_ENV_MARKER not in had: + printError("%s exists and is not ours -- left alone" + % SYSTEM_ENV_FILE) + printInfo(" move it aside, or export these yourself:") + print(text, end="") + return 1 + with open(SYSTEM_ENV_FILE, "w") as f: + f.write(text) + os.chmod(SYSTEM_ENV_FILE, 0o644) + # ROOT WROTE IT, SO ROOT OWNS IT. Writing into an existing file keeps + # that file's owner, and this one was first created inside a package + # user's build window -- so it stayed p_gstreamer10's, a package owning a + # file no package installed, in a directory every login shell sources. + # Re-asserted on every write; a chown that cannot run is not fatal. + try: + if os.stat(SYSTEM_ENV_FILE).st_uid != 0: + _run(["chown", "root:root", SYSTEM_ENV_FILE]) + except OSError: + pass + printSuccess("wrote %s" % SYSTEM_ENV_FILE) + for k in sorted(vals): + if vals[k]: + printInfo(" %-28s %s" % (k, vals[k])) + if missing: + printWarning(" not known yet: %s" % ", ".join(missing)) + printInfo(" login shells read /etc/profile, which reads that directory.") + printInfo(" in this shell, now: . %s" % SYSTEM_ENV_FILE) + return 0 + + def cmd_env(args): + if getattr(args, "system", False): + return cmd_env_system(args) vals = _read_build_env() if args.set: for pair in args.set: @@ -8731,17 +12459,68 @@ def _find_stack_file(name): "stacks", c)) for c in out: if os.path.isfile(c): - return os.path.abspath(c) + found = os.path.abspath(c) + _warn_shadowed_stack(found) + return found return None +def _warn_shadowed_stack(found): + """A stack file found BEFORE the installed one, that differs from it. + + The path as given is tried first, so `packagemanager stack services` run + from ~/lfs read ~/lfs/services.stack -- and a `cfg` entry runs + <that directory>/<name>.sh. An old shadow-pam.sh there ran instead of + the 1.14.262 one in /etc/pkgusr/stacks, printed the old "su and + system-auth are in place", and nothing said which file had run. Say it + when the two differ; a checkout that matches is not worth a word.""" + etc = "/etc/pkgusr/stacks" + if os.path.dirname(found) == etc: + return + inst = os.path.join(etc, os.path.basename(found)) + if not os.path.isfile(inst): + return + try: + import filecmp + differ = not filecmp.cmp(found, inst, shallow=False) + # the cfg scripts beside it are what actually run + d = os.path.dirname(found) + scripts = [f for f in sorted(os.listdir(d)) if f.endswith(".sh") + and os.path.isfile(os.path.join(etc, f)) + and not filecmp.cmp(os.path.join(d, f), + os.path.join(etc, f), shallow=False)] + except OSError: + return + if not differ and not scripts: + return + printWarning(" NOTE: using %s, not the installed %s" % (found, inst)) + if scripts: + printWarning(" cfg scripts that differ from the installed ones and " + "will run from %s: %s" % (d, ", ".join(scripts))) + printWarning(" run from another directory, or give the full path, " + "to use the installed stack") + + def _parse_stack_file(path): """-> list of {kind, target, opts{}} in file order. Raises ValueError with a line number on anything malformed -- a plan half-understood is worse than one refused.""" import shlex entries = [] - kinds = ("book", "git", "tar", "check", "cfg", "avoid") + kinds = ("book", "git", "tar", "check", "cfg", "avoid", "include", + "from-book") + # `from-book svn`: the book entries of THIS file below it read that book + # (an entry's own from-book= wins; `from-book default` ends it). Not + # inherited by an included file, and not passed up to the includer. + _file_book = None + # THE "INCLUDE ONCE" SET BELONGS TO THE WHOLE PARSE, not to one include + # line. Owning it per line meant sway.stack's `include avoid` finished + # and cleared it, so `include media` started fresh and media's own + # `include avoid` spliced the 87 exclusions a second time -- and a third + # for the nesting: 332 entries where there are 158. + _top_parse = getattr(_parse_stack_file, "_spliced", None) is None + if _top_parse: + _parse_stack_file._spliced = {os.path.abspath(path)} # A LINE MAY CONTINUE. # # Every entry had to be one line, so sway.stack's are 300 characters wide @@ -8764,7 +12543,8 @@ def _parse_stack_file(path): if _buf: raise ValueError("%s:%d: line ends with a backslash and nothing follows" % (path, _start)) - for n, raw in _joined: + try: + for n, raw in _joined: line = raw.strip() if not line or line.startswith("#"): continue @@ -8780,6 +12560,64 @@ def _parse_stack_file(path): if kind not in kinds: raise ValueError("%s:%d: unknown entry kind %r (one of %s)" % (path, n, kind, "/".join(kinds))) + # ONE LIST, TWO ENTRY POINTS. + # + # The media chain -- codecs, gstreamer, the WebRTC pieces, + # webkitgtk -- is worth updating on its own without replaying a + # whole desktop. Copying those entries into a second stack file + # would give two lists of the same packages, and two lists drift: + # gst10-libav was added to one of them for three releases before + # anyone noticed the other. + # + # `include <name>` splices another stack's entries in at this + # point, in order. The included file is a stack in its own right, + # so `packagemanager stack media --run` builds just that part. + if kind == "from-book": + if len(parts) != 2: + raise ValueError("%s:%d: from-book takes one book selector" + " (svn, 13.0-systemd, a file)" % (path, n)) + _file_book = None if parts[1] in ("default", "-") else parts[1] + continue + if kind == "include": + if len(parts) != 2: + raise ValueError("%s:%d: include takes one stack name" + % (path, n)) + _inc = parts[1] + _p = _inc if os.path.sep in _inc else os.path.join( + os.path.dirname(os.path.abspath(path)), + _inc if _inc.endswith(".stack") else _inc + ".stack") + if not os.path.isfile(_p): + raise ValueError("%s:%d: include %s: no such stack (%s)" + % (path, n, _inc, _p)) + if os.path.abspath(_p) == os.path.abspath(path): + raise ValueError("%s:%d: include %s includes itself" + % (path, n, _inc)) + # A CYCLE WOULD RECURSE FOR EVER, and the error a reader gets + # from a RecursionError names none of the files involved. + _seen = getattr(_parse_stack_file, "_including", set()) + if os.path.abspath(_p) in _seen: + raise ValueError("%s:%d: include %s: circular include" + % (path, n, _inc)) + # INCLUDE ONCE PER COMPOSITION. + # + # sway.stack includes `avoid` AND `media`, and media.stack + # includes `avoid` too -- so the 87 exclusions were spliced + # three times: 332 entries where there are 158, and every + # avoided package listed three times in the plan. A second + # include of the same file adds nothing by definition, so + # skip it. + _sp = _parse_stack_file._spliced + if os.path.abspath(_p) in _sp: + continue # already in this plan + _sp.add(os.path.abspath(_p)) + _seen = _seen | {os.path.abspath(path)} + _parse_stack_file._including = _seen + try: + entries.extend(_parse_stack_file(_p)) + finally: + _parse_stack_file._including = \ + _seen - {os.path.abspath(path)} + continue if len(parts) < 2: raise ValueError("%s:%d: %r needs a target" % (path, n, kind)) target = parts[1] @@ -8801,9 +12639,16 @@ def _parse_stack_file(path): raise ValueError("%s:%d: a git entry needs url=" % (path, n)) if kind == "tar" and ("url" not in opts or "pkg" not in opts): raise ValueError("%s:%d: a tar entry needs url= and pkg=" % (path, n)) + if kind == "book" and _file_book: + opts.setdefault("from-book", _file_book) + if opts.get("from-book") in ("default", "-"): + del opts["from-book"] entries.append({"kind": kind, "target": target, "opts": opts}) - if not entries: + if not entries: raise ValueError("%s: the stack file has no entries" % path) + finally: + if _top_parse: + _parse_stack_file._spliced = None return entries @@ -8990,7 +12835,97 @@ def _check_stack_refs(rows): return 1 if bad else 0 +def _list_stacks(): + """`packagemanager stack` with no name: what stacks there are. + + Every *.stack in /etc/pkgusr/stacks (and stacks/ beside this tool, when + that is a different directory), with its one-line description, how many + entries it has and how many the cache records as done, and whether a + newer shipped copy (.new) is waiting. avoid.stack is a list of what is + never built, not something to run, and says so. + """ + dirs = [] + for d in ("/etc/pkgusr/stacks", + os.path.join(os.path.dirname(os.path.abspath(__file__)), "stacks")): + if os.path.isdir(d) and os.path.realpath(d) not in [os.path.realpath(x) for x in dirs]: + dirs.append(d) + if not dirs: + printError("no stack directory (/etc/pkgusr/stacks) -- `make install` puts them there") + return 1 + any_behind = [] + # DONE IS WHAT ANY STACK RECORDED. sway includes media, so media's + # packages were built -- and recorded -- under sway, and `media` said + # "1/19 done" with all of it installed. + done_all = set() + for d in dirs: + for f in os.listdir(d): + if f.endswith(".stack"): + done_all |= _stack_done_set(os.path.join(d, f)) + # INSTALLED IS NOT CURRENT. The cache says an entry was built once; it + # never looks again. Compare each book entry's installed version with + # the book, so "79/79 done" cannot hide a sway that is three releases + # behind. One map for the whole book (cached); no book, no judgement. + try: + _bv = book_versions_all() or {} + except Exception: + _bv = {} + for d in dirs: + files = sorted(f for f in os.listdir(d) if f.endswith(".stack")) + printHeader("stacks in %s" % d) + if not files: + printInfo(" (none)") + continue + for f in files: + path = os.path.join(d, f) + name = f[:-len(".stack")] + desc = "" + try: + first = open(path, encoding="utf-8", errors="replace").readline() + if " -- " in first: + desc = first.split(" -- ", 1)[1].strip() + except OSError: + pass + try: + # what it BUILDS: the included avoid list is not an entry + entries = [e for e in _parse_stack_file(path) if e["kind"] != "avoid"] + except (OSError, ValueError) as ex: + print(" %-10s (does not parse: %s)" % (name, ex)) + continue + behind = "" + if name == "avoid": + state = "never built" + else: + done = len(done_all + & {"%s %s" % (e["kind"], e["target"]) for e in entries}) + state = "%d/%d done" % (done, len(entries)) + if _bv: + old_n = 0 + for e in entries: + if e["kind"] != "book": + continue + want = _bv.get(e["target"]) or _bv.get(e["target"].lower()) + have = gather_user(e["target"]).version if want else None + if want and have and not _same_version(want, have): + old_n += 1 + if old_n: + behind = " %d entr%s behind the book" % (old_n, "y" if old_n == 1 else "ies") + any_behind.append(name) + new = " (.new waiting: --take-new)" if os.path.isfile(path + ".new") else "" + print(" %-10s %-15s %s%s%s" % (name, state, desc, behind, new)) + print() + printInfo(" plan one: packagemanager stack <name>") + printInfo(" where it is, and what is behind (dependencies too):" + " packagemanager stack <name> --status") + if not _bv: + printInfo(" (no book on disk: versions not compared)") + printInfo(" (\"done\" means built once -- a stack run never updates; " + "packagemanager update --stack <name> does)") + return 0 + + def cmd_stack(args): + if not args.name: + return _list_stacks() path = _find_stack_file(args.name) if not path: printError("no stack file for %r (looked in ., /etc/pkgusr/stacks, " @@ -9052,15 +12987,40 @@ def cmd_stack(args): sys.exit(1) printSuccess("took the shipped stack file; yours is %s" % _bak) entries = _parse_stack_file(path) - printInfo(" %d entries now." % len(entries)) + # THE SAME MEASURE THE HEADER USES. This said "149 entries + # now" while the header two lines later said "(62 entries)": + # one counts every line of the file, the other the buildable + # plan after `avoid`. Two numbers for one file, both called + # entries. + printInfo(" %d line(s) in the file; the plan below is what" + " will be built." % len(entries)) print() _new = None if _new and os.path.isfile(_new): + # WHAT DIFFERS, NOT HOW MANY. The notice printed + # (149 entries, this file has 149) + # for two files that differ in every entry's options: the counts + # were equal, so its own evidence argued there was nothing to + # take. An entry count cannot see a changed expect= or a + # reordered library. printWarning("a newer version of this stack ships beside it:") - printWarning(" %s%s" % (_new, - "" if _n_entries is None else - " (%d entries, this file has %d)" - % (_n_entries, len(entries)))) + printWarning(" %s" % _new) + try: + import difflib as _dl + a = open(path, encoding="utf-8", errors="replace").readlines() + b = open(_new, encoding="utf-8", errors="replace").readlines() + added = sum(1 for l in _dl.unified_diff(a, b, n=0) + if l.startswith("+") and not l.startswith("+++")) + removed = sum(1 for l in _dl.unified_diff(a, b, n=0) + if l.startswith("-") and not l.startswith("---")) + _same = (_n_entries == len(entries)) + printWarning(" %d line(s) added, %d removed%s" + % (added, removed, + "; the entry count is the same, so the change" + " is in the entries themselves" + if _same and (added or removed) else "")) + except OSError: + pass printInfo(" diff %s %s" % (path, _new)) printInfo(" ...and take what you want; this run uses YOUR file.") print() @@ -9075,6 +13035,32 @@ def cmd_stack(args): avoid = [e["target"] for e in entries if e["kind"] == "avoid"] entries = [e for e in entries if e["kind"] != "avoid"] + if getattr(args, "avoided", False): + # the list the header only counts + printHeader("never built by stack %s (%d)" % (args.name, len(avoid))) + for a in sorted(set(avoid), key=str.lower): + print(" " + a) + return + # THE BOOKS THIS STACK READS. Its from-book entries become the + # packages' book for this run and for every install it starts; the + # book itself is refreshed once here (svn moves daily), and every + # lookup after that reads the file on disk. + _sb = {e["target"]: e["opts"]["from-book"] for e in entries + if e["kind"] == "book" and e["opts"].get("from-book")} + if _sb: + _stack_book_map().update({k.lower(): v for k, v in _sb.items()}) + _BOOK_FOR_PKG.clear() + os.environ["PM_STACK_BOOKS"] = ",".join( + "%s=%s" % kv for kv in sorted(_sb.items())) + for _sel in sorted(set(_sb.values())): + _bp = _book_path_for(_sel, refresh=not getattr(args, "status", + False)) + if not getattr(args, "status", False): + printInfo(" book %s: %d entr%s (%s)" + % (_sel, sum(1 for v in _sb.values() if v == _sel), + "y" if list(_sb.values()).count(_sel) == 1 + else "ies", + os.path.basename(_bp) if _bp else "NOT ON DISK")) if getattr(args, "no_git", False): # git and tar entries are the extras -- cloned tools, fonts. Skipping # them finishes the book part of the desktop first, and matters while @@ -9109,7 +13095,11 @@ def cmd_stack(args): # stack" and then passed nothing to the installer, so the # packages were built anyway. It is the stack's exclusion list # now, handed to every entry's resolve. - print(" never built by this stack: %s\n" % ", ".join(avoid)) + # A COUNT, NOT THE LIST: eighty names above every plan pushed + # the plan itself off the screen. + print(" %d package(s) never built by this stack " + "(list: packagemanager stack %s --avoided)\n" + % (len(set(avoid)), args.name)) print(" file: %s\n" % path) import shutil as _sh @@ -9189,6 +13179,23 @@ def cmd_stack(args): # (entries are already filtered to --only above; the old `only` # variable was removed with the duplicate implementation, and this # reference outlived it -- NameError on every stack run.) + # A CACHED cfg ENTRY IS RE-CHECKED WHEN IT CAN BE. + # + # app-users puts the pam_exec line into /etc/pam.d/su; shadow's + # configure phase later rewrote su without it, and every stack run + # after that said `cfg app-users done (stack cache)` -- so + # `su - u_browser-mini` asked for a password and nothing re-ran the + # step that fixes it. A cfg entry's expect= states what it keeps + # true; when that no longer holds, the cache is wrong, not the check. + if (e["_key"] in donekeys and e["kind"] == "cfg" + and e["opts"].get("expect") + and subprocess.run(["/bin/bash", "-c", e["opts"]["expect"]], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL).returncode != 0): + donekeys.discard(e["_key"]) + if not getattr(args, "status", False): + printWarning(" cfg %s is cached as done, but its check fails" + " now -- running it again" % e["target"]) if e["_key"] in donekeys: rows.append((e, "done (stack cache)")) if not getattr(args, "status", False): @@ -9235,11 +13242,29 @@ def cmd_stack(args): notinbook = [e["target"] for e, st in rows if st == "NOT IN BOOK"] if notinbook: printWarning("\nnot in the book: " + ", ".join(notinbook) - + "\n fix the entry (rename to the book's anchor, or make it" - "\n a git entry with build=/install= commands)") - if args.run: + + "\n WHY is the first question -- the page may be there under" + "\n another anchor, or this book may be a different one than" + "\n you think:" + + "".join("\n blfs debug %s" % t for t in notinbook) + + "\n then: rename the entry to the book's anchor, or make it a" + "\n tar/git entry with build=/install= commands.") + if args.run and not getattr(args, "skip_unknown", False): + # ONE BAD LINE MUST NOT COST THE OTHER EIGHT. Refusing the whole + # plan is right by default -- a plan that silently drops entries + # is how a stack looks finished while a package is missing -- but + # an entry the book has no page for blocked the eight that build + # and the checks that report, with no way past it but editing the + # file. printError("refusing to run a plan with unknown targets.") + printInfo(" to run the rest and come back to it:" + "\n packagemanager stack %s --run --skip-unknown" + % os.path.basename(path)) sys.exit(1) + if args.run: + printWarning(" --skip-unknown: leaving %d entr%s out of this run" + % (len(notinbook), "y" if len(notinbook) == 1 else "ies")) + entries = [e for e, st in rows if st != "NOT IN BOOK"] + rows = [(e, st) for e, st in rows if st != "NOT IN BOOK"] missing = [e["target"] for e, st in rows if st == "MISSING SCRIPT"] if missing: @@ -9281,11 +13306,31 @@ def cmd_stack(args): pkgs.append((anchor, nv)) done_pkgs = [(a, v) for a, v in pkgs if "not installed" not in install_tag(a, v)] todo = [(a, v) for a, v in pkgs if (a, v) not in set(done_pkgs)] - done_n = sum(1 for _, st in rows if st == "done (stack cache)") + # DONE BY ANY STACK: media's entries are built through sway, which + # includes it, and recorded in sway's cache -- "1 of 19 done" with + # 189 of 215 packages installed. + _done_any = set() + for _sd in {os.path.dirname(path), "/etc/pkgusr/stacks"}: + if os.path.isdir(_sd): + for _sf in os.listdir(_sd): + if _sf.endswith(".stack"): + _done_any |= _stack_done_set(os.path.join(_sd, _sf)) + done_n = sum(1 for e, st in rows + if st == "done (stack cache)" or e.get("_key") in _done_any + or ("%s %s" % (e["kind"], e["target"])) in _done_any) + _tags = {(a, v): install_tag(a, v) for a, v in done_pkgs} + behind = [a for (a, v), t in _tags.items() if "book is newer" in t] printHeader("stack %s" % args.name) print(" entries : %d of %d done" % (done_n, len(rows))) if pkgs: print(" packages: %d of %d installed" % (len(done_pkgs), len(pkgs))) + if behind: + # INSTALLED IS NOT CURRENT, and --run never updates. Say how many, + # and the one command that does. + print(" behind : %d installed older than the book (%s%s)" + % (len(behind), ", ".join(behind[:5]), " ..." if len(behind) > 5 else "")) + print(" update them, with what depends on them: " + "packagemanager update --stack %s" % args.name) if skipped_by_avoid: print(" avoided : %d named in the stack (%s%s)" % (len(skipped_by_avoid), @@ -9294,8 +13339,20 @@ def cmd_stack(args): nxt_entry = next((e["target"] for e, st in rows if st != "done (stack cache)"), None) print(" next : %s" % (nxt_entry or "nothing left to do")) if todo: - print(" building next: " + ", ".join(a for a, _ in todo[:6]) + # SAY WHAT A RUN WOULD DO, because the list above reads like a + # plan and is not one. `--run` builds what is MISSING; it never + # updates a package that is installed, however far behind the + # book it is. On a system built from an older book that is half + # the tree marked "(update available)", and it looked like + # `stack media --run` was about to rebuild glib2. + print(" to build (missing): " + ", ".join(a for a, _ in todo[:6]) + (" ..." if len(todo) > 6 else "")) + print(" %d missing in total. Installed packages are NOT" + " touched by --run," % len(todo)) + print(" whatever the book now carries: \"book is newer\" is a" + " report, not a plan.") + print(" one package, deliberately: packagemanager update" + " <name> --run") if getattr(args, "full", False): print() for a, v in pkgs: @@ -9379,7 +13436,7 @@ def cmd_stack(args): # `needs=` closes that: the named packages are installed -- # through the same door as everything else -- before the entry # runs. - for _need in [x.strip() for x in + for _need in [_resolve_need(x.strip()) for x in (e["opts"].get("needs") or "").replace(",", " ").split() if x.strip()]: if gather_user(_need).state == "installed": @@ -9439,9 +13496,38 @@ def cmd_stack(args): install_local(target, spath, True, args.yes, "install") else: # check / cfg -- a root script through the one runner sp = _stack_root_script(e, stack_dir) + # THE SETTINGS COME FROM THE RESOLVER, NOT FROM EACH SCRIPT. + # power.sh parsed packagemanager.conf itself, with the lfs + # config's JSON syntax, on a key=value file -- so on the booted + # machine it found nothing and stopped the stack. The resolver + # knows every source (lfs config, this conf, the collector + # groups on the system); a cfg script reads the environment. + # Order: environment, lfs config, packagemanager.conf (both via + # load_config, which lets the lfs config win), then what the + # system shows. A built-in default is never exported. + _cenv = dict(os.environ) + _known = _adopt_existing_settings() + _file_keys = set() + try: + with open(CONFIG_PATH) as _cf: + _file_keys = {l.split("=", 1)[0].strip() for l in _cf + if "=" in l and not l.lstrip().startswith("#")} + except OSError: + pass + for _key, _var in (("collector_prefix", "LFS_COLLECTOR_PREFIX"), + ("main_user", "LFS_MAIN_USER")): + if _cenv.get(_var): + continue + if _key in _known and _known[_key][1] == "the lfs config": + _cenv[_var] = _known[_key][0] + elif _key in _file_keys and load_config().get(_key): + _cenv[_var] = str(load_config()[_key]).rstrip("_") + elif _key in _known: + _cenv[_var] = _known[_key][0] rc = subprocess.run(["lfs-helper", "run-script", "--user", "root", "--as-root", "--script", sp, - "--phase", "all", "--name", ""]).returncode + "--phase", "all", "--name", ""], + env=_cenv).returncode if rc != 0: printError("stack stopped at %s %r (exit %d, %d of %d done)." % (kind, target, rc, done, len(rows))) @@ -9453,14 +13539,146 @@ def cmd_stack(args): # installed (prog=/have=); if that is still absent after a build the # tools said succeeded, say so and leave the entry unrecorded rather # than hiding it forever. - if e["kind"] in ("git", "tar") and not _git_installed(e): + # EXPECT: A FEATURE IS NOT PROVED BY A FILE EXISTING. + # + # ffmpeg and SDL3 autodetect ALSA. Built before alsa-lib existed -- + # which is what happens when a library is listed AFTER a package + # whose dependency plan pulls its consumers in -- they configure + # cleanly, install cleanly, pass have=, and have no sound. Nothing + # in a build log says so, because nothing went wrong. + # + # expect="<shell>" runs after the entry and must succeed. It is the + # only thing here that checks what was built rather than whether a + # build happened. + _expect = e["opts"].get("expect") + _expect_ok = True + if _expect: + printInfo(" checking what %s was built with: %s" % (target, _expect)) + _erc = subprocess.run(["/bin/bash", "-c", _expect], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL).returncode + if _erc != 0: + _expect_ok = False + printError("%s installed, but the feature check failed:" % target) + printError(" %s" % _expect) + if e["kind"] == "cfg": + # A cfg STEP HAS NO OPTIONAL DEPENDENCIES. Its check is + # what the step keeps true; after running it, that is + # still false -- the step's own output above says why. + printWarning(" This is a configuration step: it ran, and what it" + " keeps true is still not true.") + printWarning(" Read its output above for the part it could not" + " do, then run it again:") + printInfo(" packagemanager stack %s --only %s --run" + % (os.path.basename(path).replace(".stack", ""), + target)) + _expect_msg_done = True + else: + _expect_msg_done = False + if _erc != 0 and not _expect_msg_done: + printWarning(" The package is built and installed -- it is" + " MISSING A FEATURE, which almost always means") + printWarning(" its optional dependency was not installed yet" + " when it configured.") + printInfo(" What the check says is missing is usually fixed by:") + printInfo(" packagemanager install <the dependency> --run") + # A STACK ENTRY DOES NOT REBUILD AN INSTALLED PACKAGE. + # + # This used to suggest `stack <file>.stack --only <pkg> --run` + # -- with the .stack suffix, which is not a stack name -- and + # that command runs `install`, which answers "everything is + # installed and up to date" and changes nothing. Re-running + # it can never fix the feature, so recommending it was a loop. + # ...AND THE RIGHT COMMAND DEPENDS ON THE KIND OF ENTRY. + # + # `update --reinstall --regenerate` regenerates from the + # BOOK, and a git or tar entry has no book page -- for + # gst-plugins-rs that advice could not work at all. Those + # are rebuilt by the stack itself, with --force. + if e["kind"] == "book": + printInfo(" %s" % " ".join(_rebuild_cmd(target))) + printInfo(" Re-running this stack entry will NOT rebuild" + " it: the package is installed,") + printInfo(" so the entry's install step has nothing to" + " do. The line above does.") + else: + printInfo(" packagemanager stack %s --only %s --run" + " --force" + % (os.path.basename(path).replace(".stack", ""), + target)) + printInfo(" This is a %s entry, not a book page: there is" + " nothing to regenerate" % e["kind"]) + printInfo(" from the book, and --force is what rebuilds" + " it. (`update --reinstall`") + printInfo(" would look for a book page that does not" + " exist.)") + # AND SAY WHAT have= PROVED, WHICH IS LESS THAN IT LOOKS. + # A part-installed package satisfies it: gst-plugins-rs's + # first run installed libgstrswebrtc.so and then died on the + # next plugin, so have= said "installed" and audiornnoise + # was never there. + _have_f = e["opts"].get("have") + if _have_f: + printWarning(" NOTE: have=%s exists, which is why this" + " was skipped as installed --" % _have_f) + printWarning(" but one file is not the whole package." + " The probe is the better answer.") + + # A BOOK ENTRY WAS RECORDED DONE WITHOUT BEING CHECKED AT ALL. + # + # The guard below covered git and tar only. poppler -- a `book` + # entry -- came out `done (stack cache)` while `pkg-config + # --list-all | grep poppler` found nothing: not installed, and the + # stack skipped it on every later run, so zathura-pdf-poppler kept + # failing on a dependency the plan believed it had. + # + # Every kind gets the same question now: is it there AFTERWARDS? + # For a book package that is the package state, which is what + # `verify` reads -- an empty manifest is not an install. + _installed_now = True + _missing_what = "" + if e["kind"] in ("git", "tar"): + _installed_now = _git_installed(e) + _missing_what = (e["opts"].get("have") + or e["opts"].get("prog") or target) + elif e["kind"] == "book": + _u_now = gather_user(target) + _st_now, _ = assess(_u_now, None, run_cmd=False) + _installed_now = _st_now in ("installed", "unvalidated") + _missing_what = "%s (%s)" % (target, _st_now) + if not _expect_ok: + pass + elif not _installed_now: printWarning("%s finished, but %s is still missing -- NOT recording" " it as done, so the next run tries again." - % (target, e["opts"].get("have") - or e["opts"].get("prog") or target)) + % (target, _missing_what)) else: _stack_mark_done(path, e["_key"]) done += 1 + # A SECOND PASS IS PART OF THE JOB, not an afterthought. + # + # Optional features are decided at configure time, and a stack builds + # things in one order -- so anything whose optional dependency came later + # in the same run is installed without it. Nothing fails, so this is the + # only place the question gets asked. + if done and not getattr(args, "status", False): + printInfo("") + printInfo("Optional features are decided when a package configures, so" + " anything built") + printInfo("before a dependency it can use was built WITHOUT it. " + " Check, and fix:") + printInfo(" packagemanager late-deps") + printInfo(" packagemanager late-deps --rebuild --run") + + # A RUN THAT LEFT SOMETHING OUT DID NOT FINISH. "all N entries done" with + # a skipped entry is exactly the sentence that makes a missing package + # invisible for a month. + if notinbook and getattr(args, "skip_unknown", False): + printWarning("\nstack %s: %d entries done, %d SKIPPED (not in the book): %s" + % (args.name, done, len(notinbook), ", ".join(notinbook))) + printInfo(" the stack is not complete. Why each one is missing:" + + "".join("\n blfs debug %s" % t for t in notinbook)) + sys.exit(1) printSuccess("\nstack %s: all %d entries done." % (args.name, done)) @@ -9514,6 +13732,18 @@ def cmd_reclaim(args): if os.geteuid() != 0: printError("reclaim changes ownership; run it as root.") return 1 + if getattr(args, "all", False): + # ONE ACCIDENT, EVERY PACKAGE. A tree-wide chown -- lfs-kernel's + # headers step gave p_linux-headers all of /usr/include -- is not a + # per-package problem, and naming the victims one at a time means + # finding them first. + args.packages = sorted(_pkg_dirs()) + printInfo("checking %d installed package(s) against their manifests" + % len(args.packages)) + elif not args.packages: + printError("name a package, or --all") + return 2 + rc = 0 for name in args.packages: acct = pkgusr_name(name) @@ -9521,6 +13751,11 @@ def cmd_reclaim(args): try: paths = [l.strip() for l in open(manifest) if l.strip()] except OSError as e: + # under --all a package with no manifest is not an error, it is + # a package that has not recorded one -- saying so 90 times + # buries the packages that DID lose files + if getattr(args, "all", False): + continue printError("%s: cannot read %s (%s)" % (acct, manifest, e)) rc = 1 continue @@ -9562,6 +13797,198 @@ def cmd_reclaim(args): return rc +def _setid_table(): + """setid.conf as {path: (mode, owner, group, package)}. + + The 5th column (package) is the one this command is here for: it is the + only written-down statement of WHICH package installs a setuid file, and + a setuid file held by any other package is damage, not history. + """ + out = {} + for d in (os.environ.get("PM_STACK_DIR") or "/etc/pkgusr/stacks", + os.path.join(os.path.dirname(os.path.abspath(__file__)), "stacks")): + conf = os.path.join(d, "setid.conf") + if not os.path.isfile(conf): + continue + try: + with open(conf, errors="replace") as f: + for line in f: + line = line.strip() + if not line or line.startswith("#"): + continue + col = [c.strip() for c in line.split("|")] + if len(col) < 2 or not col[0].startswith("/"): + continue + col += [""] * (5 - len(col)) + out[col[0]] = (col[1], col[2] or "root", col[3] or "-", col[4]) + except OSError: + continue + break + return out + + +def cmd_disown(args): + """Take back the files a package cannot have installed. + + `reclaim` is the other direction: it pushes ownership TOWARDS pkg.lst, + which is right when a package lost files and wrong when the record is + what went bad. This one distrusts the record, and it exists because the + record did go bad -- three times, the same way: + + -rwx--x--x 1 p_webkitgtk p_webkitgtk /usr/bin/su + lrwxrwxrwx 1 p_webkitgtk p_webkitgtk /usr/sbin/iptables -> xtables-nft-multi + + Neither is webkitgtk's. They were swept into its manifest because + attribution is by TIME WINDOW and a webkitgtk build runs for hours, and + the lease then acted on that manifest -- clearing su's setuid bit on the + way, because `chown` does that and says nothing. 1.14.310 shuts both + doors; this repairs machines that already went through them. + + Two rules, and nothing is guessed from either: + + * setid.conf names the package that installs each setuid file. Anyone + else holding it gives it back to root, with the mode from the table. + * a symlink is installed by whoever installed its target. A link whose + target belongs to another package goes back to that package. + + Whatever is handed back is struck from the package's pkg.lst as well: + an ownership fix that leaves the record claiming the file is a fix that + undoes itself on the next `--clean`, which would DELETE these paths. + """ + if args.run and os.geteuid() != 0: + printError("disown changes ownership; run it as root.") + return 1 + + setid = _setid_table() + if not setid: + printWarning("no setid.conf found -- only the symlink rule can run") + + if getattr(args, "all", False): + names = sorted(_pkg_dirs()) + elif args.packages: + names = args.packages + else: + printError("name a package, or --all") + return 2 + + accounts = set() + for root in pkgusr_roots(): + try: + accounts.update(os.listdir(root)) + except OSError: + pass + + def owner_of(path, follow=False): + try: + st = os.stat(path) if follow else os.lstat(path) + return pwd.getpwuid(st.st_uid).pw_name + except (KeyError, OSError): + return None + + rc, total = 0, 0 + for name in names: + acct = pkgusr_name(name) + pkg = re.sub(r"^" + re.escape(pkgusr_prefix()), "", acct) + lst = os.path.join(pkgusr_home(acct), "pkg.lst") + try: + paths = [l.strip() for l in open(lst, errors="replace") if l.strip()] + except OSError as e: + if getattr(args, "all", False): + continue + printError("%s: cannot read %s (%s)" % (acct, lst, e)) + rc = 1 + continue + + wrong = [] # (path, reason, give-to, mode-or-None) + for path in paths: + if owner_of(path) != acct: + continue + pol = setid.get(path) + if pol and pol[3] != pkg: + who = pol[3] or "no package" + wrong.append((path, "setid.conf says %s installs it" % who, + "root", pol[0])) + continue + if os.path.islink(path): + tgt = os.path.realpath(path) + to = owner_of(tgt, follow=True) + if to and to != acct and to != "root" and to in accounts: + wrong.append((path, "points at %s's %s" % (to, tgt), to, None)) + + if not wrong: + if not getattr(args, "all", False): + printSuccess("%s: nothing it could not have installed (%d checked)" + % (acct, len(paths))) + continue + + total += len(wrong) + printWarning("%s: %d recorded path(s) belong somewhere else" % (acct, len(wrong))) + for path, why, to, _m in wrong: + printInfo(" %-46s -> %-12s %s" % (path, to, why)) + if not args.run: + printInfo(" Dry run -- re-run with --run to hand them back.") + continue + + done = [] + for path, _why, to, mode in wrong: + try: + if to == "root": + os.lchown(path, 0, 0) + # THE MODE COMES BACK TOO. The chown that took the file + # cleared its setuid bit; handing it to root without the + # bit leaves `su` exactly as broken as it was. + if mode and not os.path.islink(path): + os.chmod(path, int(mode, 8)) + else: + os.lchown(path, pwd.getpwnam(to).pw_uid, grp.getgrnam(to).gr_gid) + done.append(path) + except (OSError, KeyError) as e: + printError(" could not hand back %s (%s)" % (path, e)) + rc = 1 + + # ...AND OUT OF THE RECORD, or `update <pkg> --clean` deletes them. + if done: + keep = [p for p in paths if p not in set(done)] + try: + with open(lst + ".new", "w") as f: + f.write("\n".join(keep) + ("\n" if keep else "")) + os.replace(lst + ".new", lst) + except OSError as e: + printError(" %s: could not rewrite pkg.lst (%s)" % (acct, e)) + rc = 1 + # lfs-helper's build manifests: $LFS_PKGUSR_DIR/manifests/<pkg>.files, + # which defaults to /usr/src/lfs-pkgusr. It is the list the LEASE + # reads, so leaving the path in it re-arms the whole thing. + _state = (os.environ.get("LFS_PKGUSR_DIR") + or os.path.join(os.path.dirname(BASE_DIR.rstrip("/")) or "/usr/src", + "lfs-pkgusr")) + for man in dict.fromkeys( + [os.path.join(_state, "manifests", pkg + ".files"), + os.path.join("/usr/src/lfs-pkgusr/manifests", pkg + ".files")]): + if not os.path.isfile(man): + continue + try: + lines = [l.rstrip("\n") for l in open(man, errors="replace")] + left = [l for l in lines if l.strip() not in set(done)] + if len(left) != len(lines): + with open(man + ".new", "w") as f: + f.write("\n".join(left) + ("\n" if left else "")) + os.replace(man + ".new", man) + printInfo(" %s: %d line(s) removed" % (man, len(lines) - len(left))) + except OSError as e: + printError(" %s (%s)" % (man, e)) + rc = 1 + printSuccess("%s: %d path(s) handed back" % (acct, len(done))) + + if total == 0: + printSuccess("nothing to hand back") + elif not args.run: + printInfo("") + printInfo(" After --run, put the setuid bits back and check them:") + printInfo(" lfs-helper setid --run") + return rc + + def cmd_which_package(args): """Which package installed this path. @@ -9885,6 +14312,20 @@ commands, by what you are doing: fix-group-dir / fix-install repair permissions + a whole environment at once + stack <name> replay a stack file: packages, order, options + stack <name> --only <pkg> one entry of it + stack <name> --take-new take the shipped stack beside your edited one + bootstrap get a freshly booted system to where this works + env build-time variables the book expects exported + + features, not files + probe [<pkg>] run the stacks' expect= checks against what is + installed: is the FEATURE there, not the file + late-deps what was built before a dependency it can use + late-deps --rebuild --run rebuild those, in dependency order + late-deps <pkg> --why the edges it used, and which names resolved + settings and housekeeping config show or change settings paths where everything lives @@ -9892,6 +14333,11 @@ commands, by what you are doing: errors what went wrong in the last run plan what a run would do, before doing it reload-pkg-list refresh the cached package list + reclaim <pkg> | --all give a package back files another took from it + disown <pkg> | --all the other direction: take back what a package + cannot have installed (setuid files, other + packages' symlinks) and clean its pkg.lst + which-package <path> which package installed it blfs ... pass a command straight to blfs LFS base packages are the `lfs` tool's job; BLFS packages are handled here. @@ -10113,13 +14559,38 @@ def _file_md5(path): return None +def _in_chroot(): + """Is there a host at all? + + The whole point of the staleness warning is "the copy outside is newer + than the copy in here", and outside stops existing the moment the system + boots on its own. /proc/1/root is pid 1's root: in a chroot it is not + this root. + """ + try: + return os.stat("/proc/1/root/.").st_ino != os.stat("/").st_ino \ + or os.stat("/proc/1/root/.").st_dev != os.stat("/").st_dev + except OSError: + return False + + def warn_if_tools_are_stale(): - """Tell the user when the chroot is running older tools than the host. + """Tell the user when the CHROOT is running older tools than the host. The tools are copied into the tree; edit them outside and the chroot keeps running the old copy until they are copied again. That is a genuinely confusing failure mode -- behaviour that was fixed "yesterday" reappears -- - so say so plainly.""" + so say so plainly. + + ON A BOOTED SYSTEM THERE IS NO HOST. The stamp then records what was + copied in during the build, so every later `make install` on the machine + itself made all seven tools "differ" and this printed, before every + command, an instruction to run sync-tools on a host that no longer + exists. `make install` writes the stamp now; this says nothing unless + there is an outside to be behind. + """ + if not _in_chroot(): + return stamp = "/usr/share/lfs/tool-stamps.json" if not os.path.isfile(stamp): return @@ -10229,6 +14700,9 @@ def main(): p.add_argument("--clean", action="store_true", help="after install, remove stale files from a previous " "version (configs and the source tree kept)") + p.add_argument("--no-git-check", action="store_true", + help="do not ask the git remotes whether a branch moved " + "(no network)") p.add_argument("--regenerate", action="store_true", help="rebuild scripts from the book, ignoring any edited " "script in the package user's home") @@ -10263,6 +14737,13 @@ def main(): p.add_argument("-R", "--dependents", action="store_true", help="also update everything that (recursively) depends on the " "named package(s) -- rebuild reverse deps too") + p.add_argument("-v", "--verbose", action="store_true", + help="show every requirement that differs, not the first 12") + p.add_argument("--stack", metavar="NAME", + help="only the packages of that stack (media, sway, " + "network, services, vision) -- 'what in the media " + "chain has the book moved past', without typing the " + "names. Combine with --select to pick from them.") p.add_argument("--select", action="store_true", help="interactively pick which packages of the plan to update " "(all selected by default; combine with --run to update the " @@ -10366,12 +14847,21 @@ def main(): p.add_argument("--refresh", action="store_true", help="drop scripts from an older blfs so the current " "questions can be seen") + p.add_argument("--system", action="store_true", + help="this system's own facts -- main user, the three " + "prefixes, the Shared directory -- as PKGUSR_SYS_* " + "in /etc/profile.d/pkgusr-system.sh, so a script can " + "ask instead of guessing") + p.add_argument("--run", action="store_true", + help="with --system: write the file (default: print it)") p.set_defaults(func=cmd_env) p = sub.add_parser( "stack", help="replay a stack file: one command, one whole environment") - p.add_argument("name", help="stack name or path (e.g. sway, ./sway.stack)") + p.add_argument("name", nargs="?", + help="stack name or path (e.g. sway, ./sway.stack); " + "without one, list the stacks there are") p.add_argument("--run", action="store_true", help="actually build (default: print the plan)") p.add_argument("--yes", "-y", action="store_true", @@ -10388,6 +14878,9 @@ def main(): p.add_argument("--ignore-prereqs", action="store_true", help="start entries even when a build tool or variable " "their scripts need is missing") + p.add_argument("--skip-unknown", action="store_true", + help="run the entries the book does have, and report the " + "others (exits non-zero: the stack is NOT complete)") p.add_argument("--only", action="append", metavar="ENTRY", help="run just these entries (repeatable or comma-separated); " "they are rebuilt even if the cache says done") @@ -10397,16 +14890,34 @@ def main(): p.add_argument("--check-refs", action="store_true", help="ask each git entry's remote whether its ref= still " "exists, and stop -- builds nothing") + p.add_argument("--avoided", action="store_true", + help="list the packages this stack never builds, and stop") p.set_defaults(func=cmd_stack) p = sub.add_parser("reclaim", help="give a package back files another package took " "from it (ownership only; nothing is rebuilt)") - p.add_argument("packages", nargs="+") + p.add_argument("packages", nargs="*") + # ONE ACCIDENT, EVERY PACKAGE. A tree-wide chown (lfs-kernel's headers + # step gave p_linux-headers all of /usr/include) is not a per-package + # problem, and naming the victims one at a time means finding them first. + p.add_argument("--all", action="store_true", + help="every installed package, not a named list") p.add_argument("--run", action="store_true", help="actually change ownership (default: dry run)") p.set_defaults(func=cmd_reclaim) + p = sub.add_parser("disown", + help="hand back the files a package cannot have " + "installed (setuid files and other packages' " + "symlinks), and strike them from its pkg.lst") + p.add_argument("packages", nargs="*") + p.add_argument("--all", action="store_true", + help="every installed package, not a named list") + p.add_argument("--run", action="store_true", + help="actually change ownership (default: dry run)") + p.set_defaults(func=cmd_disown) + p = sub.add_parser("which-package", aliases=["which-pkg"], help="which package installed a path (reads pkg.lst)") p.add_argument("paths", nargs="+", metavar="PATH") @@ -10454,6 +14965,31 @@ def main(): "(configs and source tree kept)") p.set_defaults(func=cmd_script) + p = sub.add_parser("probe", help="run the stacks' expect= feature checks " + "against what is installed") + p.add_argument("package", nargs="*", help="only these (default: all)") + p.set_defaults(func=cmd_probe) + + p = sub.add_parser("late-deps", help="packages built BEFORE a dependency " + "they can use (so its support was compiled out)") + p.add_argument("package", nargs="*", help="only these (default: all)") + p.add_argument("--kind", help="which dependency kinds to check " + "(default recommended,optional; add required to include it)") + p.add_argument("--verbose", action="store_true", + help="also list dependency names that could not be matched") + p.add_argument("--include-docs", action="store_true", + help="include dependencies that only affect documentation " + "or are build tools (graphviz, docutils, git ...)") + p.add_argument("--why", action="store_true", + help="show the dependency edges actually used for the named " + "package(s), and which names did not resolve") + p.add_argument("--rebuild", action="store_true", + help="print the rebuild plan in dependency order; with " + "--run, carry it out") + p.add_argument("--run", action="store_true", + help="with --rebuild: actually rebuild") + p.set_defaults(func=cmd_late_deps) + p = sub.add_parser("errors", help="show which packages failed in the last " "install/update run (with log locations)") p.add_argument("--all", action="store_true", @@ -10488,6 +15024,10 @@ def main(): p.add_argument("--collector-prefix", help="prefix for shared install-dir groups") p.add_argument("--user-prefix", help="prefix for shared users") p.add_argument("--main-user", help="your human login account (e.g. n76310)") + p.add_argument("--shared-hub", metavar="DIR", + help="where the links to the shared directories live " + "(default /home/Shared); your ~/Shared and each " + "account's ~/Shared link into it") p.add_argument("--editor", help="editor for 'verify --set-validate-command' " "(default: $EDITOR or vim)") p.add_argument("--difftool", help="diff tool for 'migrate-scripts --vimdiff' " @@ -10516,6 +15056,15 @@ def main(): formatter_class=argparse.RawDescriptionHelpFormatter) q.add_argument("name") q.add_argument("--groups", help="also add these groups, e.g. audio,video") + q.add_argument("--shared-slim", dest="shared_slim", action="store_true", + help="share the DESKTOP only: the account joins the group " + "on your XDG runtime directory, so it can draw -- and " + "gets no audio and no camera. For an account that " + "only ever runs a browser window.") + q.add_argument("--shared-full", dest="shared_full", action="store_true", + help="the desktop AND audio and video: --shared-slim plus " + "the audio and video groups, for anything that plays " + "sound or uses a camera.") q.add_argument("--shared", action="store_true", help="a SHARED user: joins the group on your XDG runtime " "directory, so it can reach the display, the session " @@ -10537,6 +15086,15 @@ def main(): formatter_class=argparse.RawDescriptionHelpFormatter) q.add_argument("name") q.add_argument("--groups", help="also add these groups, e.g. audio,video") + q.add_argument("--shared-slim", dest="shared_slim", action="store_true", + help="share the DESKTOP only: the account joins the group " + "on your XDG runtime directory, so it can draw -- and " + "gets no audio and no camera. For an account that " + "only ever runs a browser window.") + q.add_argument("--shared-full", dest="shared_full", action="store_true", + help="the desktop AND audio and video: --shared-slim plus " + "the audio and video groups, for anything that plays " + "sound or uses a camera.") q.add_argument("--shared", action="store_true", help="join the group on your XDG runtime directory") q.add_argument("--share-dir", action="store_true", @@ -10546,11 +15104,39 @@ def main(): q.add_argument("--desktop", action="store_true", help="a menu entry") q.add_argument("--app", help="program to launch (default: <name>)") q.set_defaults(func=cmd_user_setup) - q = usub.add_parser("delete", help="delete an shared user + group + home") + q = usub.add_parser("delete", help="remove an application account safely: its " + "home archived first, launchers, pam line, account, group") q.add_argument("name") - q.add_argument("--run", action="store_true", help="actually delete (default dry-run)") + q.add_argument("--run", action="store_true", help="actually remove (default dry-run)") q.add_argument("--yes", action="store_true", help="skip the confirmation prompt") + q.add_argument("--keep-home", action="store_true", help="leave the home directory") + q.add_argument("--no-archive", action="store_true", help="do not archive the home first") q.set_defaults(func=cmd_user_delete) + q = usub.add_parser("purge", help="undo the whole application-account setup: " + "every account, then the session group, the profile " + "fragments and the pam wiring they shared") + q.add_argument("--run", action="store_true", help="actually remove (default dry-run)") + q.add_argument("--yes", action="store_true", help="skip the confirmation prompt") + q.add_argument("--keep-home", action="store_true", help="leave the home directories") + q.add_argument("--no-archive", action="store_true", help="do not archive the homes") + q.add_argument("--accounts", help="only these accounts (comma separated), " + "instead of every one in app-users.conf") + q.add_argument("--conf", help=f"read the account list here (default {APP_USERS_CONF})") + q.set_defaults(func=cmd_user_purge) + q = usub.add_parser("launcher", help="a launcher + menu entry for an existing " + "application account (one account can have several)") + q.add_argument("account") + q.add_argument("name", help="the launcher's name: ~/bin/<name>, <name>.desktop") + q.add_argument("--app", required=True, help="command line; {} = what you type") + q.add_argument("--icon", default="", help="URL, simpleicons.org slug, or - for none " + "(default: the launcher name as the slug)") + q.add_argument("--no-desktop", action="store_true", help="no menu entry") + q.set_defaults(func=cmd_user_launcher) + q = usub.add_parser("remove-launcher", help="remove a launcher, its menu entry and " + "icon (only ones these tools wrote)") + q.add_argument("name") + q.add_argument("--run", action="store_true") + q.set_defaults(func=cmd_user_remove_launcher) q = usub.add_parser("allow", help="add supplementary group(s) to a user " "(e.g. audio video)") q.add_argument("user") @@ -10561,6 +15147,47 @@ def main(): q.add_argument("groups", nargs="+") q.set_defaults(func=cmd_user_disallow) + p = sub.add_parser("git-status", aliases=["git"], + help="what each git package follows (branch or tag) and " + "whether the remote has moved past it") + p.add_argument("packages", nargs="*", help="default: every git package here") + p.add_argument("--local-only", action="store_true", + help="do not ask the remotes (no network)") + p.set_defaults(func=cmd_git_status) + + p = sub.add_parser("git-ref", + help="show the branches and tags a git package could " + "follow, or move it to one") + p.add_argument("package") + p.add_argument("ref", nargs="?", help="omit to list what the remote has") + p.add_argument("--run", action="store_true", + help="actually change git_ref= (default dry-run)") + p.set_defaults(func=cmd_git_ref) + + p = sub.add_parser("used-by", aliases=["needed-by"], + help="what still links against a library, or against " + "this package's libraries (DT_NEEDED, not the " + "book's dependency list)") + p.add_argument("name", help="a package name, a soname (libfoo.so.4) or a " + "path to a library") + p.set_defaults(func=cmd_used_by) + + p = sub.add_parser("service", help="the SysV services: list them, start/stop " + "one, enable/disable it at boot (start does not need " + "enable)") + p.add_argument("name", nargs="?", help="the service, as in /etc/rc.d/init.d/<name>") + p.add_argument("action", nargs="?", + help="list | status | start | stop | restart | reload | " + "enable | disable (either order: `service sshd start` " + "or `service start sshd`; no action means status)") + p.add_argument("--levels", help="enable: the runlevels to start in " + "(default: the script's own Default-Start, else 2345)") + p.add_argument("--priority", help="enable: the two-digit order number " + "(default: the one its other links already use)") + p.add_argument("--quick", action="store_true", + help="list: do not ask each script whether it is running") + p.set_defaults(func=cmd_service) + p = sub.add_parser("init", help="set up the install group + make install dirs " "(dry-run unless --run; shows/confirms each command)") p.add_argument("dirs", nargs="*", help="directories to turn into install dirs") @@ -10717,7 +15344,14 @@ def main(): if (not config_exists() and args.command != "config" and sys.stdin.isatty() and sys.stdout.isatty()): first_run_setup() - args.func(args) + # THE EXIT CODE IS AN ANSWER. Every command's return value was thrown + # away here, so `packagemanager` exited 0 whatever happened -- including + # `service <name> status` on a service that is not running, which is the + # one command people put in an `if`. Only a non-zero INT becomes the exit + # status; a command that returns something else (or nothing) is unchanged. + rc = args.func(args) + if isinstance(rc, int) and not isinstance(rc, bool) and rc != 0: + sys.exit(rc) if __name__ == "__main__": diff --git a/stacks/accounts.conf b/stacks/accounts.conf new file mode 100644 index 0000000..b205193 --- /dev/null +++ b/stacks/accounts.conf @@ -0,0 +1,29 @@ +# accounts.conf -- the SYSTEM accounts this machine needs, declared once. +# +# A package user cannot create an account: every book "Configuring" section +# that says `useradd` is refused during the build and recorded, and then +# somebody types it out of the failure box hours later, from memory. This is +# where it gets written down instead, and `cfg accounts` creates them before +# the package that wants them. +# +# One line per account: +# +# name:uid:gid:home:shell:comment +# +# name the account, and the group of the same name, both created +# uid/gid a number, or `-` for "let the system pick a system id" +# home `-` for none (a daemon account usually needs none) +# shell `-` becomes /bin/false: a system account does not log in +# comment what it is for, so the next reader does not have to guess +# +# THE NUMBERS MATTER WHEN THEY MATTER. The LFS and BLFS books fix a few uids +# on purpose so that a file copied between systems keeps its owner -- sshd is +# 50 in the book, and changing it here does not make the book wrong, it makes +# this machine different. Where the book gives no number, use `-`. +# +# These are created, never removed: this file adding a line is a request, and +# deleting one is not. Remove an account by hand, deliberately, once you know +# nothing owns its files: +# find / -xdev -user <name> | head + +sshd:50:50:/var/lib/sshd:-:OpenSSH privilege separation diff --git a/stacks/accounts.sh b/stacks/accounts.sh new file mode 100644 index 0000000..8ee37fc --- /dev/null +++ b/stacks/accounts.sh @@ -0,0 +1,117 @@ +#!/bin/bash +# accounts.sh -- create the system accounts declared in accounts.conf. +# +# SYSTEM ACCOUNTS ARE ROOT'S, AND THIS IS WHERE THEY HAPPEN. A package user +# cannot create one, so a book's `useradd` line is refused mid-build and +# recorded; that refusal is correct, and the answer is not to type the +# command from the failure box later but to have created the account first. +# +# Idempotent: an account that exists is left exactly as it is, including a +# uid that does not match the file -- changing the uid of an account that +# already owns files is how a system loses track of who owns what, and it is +# not something a stack step should do behind anyone's back. +set -u + +CONF="${ACCOUNTS_CONF:-/etc/pkgusr/stacks/accounts.conf}" +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } + +[ -r "$CONF" ] || { warn "no $CONF to read"; exit 1; } + +made=0 kept=0 +while IFS=: read -r name uid gid home shell comment; do + case "${name:-}" in ""|\#*) continue ;; esac + [ -n "$comment" ] || comment="declared in $(basename "$CONF")" + + # ---- the group first: useradd -g needs it to exist ---------------- + if getent group "$name" >/dev/null 2>&1; then + note "group $name exists (gid $(getent group "$name" | cut -d: -f3))" + else + if [ "$gid" = "-" ]; then + groupadd -r "$name" || { warn "could not create group $name"; continue; } + elif ! groupadd -g "$gid" "$name"; then + # SAY WHO HOLDS THE NUMBER. "could not create group sshd (gid + # 50)" leaves the reader to find out why; groupadd knows, and + # the answer decides what to do about it. + _held="$(getent group "$gid" | cut -d: -f1)" + if [ -n "$_held" ]; then + warn "gid $gid is already group '$_held', so $name cannot have it" + note " either give $name another number in accounts.conf," + note " or write '-' there and let the system choose one." + note " The book's number only matters if files from another" + note " system will arrive owned by it." + else + warn "could not create group $name with gid $gid" + fi + continue + else + note "created group $name" + fi + fi + + # ---- then the account -------------------------------------------- + if id "$name" >/dev/null 2>&1; then + # SAY WHAT IS THERE, including a mismatch, and change nothing. + _have="$(id -u "$name")" + if [ "$uid" != "-" ] && [ "$_have" != "$uid" ]; then + note "$name exists with uid $_have, not the declared $uid --" + note " left alone: it may own files. To move it, deliberately:" + note " find / -xdev -user $name | head then usermod -u $uid $name" + else + note "$name exists (uid $_have)" + fi + kept=$((kept + 1)) + continue + fi + + set -- -c "$comment" -g "$name" + # -r WHEN NO NUMBER IS GIVEN. Without it useradd hands out the next + # ordinary uid: `picky` came out as 1016, in the range real people get, + # for an account that is meant to be a daemon's. + if [ "$uid" = "-" ]; then + set -- "$@" -r + else + set -- "$@" -u "$uid" + fi + case "$home" in + -) set -- "$@" -M -d /dev/null ;; + *) set -- "$@" -m -d "$home" ;; + esac + case "$shell" in + -) set -- "$@" -s /bin/false ;; + *) set -- "$@" -s "$shell" ;; + esac + if useradd "$@" "$name"; then + # CONFIRM YOUR OWN EFFECT: useradd has returned 0 and created + # nothing on this project before. + if id "$name" >/dev/null 2>&1; then + note "created $name (uid $(id -u "$name")), home ${home/-//dev/null}" + made=$((made + 1)) + else + warn "useradd $name returned 0 and the account is not there" + fi + else + warn "could not create $name" + fi + + # the home of a daemon account belongs to it, and nothing else should be + # able to write there + case "$home" in + -) ;; + *) if [ -d "$home" ]; then + chown "$name:$name" "$home" && chmod 700 "$home" \ + && note " $home is $name's, mode 700" + fi ;; + esac +done < "$CONF" + +echo +note "$made created, $kept already there" +if [ "$fail" = 0 ]; then + echo "# accounts: every line in $(basename "$CONF") exists now." + echo "# what a build refused, and still wants doing: lfs-helper setid" +else + echo "# fix the !! lines; each is one line of $CONF" +fi +exit $fail diff --git a/stacks/app-users.conf b/stacks/app-users.conf new file mode 100644 index 0000000..9d7d3dc --- /dev/null +++ b/stacks/app-users.conf @@ -0,0 +1,115 @@ +# app-users.conf -- the application accounts this machine runs GUI apps as. +# +# An application user is an account a program RUNS as, so a browser cannot +# read your documents, your keys, or another application's data. The +# account name gets the configured prefix (`packagemanager config +# --user-prefix`, `u_` by default), so `claude` here becomes u_claude. +# +# `cfg app-users` reads this file and calls `packagemanager user setup` for +# each line, which is idempotent: running it again changes nothing that is +# already right, and it never overwrites a file it did not write. +# +# One line per LAUNCHER, fields separated by `|`, `-` for "not this one". +# Several launchers can run as one account: +# +# account | name | app | share | dir | desktop | launcher | groups | icon +# +# account the account, without the prefix: browser -> u_browser. Also +# the shared directory's name, capitalised: /home/Shared/Browser +# name the launcher: ~/bin/<name> and <name>.desktop. Empty: the +# account's name +# app the command line. Its first word must be on PATH. +# {} what you type, as arguments (browser-mini {}) +# ...?q={} what you type as a search term; with nothing +# typed, the page before the ? (the front page) +# ...?q={URL} the same, but with nothing typed you land on +# URL. reddit's /search/ with no search in it +# is a blank page; this is where you want to be +# instead. `|` cannot appear in it -- it is +# this file's field separator +# no {} nothing you type is passed on +# ~/ the ACCOUNT's home (--download-dir ~/Shared) +# browser-mini and browser-big get --app-id <name> when the line +# has none, so sway and the taskbar tell the windows apart. The +# lines below spell it out anyway, so what sway matches on is +# readable here; the four browser lines leave it to the default. +# share slim the desktop only: it can draw, NO audio, NO camera +# full the desktop plus audio and video -- anything that plays +# no no session access +# An account takes the strongest share of its lines. +# dir yes /home/Shared/<Account> (2770, the account's group -- +# which your login is in), and $SHARED_<ACCOUNT> in your +# next login shell. The account's ~/Shared and yours +# are links into /home/Shared, so its home can stay +# 0700 (`packagemanager config --shared-hub DIR` moves it) +# no nothing shared. What the account downloads stays in the +# account -- which is the point of the account +# desktop yes a menu entry +# launcher yes ~/bin/<name> (a menu entry needs it, so desktop=yes +# writes it too) +# groups extra groups, comma separated; `-` for none +# icon empty https://cdn.simpleicons.org/<name> +# a word that simpleicons.org slug (gemini -> googlegemini) +# a URL that image. Wikimedia's stable form is +# https://commons.wikimedia.org/wiki/Special:FilePath/<File.svg> +# - no icon: the program's own +# #RRGGBB at the end paints a ONE-COLOUR svg that colour, after +# it is fetched: ...ChatGPT-Logo.svg#38BDF8 is sky blue +# Fetched once into ~/.local/share/icons/lfs-apps/. +# simpleicons.org has DROPPED some brands -- `openai` and `grok` +# are gone, and a slug that is not there means no icon. Those +# lines carry a URL instead. +# +# Missing trailing fields take: slim | yes | yes | yes | - | (auto icon). +# +# REMOVING ONE. Take the line out, then: +# packagemanager user remove-launcher <name> --run a launcher only +# packagemanager user delete <account> --run the account: home +# archived to /root first +# packagemanager user purge --run ALL of it, including +# the parts the accounts +# share (see below) +# +# THE SHELL VARIABLES. /etc/profile.d/shared-dirs.sh exports one per +# link in the hub, e.g. SHARED_BROWSER=/home/Shared/Browser, and drops +# the ones whose directory is gone. /etc/profile.d is read by LOGIN shells: +# a terminal opened inside the session is not one, so it shows what the +# session inherited when you logged in. The step also sources the fragment +# from /etc/bashrc, which every interactive shell reads -- and any shell can +# refresh itself with: . /etc/profile.d/shared-dirs.sh +# +# EXTRA SETUP PER ACCOUNT. A line starting with @setup runs a command, a +# local script or a downloaded bash script AS the account (its own HOME), +# once. `*` means every account. A changed line runs again; +# APP_USERS_RERUN=1 runs all of them again. `|` is allowed here. +# +# @setup mpv mkdir -p ~/.config/mpv && echo "volume=60" > ~/.config/mpv/mpv.conf +# @setup claude https://example.org/claude-setup.sh +# @setup * /etc/pkgusr/stacks/app-common.sh +# +# su WITHOUT A PASSWORD: /etc/pam.d/su_u_user lists the accounts your login +# may become; this step MERGES it (backup su_u_user.bak-<date> first). + +account | name | app | share | dir | desktop | launcher | groups | icon +browser | browser-mini | browser-mini {} --download-dir ~/Shared | full | yes | yes | yes | - | - +browser | browser-big | browser-big {} --download-dir ~/Shared | full | yes | yes | yes | - | - +browser | google | browser-mini https://www.google.com/search?q={https://www.google.com/} --download-dir ~/Shared | full | yes | yes | yes | - | +browser | duckduckgo | browser-big https://duckduckgo.com/?q={https://duckduckgo.com/} --download-dir ~/Shared | full | yes | yes | yes | - | +claude | | browser-mini --app-id claude https://claude.ai/new --download-dir ~/Shared | slim | yes | yes | yes | - | +chatgpt | | browser-mini --app-id chatgpt https://chatgpt.com/ --download-dir ~/Shared | slim | yes | yes | yes | - | https://commons.wikimedia.org/wiki/Special:FilePath/ChatGPT-Logo.svg +gemini | | browser-mini --app-id gemini https://gemini.google.com/app --download-dir ~/Shared | slim | yes | yes | yes | - | googlegemini +nrwgpt | | browser-mini --app-id nrwgpt https://chat.kiconnect.nrw/app --download-dir ~/Shared | slim | yes | yes | yes | - | https://commons.wikimedia.org/wiki/Special:FilePath/ChatGPT-Logo.svg#38BDF8 +grok | | browser-mini --app-id grok https://grok.com/ --download-dir ~/Shared | slim | yes | yes | yes | - | https://commons.wikimedia.org/wiki/Special:FilePath/Grok-icon.svg +mercury | | browser-mini --app-id mercury https://chat.inceptionlabs.ai/ --download-dir ~/Shared | slim | yes | yes | yes | - | - +deepseek | | browser-mini --app-id deepseek https://chat.deepseek.com/ --download-dir ~/Shared | slim | yes | yes | yes | - | +github | | browser-mini --app-id github https://github.com/search?q={https://github.com/} --download-dir ~/Shared | slim | yes | yes | yes | - | +git | | browser-mini --app-id git https://git.christianimmanuel.de --download-dir ~/Shared | slim | yes | yes | yes | - | +codeberg | | browser-mini --app-id codeberg https://codeberg.org/explore/repos?q={https://codeberg.org/} --download-dir ~/Shared | slim | yes | yes | yes | - | +reddit | | browser-mini --app-id reddit https://www.reddit.com/search/?q={https://www.reddit.com/} | full | no | yes | yes | - | +discord | | browser-big --app-id discord https://discord.com/channels/@me | full | no | yes | yes | - | +youtube | | browser-mini --app-id youtube https://www.youtube.com/results?search_query={https://www.youtube.com/} | full | no | yes | yes | - | +dokus | arte | browser-mini --app-id arte https://www.arte.tv/de/videos/dokumentationen-und-reportagen/ | full | no | yes | yes | - | https://upload.wikimedia.org/wikipedia/commons/3/3b/Arte-Logo.svg +dokus | ard | browser-mini --app-id ard https://www.ardmediathek.de/dokus | full | no | yes | yes | - | https://upload.wikimedia.org/wikipedia/commons/7/7c/ARD_2019_logo.svg +dokus | zdf | browser-mini --app-id zdf https://www.zdf.de/dokus | full | no | yes | yes | - | - +mpv | orchester | mpv https://www.youtube.com/playlist?list=PLijBjlV15UVRBj4-1D1AXKDlg2QLdPqk8 | full | no | yes | yes | - | mpv +mpv | lieder | mpv https://www.youtube.com/playlist?list=PLijBjlV15UVTJSqHt_2TtbJB5zCq0UxHm | full | no | yes | yes | - | mpv diff --git a/stacks/app-users.sh b/stacks/app-users.sh new file mode 100644 index 0000000..c314f35 --- /dev/null +++ b/stacks/app-users.sh @@ -0,0 +1,655 @@ +#!/bin/bash +# app-users.sh -- create the application accounts declared in app-users.conf. +# +# Everything here goes through `packagemanager user setup`, which is +# idempotent and already knows how to join the session group, make the home +# unreadable to others, and write a launcher. This step is the part that +# reads a FILE instead of a command line, plus the two things no per-account +# command can do: merge /etc/pam.d/su_u_user (one file listing every +# account) and write the shell variables (one file enumerating them). +# +# NOTHING HERE OVERWRITES A FILE IT DID NOT WRITE. The PAM file is merged +# and backed up; the profile fragment is only written if absent or if it +# carries this script's marker; a shared directory that exists as a real +# directory is left alone. +set -u + +CONF="${APP_USERS_CONF:-/etc/pkgusr/stacks/app-users.conf}" +PAM_FILE="${APP_USERS_PAM:-/etc/pam.d/su_u_user}" +PROFILE_D="${APP_USERS_PROFILE_D:-/etc/profile.d}" +PM="${APP_USERS_PM:-packagemanager}" +MARKER="# written by stacks/app-users.sh -- safe to delete, regenerated" +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } + +[ -r "$CONF" ] || { warn "no $CONF to read"; exit 1; } + +# ASK, DO NOT GUESS. `packagemanager env --system` writes these facts to +# /etc/profile.d/pkgusr-system.sh precisely so that three scripts stop +# rediscovering them three different ways. Use them when they are there, +# fall back to asking the tool, and only then to the old file. +MAIN="${PKGUSR_SYS_MAIN_USER:-}" +[ -n "$MAIN" ] || MAIN="$($PM config --show main-user 2>/dev/null | tr -d ' ')" +[ -n "$MAIN" ] || MAIN="$(cat /etc/pkgusr/desktop-user 2>/dev/null || true)" +if [ -z "$MAIN" ]; then + warn "no main user configured -- there is nobody to grant access to" + note " packagemanager config --main-user <your-login>" + exit 1 +fi +PREFIX="${PKGUSR_SYS_USER_PREFIX:-}" +[ -n "$PREFIX" ] || PREFIX="$($PM config --show user-prefix 2>/dev/null | tr -d ' ')" +[ -n "$PREFIX" ] || PREFIX="u_" +note "main user: $MAIN account prefix: $PREFIX" +# ...and keep those facts current for everything else, since this step is +# where the main user is first really needed. +$PM env --system --run >/dev/null 2>&1 \ + && note "refreshed /etc/profile.d/pkgusr-system.sh (PKGUSR_SYS_*)" \ + || true + +# THE HUB: where the links to every shared directory live (1.14.312). Asked +# of the tool, NOT read from $PKGUSR_SYS_SHARED -- this shell inherited that +# at login, and before 1.14.312 it named ~/Shared. /home/Shared +# unless `packagemanager config --shared-hub DIR` says otherwise. +HUB="$($PM env --system 2>/dev/null | sed -n 's/^export PKGUSR_SYS_SHARED="\(.*\)"$/\1/p' | head -n1)" +[ -n "$HUB" ] || HUB="/home/Shared" +note "shared hub: $HUB" + +# CONFIRM THE COMMAND EXISTS BEFORE THE LOOP. Every line would otherwise +# fail the same way, three times, with the parser's error rather than ours. +if ! $PM user setup --help >/dev/null 2>&1; then + warn "'$PM user setup' is not available -- nothing here can run" + note " check: $PM user --help" + exit 1 +fi +echo + +# ---- 1. the accounts, then their launchers --------------------------------- +# ONE ACCOUNT, SEVERAL LAUNCHERS. A line is +# account | name | app | share | dir | desktop | launcher | groups | icon +# so browser-mini, browser-big, google and duckduckgo can all run as +# u_browser. `name` empty means the account's own name. The account gets +# the strongest `share` of its lines (full beats slim), a shared directory +# if any line asks, and the union of the groups. The older 7-column format +# (name | app | share | dir | desktop | launcher | groups) still reads: the +# account is the name. Missing trailing fields take the defaults +# (slim, yes, yes, yes, -, auto icon). +trim() { printf '%s' "${1:-}" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//'; } +declare -A A_SHARE A_DIR A_GROUPS +accounts="" +L_ACCT=(); L_NAME=(); L_APP=(); L_DESK=(); L_LAUNCH=(); L_ICON=() +format=new +S_ACCT=(); S_CMD=() +while IFS= read -r raw || [ -n "$raw" ]; do + case "$(trim "$raw")" in ""|\#*) continue ;; esac + # @setup <account|*> <command | URL | /path/script> -- run AS the + # account once (see step 1a). Not split on `|`: a command may pipe. + case "$(trim "$raw")" in + @setup[[:space:]]*) + rest="$(trim "${raw#*@setup}")" + sacct="${rest%%[[:space:]]*}" + scmd="$(trim "${rest#"$sacct"}")" + case "$sacct" in + \*|[a-z0-9_-]*) ;; + *) warn "line '$raw': '$sacct' is not an account -- skipped"; continue ;; + esac + [ -n "$scmd" ] || { warn "line '$raw': @setup without a command -- skipped"; continue; } + S_ACCT+=("$sacct"); S_CMD+=("$scmd") + continue ;; + @*) warn "line '$raw': unknown directive -- skipped"; continue ;; + esac + IFS='|' read -r f1 f2 f3 f4 f5 f6 f7 f8 f9 <<< "$raw" + f1="$(trim "$f1")" + # the header tells the format + case "$f1" in + name) format=old; continue ;; + user|account) format=new; continue ;; + esac + if [ "$format" = old ]; then + acct="$f1"; lname="$f1"; app="$(trim "$f2")"; share="$(trim "$f3")" + dir="$(trim "$f4")"; desktop="$(trim "$f5")"; launcher="$(trim "$f6")" + groups="$(trim "$f7")"; icon="" + else + acct="$f1"; lname="$(trim "$f2")"; app="$(trim "$f3")"; share="$(trim "$f4")" + dir="$(trim "$f5")"; desktop="$(trim "$f6")"; launcher="$(trim "$f7")" + groups="$(trim "$f8")"; icon="$(trim "$f9")" + fi + case "$acct" in *[!a-z0-9_-]*|"") warn "line '$raw': account '$acct' is not a plain name -- skipped"; continue ;; esac + [ -n "$lname" ] && [ "$lname" != - ] || lname="$acct" + case "$lname" in *[!a-zA-Z0-9_.-]*) warn "$acct: launcher name '$lname' is not a plain name -- skipped"; continue ;; esac + share="${share:-slim}"; dir="${dir:-yes}"; desktop="${desktop:-yes}"; launcher="${launcher:-yes}" + case "$share" in slim|full|no|-) ;; *) warn "$lname: share='$share' is not slim, full or no -- skipped"; continue ;; esac + # the account's settings: the strongest share, any dir, all groups + case "$share:${A_SHARE[$acct]:-}" in + full:*) A_SHARE[$acct]=full ;; + slim:full) ;; + slim:*) A_SHARE[$acct]=slim ;; + *) A_SHARE[$acct]="${A_SHARE[$acct]:-no}" ;; + esac + [ "$dir" = yes ] && A_DIR[$acct]=yes + case "$groups" in -|"") ;; *) A_GROUPS[$acct]="${A_GROUPS[$acct]:+${A_GROUPS[$acct]},}$groups" ;; esac + case " $accounts " in *" $acct "*) ;; *) accounts="$accounts $acct" ;; esac + case "$app" in -|"") continue ;; esac + L_ACCT+=("$acct"); L_NAME+=("$lname"); L_APP+=("$app") + L_DESK+=("$desktop"); L_LAUNCH+=("$launcher"); L_ICON+=("$icon") +done < "$CONF" + +names="" +for name in $accounts; do + # CREATE IT IF IT IS NOT THERE: `user setup` needs an existing account, + # `user create` makes one and then applies the same setup. + if id "$PREFIX$name" >/dev/null 2>&1; then + set -- user setup "$name" + else + note "$PREFIX$name does not exist yet -- creating it" + set -- user create "$name" + fi + case "${A_SHARE[$name]}" in + slim) set -- "$@" --shared-slim ;; + full) set -- "$@" --shared-full ;; + esac + [ "${A_DIR[$name]:-}" = yes ] && set -- "$@" --share-dir + [ -n "${A_GROUPS[$name]:-}" ] && set -- "$@" --groups "${A_GROUPS[$name]}" + note "$name: $PM $*" + if $PM "$@"; then + names="$names $name" + else + warn "$name: setup failed -- not added to the PAM list, no launchers" + fi + echo +done + +for i in "${!L_NAME[@]}"; do + acct="${L_ACCT[$i]}" + case " $names " in *" $acct "*) ;; *) continue ;; esac + # a menu entry runs the launcher, so either one needs the launcher + [ "${L_LAUNCH[$i]}" = yes ] || [ "${L_DESK[$i]}" = yes ] || continue + set -- user launcher "$acct" "${L_NAME[$i]}" --app "${L_APP[$i]}" + [ "${L_DESK[$i]}" = yes ] || set -- "$@" --no-desktop + [ -n "${L_ICON[$i]}" ] && set -- "$@" --icon "${L_ICON[$i]}" + note "${L_NAME[$i]} (as $PREFIX$acct)" + $PM "$@" || warn "${L_NAME[$i]}: launcher failed" +done +echo + +[ -n "$names" ] || { warn "no usable lines in $CONF"; exit 1; } + +# ---- 1a. @setup: the account's own extra steps ------------------------------ +# A command, a local script or a URL to a bash script, run AS the account +# (login shell, its own HOME) -- never as root. ONCE: a step that ran is +# recorded by its text in $SETUP_DB/<account>, so a later run skips it and +# a CHANGED line runs again. APP_USERS_RERUN=1 runs them all again. +SETUP_DB="${APP_USERS_SETUP_DB:-/var/lib/pkgusr/app-setup}" +fetch_to() { # url file + if command -v curl >/dev/null 2>&1; then curl -fsSL "$1" -o "$2" + elif command -v wget >/dev/null 2>&1; then wget -q -O "$2" "$1" + else echo "no curl or wget" >&2; return 1; fi +} +if [ "${#S_CMD[@]}" -gt 0 ]; then + mkdir -p "$SETUP_DB" && chmod 700 "$SETUP_DB" + for i in "${!S_CMD[@]}"; do + cmd="${S_CMD[$i]}" + if [ "${S_ACCT[$i]}" = "*" ]; then targets="$names"; else targets="${S_ACCT[$i]}"; fi + key="$(printf '%s' "$cmd" | md5sum | cut -c1-12)" + for acct in $targets; do + case " $names " in *" $acct "*) ;; *) warn "@setup $acct: no such account in $CONF -- skipped"; continue ;; esac + db="$SETUP_DB/$acct" + if [ "${APP_USERS_RERUN:-0}" != 1 ] && grep -qx "$key" "$db" 2>/dev/null; then + note "$acct: setup already done -- $cmd" + continue + fi + note "$acct: setup -- $cmd" + case "$cmd" in + http://*|https://*) + tmp="$(mktemp /tmp/app-setup.XXXXXX)" + if ! fetch_to "$cmd" "$tmp"; then + rm -f "$tmp"; warn "$acct: could not download $cmd"; continue + fi + chmod 644 "$tmp" + su - "$PREFIX$acct" -s /bin/bash -c "bash '$tmp'"; rc=$? + rm -f "$tmp" ;; + /*) + if [ -f "$cmd" ]; then + su - "$PREFIX$acct" -s /bin/bash -c "bash '$cmd'"; rc=$? + else + su - "$PREFIX$acct" -s /bin/bash -c "$cmd"; rc=$? + fi ;; + *) su - "$PREFIX$acct" -s /bin/bash -c "$cmd"; rc=$? ;; + esac + if [ "$rc" = 0 ]; then + grep -qx "$key" "$db" 2>/dev/null || echo "$key" >> "$db" + else + warn "$acct: setup failed (exit $rc) -- runs again next time: $cmd" + fi + done + done + echo +fi +# TRIM ONCE, HERE. $names is built by appending " $n", so it starts with a +# space and ${names%% *} is empty -- which printed `su - u_` twice before, +# in two different blocks. One trim, at the top, and every later use is +# right. +names="$(printf '%s' "$names" | sed 's/^ *//')" + +# ---- 1b. a line that STOPPED asking for a shared directory ----------------- +# `user setup` makes what the line asks for; it never takes away, so an +# account switched from dir=yes to dir=no keeps its /home/Shared/<Name> -- +# and /etc/profile.d/shared-dirs.sh keeps exporting $SHARED_<NAME> for it. +# An EMPTY leftover is removed here (nothing can be lost). One with files in +# it is yours: this says where it is and what would remove it. +MAIN_HOME="$(getent passwd "$MAIN" | cut -d: -f6)" +for name in $accounts; do + [ "${A_DIR[$name]:-}" = yes ] && continue + cap="$(printf '%s' "$name" | cut -c1 | tr '[:lower:]' '[:upper:]')$(printf '%s' "$name" | cut -c2-)" + d="$HUB/$cap" + # since 1.14.315 the hub holds the DIRECTORY and the account's ~/Shared + # is the link; before that, the hub held a link. Either is handled. + ah="$(getent passwd "$PREFIX$name" | cut -d: -f6)" + if [ -L "$d" ]; then + tgt="$(readlink -f "$d" 2>/dev/null || true)" + elif [ -d "$d" ]; then + tgt="$d" + else + continue + fi + if [ ! -d "$tgt" ] || [ -z "$(ls -A "$tgt" 2>/dev/null)" ]; then + [ -L "$d" ] && rm -f "$d" + [ -d "$tgt" ] && rmdir "$tgt" 2>/dev/null + note "$name: dir=no -- removed the empty $d" + if [ -n "$ah" ] && [ -L "$ah/Shared" ] && [ ! -e "$ah/Shared" ]; then + rm -f "$ah/Shared" && note " and the link $ah/Shared" + fi + else + note "$name: dir=no, but $d still has files in it -- left alone" + note " look, then remove it yourself" + fi +done +echo + +# ---- 2. /etc/pam.d/su_u_user, MERGED -------------------------------------- +# One file lists every account your login may become without a password. +# It is hand-editable and may well have entries this file knows nothing +# about, so: keep what is there, add what is missing, back it up first. +pam_has() { grep -qE "^[[:space:]]*$1\)" "$PAM_FILE" 2>/dev/null; } + +missing="" +for n in $names; do + pam_has "$PREFIX$n" || missing="$missing $n" +done + +if [ ! -e "$PAM_FILE" ]; then + note "writing $PAM_FILE (it did not exist)" + mkdir -p "$(dirname "$PAM_FILE")" + { + echo '#!/bin/bash' + echo "$MARKER" + echo '# Which accounts may be entered with `su -` without a password.' + echo '# Hand-edited entries are kept: this file is MERGED, never' + echo '# overwritten.' + echo '' + echo '# PAM_USER is the account being entered, PAM_RUSER the one asking.' + echo '# No caller, no pass: su always sets it.' + echo '[[ -n $PAM_RUSER ]] || exit 1' + echo '[[ "$PAM_TYPE" == "auth" ]] || exit 1' + echo 'case "$PAM_USER" in' + for n in $names; do + printf ' %s%s) USERS="%s%s %s"; ;;\n' \ + "$PREFIX" "$n" "$PREFIX" "$n" "$MAIN" + done + echo ' *) exit 1 ; ;;' + echo 'esac' + echo '[[ " $USERS " == *" $PAM_RUSER "* ]]' + } > "$PAM_FILE" + # IT IS A SCRIPT, NOT A CONFIG FILE. /etc/pam.d/su_u_user starts + # with `#!/bin/bash` and is RUN by pam_exec.so. At 644 pam cannot + # execute it, and `su - u_claude` then just asks for a password + # with nothing in the logs to say why. + chmod 755 "$PAM_FILE" + note " $(grep -c ') USERS=' "$PAM_FILE") account(s) listed" +elif [ -z "$missing" ]; then + note "$PAM_FILE already lists every account here" +else + # BACK IT UP BEFORE TOUCHING IT, with a name that says when. + bak="$PAM_FILE.bak-$(date +%Y%m%d-%H%M%S)" + cp -p "$PAM_FILE" "$bak" || { warn "could not back up $PAM_FILE"; exit 1; } + note "backed up $PAM_FILE -> $(basename "$bak")" + # insert before the `*)` default, which is where new cases belong; if + # there is no default we cannot know the shape and will not guess. + if grep -qE '^[[:space:]]*\*\)' "$PAM_FILE"; then + tmp="$(mktemp)" + ins="$tmp.ins" + : > "$ins" + for n in $missing; do + printf ' %s%s) USERS="%s%s %s"; ;;\n' \ + "$PREFIX" "$n" "$PREFIX" "$n" "$MAIN" >> "$ins" + done + awk -v ins="$ins" ' + /^[[:space:]]*\*\)/ && !done { + while ((getline line < ins) > 0) print line + close(ins); done = 1 + } + { print } + ' "$PAM_FILE" > "$tmp" && cat "$tmp" > "$PAM_FILE" + rm -f "$tmp" "$ins" + for n in $missing; do + pam_has "$PREFIX$n" \ + || warn "$PREFIX$n is still not in $PAM_FILE after the merge" + done + # the merge rewrites the file, so re-assert the exec bit: pam + # cannot run a 644 script, and a merge that silently drops it + # breaks su for every account in there. + chmod 755 "$PAM_FILE" + note "added to $PAM_FILE:$missing" + else + warn "$PAM_FILE has no '*)' default case -- not editing it" + note " add these lines yourself, before the end of the case:" + for n in $missing; do + printf ' %s%s) USERS="%s%s %s"; ;;\n' \ + "$PREFIX" "$n" "$PREFIX" "$n" "$MAIN" + done + fi +fi + +# THE CHECK MUST BE ON THE CALLER. +# +# The last line compared the TARGET with a list that contains the target: +# [[ "$USERS" =~ "$PAM_USER" ]] +# which is always true, and an empty PAM_RUSER was filled with PAM_USER. So +# pam_exec let ANY account -- another u_*, any package user -- enter every +# listed account without a password. The intended test is "is the account +# ASKING in the list", as a whole word (=~ also matched u_claude inside +# u_claudex). A file written before this is repaired in place, with a backup. +if [ -f "$PAM_FILE" ] && grep -qE '^\[\[ "\$USERS" =~ "\$PAM_USER" \]\]|^\[\[ -z \$PAM_RUSER \]\] && PAM_RUSER=\$PAM_USER' "$PAM_FILE"; then + bak="$PAM_FILE.bak-$(date +%Y%m%d-%H%M%S)" + cp -p "$PAM_FILE" "$bak" || { warn "could not back up $PAM_FILE"; exit 1; } + sed -i \ + -e 's/^\[\[ -z \$PAM_RUSER \]\] && PAM_RUSER=\$PAM_USER$/[[ -n $PAM_RUSER ]] || exit 1/' \ + -e 's/^\[\[ "\$USERS" =~ "\$PAM_USER" \]\]$/[[ " $USERS " == *" $PAM_RUSER "* ]]/' \ + "$PAM_FILE" + if grep -qF '[[ " $USERS " == *" $PAM_RUSER "* ]]' "$PAM_FILE" \ + && ! grep -qF '=~ "$PAM_USER"' "$PAM_FILE"; then + note "fixed $PAM_FILE: it let ANY account in without a password;" + note " it now checks the account asking (backup: $(basename "$bak"))" + else + warn "could not repair the caller check in $PAM_FILE -- restoring the backup" + cp -p "$bak" "$PAM_FILE" + note " its last line must read: [[ \" \$USERS \" == *\" \$PAM_RUSER \"* ]]" + fi +fi + +# THE ACCOUNTS THEMSELVES: LOCKED, AND NOT DUE A PASSWORD CHANGE. +# +# su: Authentication token manipulation error +# +# That is pam asking for a NEW password: /etc/shadow's "last changed" field +# is 0, which means "must change at next login", and su -- run by a user -- +# cannot change another account's password. pam_exec lets the AUTH step +# through, but the ACCOUNT step (pam_unix in system-account) still enforces +# it. And an EMPTY password field would let anyone in with Enter. Each +# account is locked (no password at all; only su from a listed account enters +# it) and its last-change date set when it is 0. Nothing else is touched. +if [ -r /etc/shadow ]; then + for n in $names; do + u="$PREFIX$n" + line="$(grep "^$u:" /etc/shadow 2>/dev/null)" || continue + pwf="$(printf '%s' "$line" | cut -d: -f2)" + lastchg="$(printf '%s' "$line" | cut -d: -f3)" + if [ -z "$pwf" ]; then + usermod -L "$u" >/dev/null 2>&1 && note "$u had an EMPTY password -- locked" + grep -q "^$u:!" /etc/shadow || warn "$u still has an empty password field" + fi + if [ "$lastchg" = 0 ]; then + chage -d "$(date +%Y-%m-%d)" "$u" \ + && note "$u was marked 'must change password' -- cleared (su would fail)" + fi + done +fi + +# ROOT RUNS IT, SO ROOT OWNS IT. pam_exec executes this as root; it was +# owned by p_librice (written while librice's build window was open, and +# attributed to it), so anything building as p_librice could replace a script +# root runs on every su. Re-asserted on every run. +if [ -f "$PAM_FILE" ]; then + chown root:root "$PAM_FILE" && chmod 755 "$PAM_FILE" + [ "$(stat -c %U:%G:%a "$PAM_FILE")" = root:root:755 ] \ + || warn "$PAM_FILE is $(stat -c %U:%G:%a "$PAM_FILE"), not root:root:755" +fi +echo + +# ---- 3. the shell variables ---------------------------------------------- +# ENUMERATE, DO NOT LIST. The fragment reads the hub at login, so an +# account added to the conf file shows up in the next shell without this +# file being rewritten -- and a link you make by hand appears too. +frag="$PROFILE_D/shared-dirs.sh" +if [ -d "$PROFILE_D" ]; then + if [ -e "$frag" ] && ! grep -qF "$MARKER" "$frag"; then + warn "$frag exists and is not ours -- left alone" + note " it would have exported SHARED_<NAME> for each $HUB/<Name>" + else + cat > "$frag" <<PROF +$MARKER +# One variable per shared directory: $HUB/Claude -> \$SHARED_CLAUDE. +# The SAME set in every account: the hub is a system path, not \$HOME/Shared +# (which in an app account is its own data directory -- enumerating that +# exported one SHARED_* per FILE in it, before 1.14.312). +# Enumerated, not listed, so an account added or removed needs no change +# here -- and a link you make by hand appears too. +# +# IT ALSO TAKES AWAY. An account removed after you logged in leaves its +# variable in the environment, pointing at nothing, and every shell started +# from that session inherits it. So: drop the ones whose directory is gone, +# then export what is really there. Safe to source again at any time. +for _v in \$(env | sed -n 's/^\(SHARED_[A-Z0-9_]*\)=.*/\1/p'); do + eval "_p=\\\${\$_v}" + [ -d "\$_p" ] || unset "\$_v" +done +if [ -d "$HUB" ]; then + for _sd in "$HUB"/*; do + [ -e "\$_sd" ] || continue + _n="\$(basename "\$_sd" | tr '[:lower:]-' '[:upper:]_')" + export "SHARED_\$_n=\$_sd" + done +fi +unset _sd _n _v _p +PROF + chmod 644 "$frag"; chown root:root "$frag" 2>/dev/null + note "wrote $frag" + note " in a new login shell: echo \$SHARED_CLAUDE" + note " (login shells read /etc/profile, which reads this directory." + note " Nothing of yours was edited -- no .bashrc, no .bash_profile.)" + fi + # ...AND IN A TERMINAL THAT IS NOT A LOGIN SHELL. + # + # /etc/profile.d is read by LOGIN shells. A terminal opened inside sway + # is not one: it reads ~/.bashrc, so it shows whatever the session + # inherited when you logged in -- which is why the variables looked + # "not set up" while the file was right. /etc/bashrc is the file every + # interactive shell reads (shell-env.sh writes it, ~/.bashrc sources + # it), so the fragment is sourced from there too. Only if that file is + # the one shell-env.sh wrote; anything else is yours and only gets the + # line printed. + # WHICH /etc/bashrc COUNTS AS THE SYSTEM'S. Looking only for the line + # shell-env.sh writes was too narrow: a machine that took the BLFS + # "Bash Shell Startup Files" file instead has a perfectly ordinary + # /etc/bashrc, and the step just printed the line and left the variables + # unreachable. Both start `# Begin /etc/bashrc`, which is the marker. + # ONE line is APPENDED -- nothing is rewritten -- after a backup, and + # starting with a newline: a file here ends in `# End /etc/bashrcCK=$(tput + # setaf 0)`, which is what an append without one does. + BASHRC="${APP_USERS_BASHRC:-/etc/bashrc}" + SRC_LINE='[ -r '"$frag"' ] && . '"$frag" + if [ ! -f "$BASHRC" ]; then + note "no $BASHRC -- add this line to ~/.bashrc so a NEW TERMINAL sees them:" + note " $SRC_LINE" + elif grep -qF "$frag" "$BASHRC"; then + note "$BASHRC already sources $(basename "$frag")" + elif head -n1 "$BASHRC" | grep -q '^# Begin /etc/bashrc' \ + || grep -q 'Based on the BLFS "Bash Shell Startup Files" page' "$BASHRC"; then + bak="$BASHRC.bak-$(date +%Y%m%d-%H%M%S)" + if cp -p "$BASHRC" "$bak"; then + printf '\n# the shared application directories (SHARED_<NAME>), for shells\n# that are not login shells -- a terminal opened inside the session\n%s\n' \ + "$SRC_LINE" >> "$BASHRC" + note "added one line to $BASHRC (backup: $(basename "$bak"))" + note " a NEW TERMINAL now has the variables, without logging in again" + else + warn "could not back up $BASHRC -- not touching it" + note " add by hand: $SRC_LINE" + fi + else + note "$BASHRC is not the system's -- add this line to it, or to ~/.bashrc," + note " so a NEW TERMINAL sees the variables without logging in again:" + note " $SRC_LINE" + fi +else + warn "no $PROFILE_D -- cannot install the SHARED_* variables" + note " source this from your own shell instead, and it stays yours:" + note " for d in $HUB/*; do export \"SHARED_\$(basename \$d | tr a-z- A-Z_)=\$d\"; done" +fi + +# ---- 3b. the session group, applied by the session ------------------------- +# Shared accounts reach the main user's display through the XDG runtime +# directory, which needs the group <prefix>_xdg-runtime (created and joined +# by `user setup`). The directory is made fresh at every boot (and chmod-ed +# 0700 by whatever creates it), so root cannot set its group once and for +# all: the main user's own session has to. Two pieces, both idempotent: +# /usr/bin/xdg-runtime-share run in the session (sway: exec it) -- +# the directory, then the wayland and +# pipewire sockets as they appear +# /etc/profile.d/xdg-runtime-share.sh the directory, at every login shell +pfx="${LFS_COLLECTOR_PREFIX:-}" +[ -n "$pfx" ] || pfx="$(sed -n 's/^collector_prefix=\(.*\)$/\1/p' /etc/pkgusr/packagemanager.conf 2>/dev/null | head -n1)" +if [ -z "$pfx" ]; then + warn "no collector prefix -- the session group cannot be named" +else + XDG_GROUP="${pfx%_}_xdg-runtime" + cat > /usr/bin/xdg-runtime-share <<SHARE +#!/bin/bash +$MARKER +# xdg-runtime-share -- open THIS session to the shared application accounts. +# Run it as $MAIN inside the session; in ~/.config/sway/config: +# exec xdg-runtime-share +# Group $XDG_GROUP on \$XDG_RUNTIME_DIR (0750: enter, not list-and-write), +# then group read/write on the sockets a shared program connects to. The +# session bus is NOT shared: dbus refuses other uids anyway. +g=$XDG_GROUP +d="\${XDG_RUNTIME_DIR:-}" +if [ -z "\$d" ] || [ ! -d "\$d" ] || [ ! -O "\$d" ]; then + echo "xdg-runtime-share: XDG_RUNTIME_DIR is not a directory of yours" >&2; exit 1 +fi +if ! id -Gn | tr ' ' '\n' | grep -qx "\$g"; then + echo "xdg-runtime-share: you are not in \$g yet -- log out and in again" >&2; exit 1 +fi +chgrp "\$g" "\$d" && chmod 0750 "\$d" || exit 1 +# the sockets appear as the compositor and pipewire start: look for a while +for _i in \$(seq 1 "\${1:-30}"); do + for s in "\$d"/wayland-[0-9]* "\$d"/pipewire-0 "\$d"/pipewire-0-manager; do + case "\$s" in *.lock) continue ;; esac + [ -S "\$s" ] && [ -O "\$s" ] || continue + chgrp "\$g" "\$s" && chmod g+rw "\$s" + done + sleep 1 +done +SHARE + chmod 755 /usr/bin/xdg-runtime-share + chown root:root /usr/bin/xdg-runtime-share 2>/dev/null + cat > "$PROFILE_D/xdg-runtime-share.sh" <<PROF +$MARKER +# The runtime directory of $MAIN is entered by the shared accounts through +# group $XDG_GROUP. Applied at every login shell; the sockets are done by +# xdg-runtime-share inside the session. +if [ "\$(id -un)" = "$MAIN" ] && [ -n "\${XDG_RUNTIME_DIR:-}" ] && [ -O "\$XDG_RUNTIME_DIR" ]; then + chgrp $XDG_GROUP "\$XDG_RUNTIME_DIR" 2>/dev/null && chmod 0750 "\$XDG_RUNTIME_DIR" 2>/dev/null +fi +PROF + chmod 644 "$PROFILE_D/xdg-runtime-share.sh" + chown root:root "$PROFILE_D/xdg-runtime-share.sh" 2>/dev/null + note "session group: $XDG_GROUP" + note " wrote /usr/bin/xdg-runtime-share and $PROFILE_D/xdg-runtime-share.sh" + _mh="$(getent passwd "$MAIN" | cut -d: -f6)" + if [ -f "$_mh/.config/sway/config" ] \ + && ! grep -q 'xdg-runtime-share' "$_mh/.config/sway/config"; then + note " add to $_mh/.config/sway/config (not edited -- it is yours):" + note " exec xdg-runtime-share" + fi + note " $MAIN must log in again once to be in $XDG_GROUP" +fi +echo + +# ---- 2b. tell pam to run it ----------------------------------------------- +# The script decides WHO may be entered without a password, and pam only +# consults it if /etc/pam.d/su says so. That line is added here, with a +# backup -- but only when there is a PAM to add it to. +PAM_SU="${APP_USERS_PAM_SU:-/etc/pam.d/su}" +PAM_LINE="auth sufficient pam_exec.so quiet $PAM_FILE" + +if [ ! -d "$(dirname "$PAM_SU")" ]; then + # NO PAM ON THIS MACHINE, AND THAT IS A DECISION, NOT AN OVERSIGHT. + # avoid.stack avoids linux-pam (it arrived only as systemd's dependency), + # so there is no /etc/pam.d at all -- and LFS's shadow is built without + # PAM. Adding a line to a file pam never reads would look like a fix + # and change nothing. + warn "no $(dirname "$PAM_SU") -- Linux-PAM is not installed here" + note " so `su - ${PREFIX}<name>` will keep asking for a password." + note " The accounts, groups, shared directories and launchers above are" + note " all real and working; only PASSWORDLESS su needs pam." + note " To get it, deliberately:" + note " packagemanager install linux-pam --run --recursive" + note " packagemanager update shadow --reinstall --regenerate --run" + note " (shadow must be REBUILT against pam, or nothing reads" + note " /etc/pam.d at all -- and read the book's Shadow page first:" + note " a wrong pam setup can lock you out of su and login.)" + note " $PAM_FILE is written and ready for when pam is there." +elif [ ! -f "$PAM_SU" ]; then + warn "$PAM_SU does not exist -- not creating it" + note " /etc/pam.d exists, so pam is here, but su has no pam file: that" + note " comes from the book's Shadow page and this is not the place to" + note " invent one. The line it needs:" + note " $PAM_LINE" +elif grep -qE "^[[:space:]]*auth[[:space:]]+sufficient[[:space:]]+pam_exec\.so.*$(basename "$PAM_FILE")" "$PAM_SU"; then + # an ACTIVE auth line -- a comment or a backup note naming the file is + # not pam calling it + note "$PAM_SU already calls $(basename "$PAM_FILE")" +else + bak="$PAM_SU.bak-$(date +%Y%m%d-%H%M%S)" + if cp -p "$PAM_SU" "$bak"; then + note "backed up $PAM_SU -> $(basename "$bak")" + # ABOVE THE OTHER auth LINES. `sufficient` succeeds the stack + # immediately, so it has to be reached before the password prompt; + # after them it would never run for the accounts it exists for. + tmp="$(mktemp)" + awk -v line="$PAM_LINE" ' + !done && /^[[:space:]]*auth[[:space:]]/ { print line; done = 1 } + { print } + END { if (!done) print line } + ' "$PAM_SU" > "$tmp" && cat "$tmp" > "$PAM_SU" + rm -f "$tmp" + if grep -qF "$(basename "$PAM_FILE")" "$PAM_SU"; then + note "added to $PAM_SU:" + note " $PAM_LINE" + note " test it now, in another terminal, BEFORE logging out:" + note " su - ${PREFIX}${names%% *} -c id" + note " if su is broken, put the backup back: cp -p $bak $PAM_SU" + else + warn "could not add the line to $PAM_SU -- restoring the backup" + cp -p "$bak" "$PAM_SU" + note " add it by hand, above the other auth lines:" + note " $PAM_LINE" + fi + else + warn "could not back up $PAM_SU -- not touching it" + note " add by hand, above the other auth lines: $PAM_LINE" + fi +fi + +echo +if [ "$fail" = 0 ]; then + # $names is built by appending " $n", so it starts with a space and + # ${names%% *} was empty -- the hint read `su - u_`. + set -- $names + echo "# app-users: done. accounts:$names" + echo "# what they can reach: packagemanager user list" + echo "# try one: su - ${PREFIX}${1} # no password" +else + echo "# the !! lines above need attention; everything else was applied." +fi +exit $fail diff --git a/stacks/apps.stack b/stacks/apps.stack new file mode 100644 index 0000000..3c74a12 --- /dev/null +++ b/stacks/apps.stack @@ -0,0 +1,94 @@ +# apps.stack -- desktop applications: a document viewer and an image viewer. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +include avoid + +# packagemanager stack apps --run +# +# Programs you open, not the desktop they open on. Run it AFTER sway.stack: +# they need GTK and wayland from it, and nothing there needs them. They +# were the tail of sway.stack until 1.14.320. + +# VIEWERS: a document one and an image one. Neither is in the BLFS book. +# +# CHECKED with git ls-remote, all four at once: girara, zathura and +# zathura-pdf-poppler share the release tag **2026.07.18** (they are one +# project, released together -- mixing tags is the usual way to get a +# zathura that cannot open a PDF), and imv's newest is **v4.3.0**. +# imv's repo answers on both github.com/exec64/imv and .../eXeC64/imv; the +# lowercase spelling is used here. +# +# zathura is THREE packages: girara is its widget library, the core opens no +# file formats at all, and the backend is a separate plugin. A zathura with +# no plugin starts, shows a grey window and says nothing -- which is what +# the expect= below is for: it looks for the PLUGIN, not the binary. +# +# poppler (the backend's dependency) is a book page. It was in avoid.stack +# as libcupsfilters' child; it is wanted now. mupdf stays avoided -- it is +# the other backend and one is enough. +book poppler + +# girara installs **girara.pc**, not girara-gtk3.pc: its meson.build says +# `pkg.generate(name: 'girara', ...)`. The old guess made the stack refuse +# to record a girara that had installed perfectly -- "finished, but +# /usr/lib/pkgconfig/girara-gtk3.pc is still missing", which is the have= +# check doing its job on a wrong have=. +git girara url=https://github.com/pwmt/girara.git ref=2026.07.18 \ + have=/usr/lib/pkgconfig/girara.pc \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" + +# xxHash, BEFORE zathura. Mentioned eight times in the BLFS book and given +# no page of its own, so this entry is ours. +# +# zathura 2026.07.18 takes it as a hard dependency: +# Run-time dependency libxxhash found: NO (tried pkgconfig and cmake) +# meson.build:57:9: ERROR: Dependency "libxxhash" not found +# +# CHECKED in the checkout: newest tag v0.8.3; it ships libxxhash.pc.in, so +# the pkg-config module really is `libxxhash` -- the name meson asks for -- +# and `make install` runs install_libxxhash.pc among its targets. A plain +# Makefile, no cmake or meson: PREFIX and LIBDIR are what it takes. +git xxhash url=https://github.com/Cyan4973/xxHash.git ref=v0.8.3 \ + have=/usr/lib/pkgconfig/libxxhash.pc \ + build="make PREFIX=/usr LIBDIR=/usr/lib" \ + install="make PREFIX=/usr LIBDIR=/usr/lib install" \ + expect="pkg-config --exists libxxhash" + +# NO -Dsqlite: zathura has no such option. Its meson.build takes sqlite as +# an unconditional dependency -- `dependency('sqlite3', version: '>=3.35.0')` +# -- and its only options are synctex, seccomp, landlock, manpages, +# tests-x11, tests-wayland and convert-icon. `-Dsqlite=enabled` stopped +# meson dead with `Unknown option: "sqlite"`, which is what guessing an +# option name earns. sqlite3 is already installed here (p_sqlite). +git zathura url=https://github.com/pwmt/zathura.git ref=2026.07.18 \ + prog=zathura \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" + +git zathura-pdf-poppler url=https://github.com/pwmt/zathura-pdf-poppler.git \ + ref=2026.07.18 \ + have=/usr/lib/zathura/libpdf-poppler.so \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" \ + expect="test -e /usr/lib/zathura/libpdf-poppler.so" + +# imv: wayland image viewer. The binary is `imv`, which dispatches to +# imv-wayland or imv-x11 -- so the probe wants the wayland one, or on this +# machine it would pass on an imv that can only open windows under X. +# -licuuc: imv links libicuio and CALLS libicuuc (ubrk_close), and its +# link line carries -Wl,--no-undefined, so the indirect dependency is an +# error rather than a warning: +# undefined reference to symbol 'ubrk_close_78' +# /lib/libicuuc.so.78: error adding symbols: DSO missing from command line +# icu splits its API across libicuuc (common), libicui18n and libicuio; +# imv's meson names only the one it uses directly. Naming the other on the +# link line is the whole fix -- nothing is patched. +git imv url=https://github.com/exec64/imv.git ref=v4.3.0 \ + prog=imv \ + build="meson setup build --prefix=/usr --buildtype=release -D c_link_args=-licuuc; ninja -C build" \ + install="ninja -C build install" \ + expect="command -v imv-wayland >/dev/null 2>&1 || ldd /usr/bin/imv 2>/dev/null | grep -q wayland" + diff --git a/stacks/avoid.stack b/stacks/avoid.stack new file mode 100644 index 0000000..24ef08f --- /dev/null +++ b/stacks/avoid.stack @@ -0,0 +1,170 @@ +# avoid.stack -- what no stack on this machine builds, and why. +# stack-version: 1.14.320 +# +# An `avoid` entry does not build anything: it tells a dependency plan that +# this machine will not have that package, so the plan stops walking into it +# and its own dependencies never appear either. 87 entries here keep a +# desktop build from dragging in qt6, cups, texlive, openjdk, systemd and +# the rest of two other desktop environments. +# +# WHY A FILE OF ITS OWN. These lived in sway.stack, and when the media +# chain moved into media.stack it lost all of them -- so +# `packagemanager stack media --status` planned 274 packages including +# systemd, polkit, linux-pam, qt6 and cups, on a SysV machine that must +# never have systemd. The exclusions are a property of the MACHINE, not of +# one stack, so every stack includes this file: +# +# include avoid +# +# Nothing here is ever built, so including it costs nothing and forgetting +# it costs an afternoon. +# +# DUPLICATES REMOVED when this was split out: libkexiv2, libfm-qt, +# xdg-desktop-portal-lxqt and xdg-desktop-portal-gnome were each listed +# twice, which is why `stack sway` used to print some names twice in its +# "never built by this stack" line. Harmless, and it made the composed plan +# impossible to check against, so now one entry per package. + +# Never built by this stack, however they are reached. harfbuzz RECOMMENDS +# texlive and LibreOffice; a sway desktop with a webkit browser wants +# neither, and texlive alone is hours of build time and a /opt tree. +# NOT BUILT BY THIS STACK. The book's recommendations are followed -- +# a recommendation the build actually needs (curl for cmake, icu for +# libxml2) must not be dropped by a blanket rule. These are the branches +# this desktop does not want, named one at a time, and everything they alone +# pull in goes with them. +avoid libreoffice +avoid asymptote +avoid biber +avoid qt6 +avoid okular +avoid phonon +avoid phonon-backend-vlc +avoid vlc +avoid cups +avoid openjdk +avoid java +avoid apache-ant +avoid postgresql +avoid openldap +avoid unixodbc +avoid graphviz +avoid gegl + +# the KDE leaves okular pulls in: named individually, because a package can +# have several parents and pruning by one of them drops shared dependencies +# (graphite2 was lost that way) +avoid plasma-activities +avoid libkexiv2 +avoid kwindowsystem +avoid libfm-qt +avoid xdg-desktop-portal-lxqt +avoid xdg-desktop-portal-gnome +avoid gnome-desktop + +# The desktop-portal tail. The book makes xdg-desktop-portal REQUIRE all +# four backends -- gtk, wlr, gnome AND lxqt -- so one recommendation from +# gtk4 drags in the whole of xfce and lxqt, gnome-desktop and libfm-qt. +# Sway needs none of it to run; screen sharing wants xdg-desktop-portal-wlr, +# which can be built on its own later: +# packagemanager install xdg-desktop-portal-wlr --run +avoid xdg-desktop-portal +avoid xdg-desktop-portal-gtk +avoid xdg-desktop-portal-wlr +avoid xfce4-panel +avoid xfce4-notifyd +avoid xfce4-dev-tools +avoid xfconf +avoid libxfce4util +avoid libxfce4ui +avoid libxfce4windowing +avoid exo +avoid garcon +avoid liblxqt +avoid lxqt-build-tools +avoid lxqt-notificationd +avoid lxqt-kwindowsystem +avoid libqtxdg +avoid polkit-qt +avoid menu-cache +avoid libwnck +avoid notification-daemon + +# Not this desktop either: a systemd package on a SysV system, phone modem +# stacks, a mail/ssh server, spell-checking dictionaries. +avoid systemd +avoid ModemManager +avoid libmbim +avoid libqmi +avoid geoclue2 + +# openssh is AVOIDED here and INSTALLED by network.stack, where it belongs: +# it is networking, not desktop. Letting a dependency plan pull it in +# mid-sway would put its system-account question in the middle of a graphics +# build. +# openssh WAS avoided here. network.stack BUILDS it (with the install-sshd +# bootscript), so the two files contradicted each other: an `avoid` entry +# and a `book` entry for one package, and whichever a plan saw first won. +# Removed in 1.14.241 -- it is wanted. +avoid ruby +avoid aspell +avoid enchant +avoid gnome-keyring +avoid gcr +avoid startup-notification + +# Tails of things already avoided. Each of these is in the plan ONLY to +# serve a package this stack does not build, and the tools cannot yet work +# that out for themselves (see 1.12.83: pruning by the single parent that +# `blfs order` reports dropped shared dependencies, and broke harfbuzz). +# Checked one by one against the plan's parent column: +avoid libcupsfilters # cups (avoided) +avoid libppd # cups +avoid cups-filters # cups +avoid exiv2 # libkexiv2 +avoid clucene # libreoffice (avoided) +avoid redland # libreoffice +avoid rasqal # redland +avoid raptor # rasqal +avoid jasper # qt6 (avoided) +avoid libmng # qt6 +avoid double-conversion # qt6 +# poppler WAS avoided here (libcupsfilters was its only parent, and that is +# avoided). It moved to sway.stack in 1.14.241: zathura's PDF backend needs +# it, so it is now something this machine WANTS rather than something it is +# dragged into. mupdf below stays avoided -- it is the OTHER backend and +# one is enough. +avoid mupdf # libcupsfilters +avoid gs # libcupsfilters +avoid qpdf # libcupsfilters +avoid nss # poppler +avoid boost # poppler +avoid pdfio # libcupsfilters +avoid gpgmepp # poppler + +# Checked against EVERY parent, not just the one the plan prints: these have +# no parent left in the plan at all. +avoid colord # cups (avoided) -- no live parent +# linux-pam WAS avoided here: it only ever arrived as systemd's dependency, +# and systemd is avoided. It moved to services.stack in 1.14.235 because +# passwordless `su - u_<app>` needs pam_exec, which needs pam -- so it is +# now something this machine WANTS rather than something it is dragged into. +# (Left as a comment rather than deleted: the reason it was here is the +# reason it must not come back by accident.) +avoid xdg-utils # cups -- no live parent +avoid freeglut # mupdf (avoided) -- no live parent +avoid polkit # colord + systemd (both avoided) -- no live parent +avoid libgusb # colord + + +# TeX is NOT built by this stack. +# +# Putting `book texlive` last only moved the ENTRY: texlive is harfbuzz's +# recommendation, so it was still pulled into an early entry's plan and +# failed there (its configure wants freetype-config, which FreeType removed +# years ago -- an upstream problem, not this desktop's). Nothing here needs +# TeX: the book's `make ... pdf ps` manual builds are commented out (1.12.87) +# and asymptote/biber are avoided. +# +# To build it later, on its own: packagemanager install texlive --run +avoid texlive diff --git a/stacks/bootscripts.conf b/stacks/bootscripts.conf new file mode 100644 index 0000000..7ece1b1 --- /dev/null +++ b/stacks/bootscripts.conf @@ -0,0 +1,44 @@ +# bootscripts.conf -- which blfs-bootscripts target belongs to which package. +# +# A BLFS package installs a daemon. What makes that daemon start at boot is +# a SEPARATE tarball, blfs-bootscripts, with one `make install-<target>` per +# service. Nothing connects the two: install a package from a stack and its +# bootscript is simply not there, and the service silently never runs. +# +# The services stack installs a fixed list of targets, which was right on the +# day it was written -- `install-service-dhcpcd WAS MISSING` is still in the +# comment above it. This file is the list as a MAPPING instead, so `cfg +# bootscripts` can ask the only question that matters: is the package here, +# and is its bootscript not? +# +# package | proof it is installed | make targets | what those create +# +# package what you would install; for the record, nothing looks it up +# proof an absolute path (tested with -e) or a command name (looked up +# on PATH). Several, comma separated: ANY of them is enough -- +# a daemon that moved between /sbin and /usr/sbin is still that +# daemon. `-` means "no package needed" +# targets one or more `make` targets in the blfs-bootscripts tree +# creates the file those targets write. Several, comma separated: any +# one present means the work is done. This is what makes the +# step idempotent, and what it checks AFTERWARDS -- a make that +# prints no error and installs nothing is the failure to catch +# +# A line whose package is not installed is skipped: an init script for +# something that is not there is a symlink in rc?.d that fails at boot. + +package | proof | targets | creates +dbus | dbus-daemon,/usr/bin/dbus-daemon | install-dbus | /etc/rc.d/init.d/dbus +alsa-utils | alsactl,/usr/sbin/alsactl | install-alsa | /etc/rc.d/init.d/alsa +bluez | /usr/libexec/bluetooth/bluetoothd,bluetoothd | install-bluetooth | /etc/rc.d/init.d/bluetooth +acpid | acpid,/usr/sbin/acpid | install-acpid | /etc/rc.d/init.d/acpid +fcron | fcron,/usr/sbin/fcron | install-fcron | /etc/rc.d/init.d/fcron +iptables | iptables,/usr/sbin/iptables | install-iptables | /etc/rc.d/init.d/iptables +openssh | sshd,/usr/sbin/sshd | install-sshd | /etc/rc.d/init.d/sshd +wpa_supplicant | wpa_supplicant,/sbin/wpa_supplicant | install-service-wpa | /lib/services/wpa,/usr/lib/services/wpa +# dhcpcd ships NO service script of its own and BLFS has no init.d/dhcpcd at +# all: /lib/services/dhcpcd is the whole of it, and it runs only because +# /etc/sysconfig/ifconfig.<iface> says SERVICE="dhcpcd" (see below). +dhcpcd | dhcpcd,/usr/sbin/dhcpcd,/sbin/dhcpcd | install-service-dhcpcd | /lib/services/dhcpcd,/usr/lib/services/dhcpcd +# the entropy pool is saved and restored by a script that needs no package +- | - | install-random | /etc/rc.d/init.d/random diff --git a/stacks/bootscripts.sh b/stacks/bootscripts.sh new file mode 100644 index 0000000..82d17e8 --- /dev/null +++ b/stacks/bootscripts.sh @@ -0,0 +1,199 @@ +#!/bin/bash +# bootscripts.sh -- give every installed service the bootscript it needs. +# +# THE PACKAGE AND THE THING THAT STARTS IT COME FROM DIFFERENT TARBALLS. +# A BLFS package installs a daemon; blfs-bootscripts installs the script that +# runs it at boot, one `make install-<target>` per service. Nothing connects +# them, so a package added to a stack later has no bootscript and its service +# quietly never starts -- which is how `install-service-dhcpcd` came to be +# missing from a list that already had nine other targets in it. +# +# This reads bootscripts.conf, which is that list as a MAPPING, and installs +# the targets whose package is here and whose script is not. Idempotent: it +# checks what exists before and after, and a package that is not installed is +# skipped -- an init script for something absent is a symlink in rc?.d that +# fails at boot. +set -u + +CONF="${BOOTSCRIPTS_CONF:-/etc/pkgusr/stacks/bootscripts.conf}" +SRC="${BOOTSCRIPTS_SRC:-}" +BASE="${PKGUSR_BASE:-/usr/src}" +DO_IFCONFIG="${BOOTSCRIPTS_IFCONFIG:-yes}" +SYSCONFIG="${BOOTSCRIPTS_SYSCONFIG:-/etc/sysconfig}" +SYSNET="${BOOTSCRIPTS_SYSNET:-/sys/class/net}" +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } +trim() { printf '%s' "${1:-}" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//'; } + +# a path is tested, a bare name is looked up on PATH; ANY alternative counts, +# because a daemon that moved between /sbin and /usr/sbin is still installed +have_any() { + local _c + for _c in $(printf '%s' "${1:-}" | tr ',' ' '); do + case "$_c" in + -) return 0 ;; + /*) [ -e "$_c" ] && return 0 ;; + *) command -v "$_c" >/dev/null 2>&1 && return 0 ;; + esac + done + return 1 +} + +[ -r "$CONF" ] || { warn "no $CONF to read"; exit 1; } + +# ---- 1. the unpacked blfs-bootscripts tree -------------------------------- +# WHERE THE RUNNER PUTS IT, not where it seems it should be. lfs-phases +# builds in BUILD_ROOT, which is `$HOME/src` when the package user has one -- +# /usr/src/p_blfs-bootscripts/SRC/blfs-bootscripts-<date>, one level deeper +# than the home. Looking only in the home found nothing and told the reader +# to unpack a tree that was already there. +SOURCES="${LFS_SOURCES_DIR:-/sources}" +TMPTREE="" +if [ -z "$SRC" ]; then + for d in "$BASE"/*/src/blfs-bootscripts-* "$BASE"/*/blfs-bootscripts-* \ + "$BASE"/blfs-bootscripts-* "$SOURCES"/blfs-bootscripts-*; do + [ -f "$d/Makefile" ] || continue + if [ -z "$SRC" ] || [ "$d" -nt "$SRC" ]; then SRC="$d"; fi + done +fi +if [ -z "$SRC" ]; then + # anything those four patterns did not think of, bounded so this can + # never turn into a walk of the whole package tree + d="$(find "$BASE" -maxdepth 4 -type d -name 'blfs-bootscripts-*' 2>/dev/null \ + | sort | tail -n1)" + [ -n "$d" ] && [ -f "$d/Makefile" ] && SRC="$d" +fi +if [ -z "$SRC" ]; then + # THE TARBALL IS ENOUGH. A build tree can be cleaned away; the sources + # directory keeps the tarball, and these targets only copy files. + tb="$(ls -1t "$SOURCES"/blfs-bootscripts-*.tar.* 2>/dev/null | head -n1)" + if [ -n "$tb" ]; then + TMPTREE="$(mktemp -d)" + if tar -xf "$tb" -C "$TMPTREE" 2>/dev/null; then + SRC="$(ls -d "$TMPTREE"/blfs-bootscripts-* 2>/dev/null | head -n1)" + note "no build tree left -- unpacked $tb" + note " (installing from a temporary copy, so these files land" + note " owned by root rather than by the bootscripts package user)" + fi + fi +fi +if [ -z "$SRC" ] || [ ! -f "$SRC/Makefile" ]; then + warn "no blfs-bootscripts tree and no tarball -- nothing to install from" + note " looked in: $BASE/*/src, $BASE/*, $BASE, $SOURCES" + note " fetch and unpack it again:" + note " packagemanager stack services --only blfs-bootscripts --run --force" + exit 1 +fi +trap '[ -n "$TMPTREE" ] && rm -rf "$TMPTREE"' EXIT +note "bootscripts: $SRC" + +# ---- 2. what is missing --------------------------------------------------- +targets=""; skipped=""; already=""; checks="" +while IFS= read -r raw || [ -n "$raw" ]; do + case "$(trim "$raw")" in ""|\#*) continue ;; esac + IFS='|' read -r f1 f2 f3 f4 <<< "$raw" + pkg="$(trim "$f1")"; proof="$(trim "$f2")" + tgts="$(trim "$f3")"; creates="$(trim "$f4")" + case "$pkg" in package|user) continue ;; esac + [ -n "$tgts" ] || continue + if ! have_any "$proof"; then skipped="$skipped $pkg"; continue; fi + if have_any "$creates"; then already="$already $pkg"; continue; fi + note "$pkg is here, ${creates%%,*} is not -- $tgts" + targets="$targets $tgts" + checks="$checks$pkg|$creates +" +done < "$CONF" + +[ -n "$already" ] && note "already installed:$already" +[ -n "$skipped" ] && note "package not installed, so no bootscript:$skipped" + +# ---- 3. install them, as whoever owns the tree ---------------------------- +# The first install ran as the package user, so /etc/rc.d/init.d and the rest +# are that user's. Running make as root would work and leave root's files in +# a tree a package owns -- ownership is what says who installed what here, and +# it should keep saying it. Root is the fallback, said out loud. +if [ -n "$targets" ]; then + owner="$(stat -c %U "$SRC" 2>/dev/null || echo root)" + note "make$targets (as $owner)" + ok=0 + if [ "$owner" != root ] && id "$owner" >/dev/null 2>&1; then + su - "$owner" -c "cd $(printf '%q' "$SRC") && make$targets" && ok=1 + [ "$ok" = 1 ] || note " as $owner that failed -- trying as root" + fi + if [ "$ok" != 1 ]; then + ( cd "$SRC" && make$targets ) && ok=1 + fi + [ "$ok" = 1 ] || warn "make$targets failed" + # A MAKE THAT PRINTS NO ERROR IS NOT A FILE THAT EXISTS. Ask the disk. + while IFS='|' read -r pkg creates; do + [ -n "${pkg:-}" ] || continue + have_any "$creates" \ + && note " now there: ${creates%%,*} ($pkg)" \ + || warn " $pkg: make ran, ${creates%%,*} still is not there" + done <<< "$checks" +else + note "every installed service already has its bootscript" +fi +echo + +# ---- 4. dhcpcd needs one more thing, and it is not a bootscript ----------- +# /lib/services/dhcpcd is not started by anything on its own, and BLFS has no +# init.d/dhcpcd at all. The LFS `network` script reads /etc/sysconfig/ +# ifconfig.* at boot and hands each interface to the service ITS file names -- +# so a machine with dhcpcd installed and its service script in place still +# comes up with no address when no such file exists. That is the whole of +# "my DHCP does not start automatically". +# +# WIRELESS IS NOT DONE HERE. That file needs an SSID and a key, which nothing +# may invent; `check net-wifi` prints it for you to paste. A WIRED file holds +# no secret at all -- interface, service, two flags -- so there is no reason to +# make anyone type it. +if [ "$DO_IFCONFIG" = yes ] && have_any "/lib/services/dhcpcd,/usr/lib/services/dhcpcd"; then + wrote=0 + for d in "$SYSNET"/*; do + [ -d "$d" ] || continue + i="$(basename "$d")" + [ "$i" = lo ] && continue + if [ -e "$d/wireless" ] || [ -e "$d/phy80211" ]; then + note "$i is wireless -- its file needs an SSID: check net-wifi" + continue + fi + f="$SYSCONFIG/ifconfig.$i" + if [ -e "$f" ]; then + if grep -qE '^[[:space:]]*SERVICE=.?dhcpcd' "$f"; then + note "$f already hands $i to dhcpcd" + else + note "$f exists and names another service -- left alone" + fi + continue + fi + mkdir -p "$SYSCONFIG" + cat > "$f" <<EOF +# written by stacks/bootscripts.sh -- safe to delete +# $i takes its address from DHCP at boot. ONBOOT="no" turns that off +# without removing the file; SERVICE names /lib/services/dhcpcd. +ONBOOT="yes" +IFACE="$i" +SERVICE="dhcpcd" +DHCP_START="-b -q" +DHCP_STOP="-k" +EOF + chmod 644 "$f"; chown root:root "$f" 2>/dev/null + note "wrote $f -- $i now asks for an address at boot" + note " without rebooting: ifup $i" + wrote=$((wrote + 1)) + done + [ "$wrote" = 0 ] || note "$wrote interface file(s) written" +fi + +echo +if [ "$fail" = 0 ]; then + echo "# bootscripts: done." + echo "# what starts at boot: ls /etc/rc.d/rc3.d" + echo "# wireless needs an SSID, which nothing here can write:" + echo "# packagemanager stack network --only net-wifi --run" +else + echo "# the !! lines above need attention; everything else was applied." +fi +exit $fail diff --git a/stacks/dev.stack b/stacks/dev.stack new file mode 100644 index 0000000..8a8d426 --- /dev/null +++ b/stacks/dev.stack @@ -0,0 +1,43 @@ +# dev.stack -- the tools for looking at a program that has gone wrong. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +include avoid + +# +# packagemanager stack dev --run +# +# WHY ITS OWN STACK, and not one of the others. Every stack here answers one +# question. services = what starts at boot; network = how the machine gets an +# address; sway = the desktop; diag = is the GPU doing what I think; media = +# what plays. A debugger is none of those, and putting it in the nearest one +# would be the start of a stack that means nothing. This is the one that +# means "the program crashed and I want to know why". +# +# Nothing depends on these. They are safe to build last, and safe not to +# build at all -- which is the other reason they are not in a stack whose +# entries the desktop needs. + +# GDB. The book's page takes --with-system-readline, --with-system-zlib and +# --with-python=/usr/bin/python3, and installs gdb, gdbserver, gcore, gstack +# and gdb-add-index; the generated script comes from that page, so none of it +# is repeated here. +# +# Its only "required" dependency is the python module `six`, and that is for +# gdb's own Python scripting at RUNTIME -- gdb builds and debugs without it. +# The rest of the book's list (guile, doxygen, valgrind, rustc, SystemTap) is +# for gdb's test suite, not for gdb. +# +# On a package-user system the thing to remember is the OTHER half of +# debugging: a binary built by a package user carries whatever debug +# information its own build left in, and `packagemanager` does not strip +# anything. If a backtrace is all `??`, the package was built without -g and +# has to be rebuilt, not re-debugged: +# packagemanager update <pkg> --reinstall --regenerate --run +# with `[<pkg>] cflags = -g` in machine.conf. +book gdb expect="test -x /usr/bin/gdb" + +# Room here, deliberately: strace, ltrace and valgrind are the obvious +# neighbours and each is a BLFS page. They are not added unasked -- an entry +# in a stack is a package that gets built on every replay of it. diff --git a/stacks/diag.stack b/stacks/diag.stack new file mode 100644 index 0000000..11dbeb7 --- /dev/null +++ b/stacks/diag.stack @@ -0,0 +1,112 @@ +# diag.stack -- GPU diagnostic tools: glxinfo, eglinfo, vulkaninfo, vainfo, nvtop. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack. +include avoid + +# +# Replay with: packagemanager stack diag --run +# +# WHY ITS OWN STACK. Nothing needs these; they answer "is the GPU doing +# what I think?" None of the four is a BLFS page (mesa-demos is only an +# optional link on the Mesa page), so the refs, build lines and probes are +# ours to keep true. Each entry says what was checked. +# +# Probes check the programs exist, not that they talk to a GPU: the stack +# may run in a chroot with no display and no /dev/dri. On the running +# desktop, try: +# glxinfo -B eglinfo -B (Mesa renderer, e.g. radeonsi) +# vulkaninfo --summary (RADV on AMD) +# vainfo (needs Mesa built with VA-API) +# nvtop + +# =========================================================================== +# GPU VENDOR -- the only choice in this file. THIS MACHINE: AMD (onboard). +# =========================================================================== +# +# Only nvtop is vendor-specific; the other three work for any Mesa driver. +# To change vendor, edit the -D..._SUPPORT flags in the nvtop entry: +# +# AMD -DAMDGPU_SUPPORT=ON (amdgpu kernel driver, libdrm) +# Intel -DINTEL_SUPPORT=ON (i915 / xe kernel driver, libdrm) +# NVIDIA -DNVIDIA_SUPPORT=ON (proprietary driver: loads libnvidia-ml +# at runtime; nouveau is NOT supported) +# +# More than one may be ON (a laptop with two GPUs). Then rebuild it: +# packagemanager stack diag --only nvtop --run --force +# +# vainfo on NVIDIA needs a VA-API driver that is not Mesa (nvidia-vaapi- +# driver); vainfo itself is unchanged. + +# mesa-demos -> glxinfo, glxgears, eglinfo (and ~300 GL demos) +# +# NOT CHECKED from here: gitlab.freedesktop.org is unreachable where this +# was written. mesa-demos-9.0.0 is the release BLFS pointed at; check for +# a newer one before trusting the pin: +# git ls-remote --tags https://gitlab.freedesktop.org/mesa/demos.git +# glut=disabled: freeglut is in avoid.stack, and only a few demos use it. +# If meson rejects an option name, the tag's meson_options.txt has the +# real ones. +# +# libdecor-0:demo=false. The wayland demos pull libdecor in as a meson +# subproject, and its OWN demo needs pkg-config `opengl` (libglvnd's). That +# failed here once, before Mesa was switched to glvnd -- now it would build, +# but nobody runs libdecor's demo, so it stays off. +# +# `rm -rf build` first: a failed `meson setup` leaves a half-configured +# build directory, and the retry then stops on "already configured". +git mesa-demos \ + url=https://gitlab.freedesktop.org/mesa/demos.git \ + ref=mesa-demos-9.0.0 \ + needs=mesa,glu,wayland-protocols \ + have=/usr/bin/eglinfo \ + build="rm -rf build; meson setup build --prefix=/usr --buildtype=release -Dglut=disabled -Dgles1=disabled -Dosmesa=disabled -Dlibdecor-0:demo=false; ninja -C build" \ + install="ninja -C build install" \ + expect="test -x /usr/bin/glxinfo && test -x /usr/bin/eglinfo" + +# vulkan-tools -> vulkaninfo, vkcube +# +# CHECKED: github KhronosGroup/Vulkan-Tools tags are vulkan-sdk-1.4.N.M, up +# to vulkan-sdk-1.4.357.0. Its CMake asks for VulkanHeaders of ITS OWN +# version, so a fixed pin breaks the first time the book moves the headers. +# The clone is full (tags included), so the build checks out the newest tag +# for the INSTALLED loader's version -- nothing to edit on an update. +# BUILD_ICD=OFF: the mock driver is for testing the loader, not a GPU. +git vulkan-tools \ + url=https://github.com/KhronosGroup/Vulkan-Tools.git \ + ref=main \ + needs=vulkan-loader,glslang,wayland-protocols \ + have=/usr/bin/vulkaninfo \ + build='v=$(pkg-config --modversion vulkan); t=$(git tag -l "vulkan-sdk-$v.*" | sort -V | tail -1); test -n "$t" || { echo "no Vulkan-Tools tag for vulkan $v"; exit 1; }; git checkout -q "$t" && cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DBUILD_ICD=OFF -DBUILD_TESTS=OFF && cmake --build build' \ + install="cmake --install build" \ + expect="test -x /usr/bin/vulkaninfo" + +# libva-utils -> vainfo +# +# CHECKED: github intel/libva-utils tags 2.20.1 ... 2.24.0. Its minor +# follows libva's (libva 2.24 = VA-API 1.24, whose .pc reports 1.24.x), so +# the build picks the newest 2.<that minor>.* tag. On AMD, vainfo lists +# profiles only if Mesa was built with its VA-API driver (radeonsi). +git libva-utils \ + url=https://github.com/intel/libva-utils.git \ + ref=master \ + needs=libva \ + have=/usr/bin/vainfo \ + build='m=$(pkg-config --modversion libva | cut -d. -f2); t=$(git tag -l "2.$m.*" | sort -V | tail -1); test -n "$t" || { echo "no libva-utils tag for libva 2.$m"; exit 1; }; git checkout -q "$t" && rm -rf build && meson setup build --prefix=/usr --buildtype=release -Dx11=false -Dwayland=true -Ddrm=true && ninja -C build' \ + install="ninja -C build install" \ + expect="test -x /usr/bin/vainfo" + +# nvtop -> nvtop (GPU load, memory, processes) +# +# CHECKED: github Syllo/nvtop, newest tag 3.3.2; option names from its +# CMakeLists.txt. On x86 Linux EVERY backend defaults ON, so all are named +# here -- ON only for this machine's GPU (see GPU VENDOR above). +# USE_LIBUDEV_OVER_LIBSYSTEMD: this is a SysV system; libudev comes from udev. +git nvtop \ + url=https://github.com/Syllo/nvtop.git \ + ref=3.3.2 \ + needs=libdrm \ + have=/usr/bin/nvtop \ + build="cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_LIBUDEV_OVER_LIBSYSTEMD=ON -DAMDGPU_SUPPORT=ON -DINTEL_SUPPORT=OFF -DNVIDIA_SUPPORT=OFF -DMSM_SUPPORT=OFF -DPANFROST_SUPPORT=OFF -DPANTHOR_SUPPORT=OFF -DV3D_SUPPORT=OFF -DMETAX_SUPPORT=OFF -DTPU_SUPPORT=OFF -DENFLAME_SUPPORT=OFF && cmake --build build" \ + install="cmake --install build" \ + expect="nvtop --version >/dev/null" diff --git a/stacks/fonts.sh b/stacks/fonts.sh new file mode 100644 index 0000000..7f894e0 --- /dev/null +++ b/stacks/fonts.sh @@ -0,0 +1,149 @@ +#!/bin/bash +# fonts.sh -- icon, patched and Noto fonts, from release archives. +# +# None of these is a BLFS page, so the URLs and versions are OURS to keep +# true -- the same deal as gst-plugins-rs. Each line below is one font set: +# +# <name>|<url>|<glob of font files inside the archive> +# +# WHAT WAS CHECKED, AND HOW. Every URL here answered `wget --spider` from +# the machine this was written on. The Noto ones are individual faces from +# noto-fonts' hinted tree: the release-asset zips this first used +# (notofonts.github.io/.../Noto_Sans.zip) do not exist -- that repository +# publishes per-script archives under names this did not guess, and guessing +# cost a run. Six faces, fetched directly, are smaller than the zips anyway. +# +# A line can still go stale. When one does it fails on ITS OWN line with the +# URL printed, the others carry on, and the step says so at the end. Fix the +# line and run it again; what is installed is skipped. +# +# The same name may appear more than once -- each line is one file, and they +# accumulate in the same directory. +# +# THREE UNPACKERS, WHICHEVER IS THERE. bsdtar (libarchive) reads zip as +# happily as tar; unzip may not be installed at all (BLFS's UnZip is a +# separate page and nothing here requires it); python3 always is, because +# packagemanager is written in it. Depending on exactly one of the three is +# how a font step fails on a system that could perfectly well unpack the +# archive. +set -u + +FONTDIR=/usr/share/fonts +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } + +fonts() { + cat <<'FONTS' +fontawesome|https://github.com/FortAwesome/Font-Awesome/releases/download/6.7.2/fontawesome-free-6.7.2-desktop.zip|*otfs/*.otf +nerd-symbols|https://github.com/ryanoasis/nerd-fonts/releases/download/v3.3.0/NerdFontsSymbolsOnly.zip|*.ttf +nerd-jetbrains|https://github.com/ryanoasis/nerd-fonts/releases/download/v3.3.0/JetBrainsMono.zip|*.ttf +noto-sans|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSans/NotoSans-Regular.ttf|NotoSans-Regular.ttf +noto-sans|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSans/NotoSans-Bold.ttf|NotoSans-Bold.ttf +noto-sans|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSans/NotoSans-Italic.ttf|NotoSans-Italic.ttf +noto-serif|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSerif/NotoSerif-Regular.ttf|NotoSerif-Regular.ttf +noto-serif|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSerif/NotoSerif-Bold.ttf|NotoSerif-Bold.ttf +noto-mono|https://github.com/googlefonts/noto-fonts/raw/main/hinted/ttf/NotoSansMono/NotoSansMono-Regular.ttf|NotoSansMono-Regular.ttf +noto-emoji|https://github.com/googlefonts/noto-emoji/raw/main/fonts/NotoColorEmoji.ttf|NotoColorEmoji.ttf +FONTS +} + +unpack() { # unpack <archive> <into> + if command -v bsdtar >/dev/null 2>&1 && bsdtar -xf "$1" -C "$2" 2>/dev/null; then + return 0 + fi + case "$1" in + *) if command -v unzip >/dev/null 2>&1 \ + && unzip -qo "$1" -d "$2" 2>/dev/null; then return 0; fi ;; + esac + if command -v tar >/dev/null 2>&1 && tar -xf "$1" -C "$2" 2>/dev/null; then + return 0 + fi + python3 - "$1" "$2" <<'PYUNPACK' 2>/dev/null +import sys, zipfile, tarfile +src, dst = sys.argv[1], sys.argv[2] +if zipfile.is_zipfile(src): + zipfile.ZipFile(src).extractall(dst) +else: + tarfile.open(src).extractall(dst) +PYUNPACK +} + +installed=0 +while IFS='|' read -r name url glob; do + [ -n "$name" ] || continue + dest="$FONTDIR/$name" + # ALREADY THERE IS DONE. This step runs on every stack replay, and + # re-downloading a font nobody changed is time spent for nothing. + if [ -d "$dest" ] && [ -n "$(ls -A "$dest" 2>/dev/null)" ]; then + note "$name: already installed ($(ls -1 "$dest" | wc -l) file(s))" + continue + fi + tmp="$(mktemp -d)" + note "$name: fetching $(basename "$url")" + if ! wget -q -O "$tmp/archive" "$url"; then + warn "$name: could not download" + note " $url" + rm -rf "$tmp"; continue + fi + # a bare .ttf is not an archive: NotoColorEmoji is published as the font + case "$url" in + *.ttf|*.otf) + mkdir -p "$tmp/x" && cp "$tmp/archive" "$tmp/x/$(basename "$url")" ;; + *) + mkdir -p "$tmp/x" + if ! unpack "$tmp/archive" "$tmp/x"; then + warn "$name: downloaded, but nothing here could unpack it" + note " is $url still an archive? $(stat -c %s "$tmp/archive") bytes" + rm -rf "$tmp"; continue + fi ;; + esac + # find -path, NOT a shell glob. `*` does not cross a `/`, so + # `*otfs/*.otf` could never match + # fontawesome-free-6.7.2-desktop/otfs/fa-solid-900.otf -- the archive + # everyone actually publishes, with one directory at the top. find's + # -path does cross slashes, and then one pattern covers both layouts. + files="$(cd "$tmp/x" && find . -type f -path "*/$glob" -o -type f -path "./$glob" 2>/dev/null | head -400)" + if [ -z "$files" ]; then + warn "$name: nothing matched '$glob' in the archive" + note " what it contains: $(cd "$tmp/x" && ls -1 | head -5 | tr '\n' ' ')" + rm -rf "$tmp"; continue + fi + install -v -d -m755 "$dest" >/dev/null + ( cd "$tmp/x" && printf '%s\n' "$files" | while IFS= read -r ff; do + [ -n "$ff" ] && cp -f "$ff" "$dest/" + done ) 2>/dev/null \ + || { warn "$name: could not copy into $dest"; rm -rf "$tmp"; continue; } + chmod 644 "$dest"/* 2>/dev/null + note "$name: $(ls -1 "$dest" | wc -l) file(s) -> $dest" + installed=$((installed + 1)) + rm -rf "$tmp" +done < <(fonts) + +# ONE CACHE REBUILD, AT THE END. fc-cache per font set is the same work +# repeated, and a font that is present but uncached is a font nothing can +# find -- which looks exactly like a font that is not installed. +if [ "$installed" -gt 0 ]; then + note "rebuilding the font cache" + fc-cache -f >/dev/null 2>&1 || warn "fc-cache failed; fonts are installed but not indexed" +fi + +# CONFIRM YOUR OWN EFFECT: fontconfig has to be able to see them. +echo +for want in "Font Awesome" "Symbols Nerd Font" "Noto Sans"; do + if fc-list 2>/dev/null | grep -qi "$want"; then + note "fontconfig sees: $want" + else + echo "-> fontconfig does not see '$want' yet" + fi +done + +echo +if [ "$fail" = 0 ]; then + echo "# fonts: done. $(fc-list 2>/dev/null | wc -l) font file(s) known to fontconfig" +else + echo "# the !! lines are URLs or archive layouts that have moved." + echo "# Each one is a single line in this script's table; fix it and" + echo "# run the step again -- what is installed is skipped." +fi +exit $fail diff --git a/stacks/iptables-nft.sh b/stacks/iptables-nft.sh new file mode 100644 index 0000000..04c1189 --- /dev/null +++ b/stacks/iptables-nft.sh @@ -0,0 +1,86 @@ +#!/bin/bash +# iptables-nft.sh -- point the generic iptables names at the nft backend. +# +# BUILDING NFTABLES SUPPORT DOES NOT SWITCH TO IT. `--enable-nftables` +# produces BOTH backends and upstream's `make install` leaves the plain names +# on the legacy one: +# +# /usr/sbin/iptables -> xtables-legacy-multi +# /usr/sbin/iptables-nft -> xtables-nft-multi +# +# so `iptables --version` says (legacy), and on a kernel with no ip_tables +# every rule fails -- with a build log that says "nftables support: yes". +# Distributions flip these links (Debian with alternatives, Arch in the +# package); here it is this step. +# +# Reversible, and says how: the legacy binary is not removed, only unlinked +# from the generic name. +set -u + +NFT=/usr/sbin/xtables-nft-multi +LEGACY=/usr/sbin/xtables-legacy-multi +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } + +if [ ! -x "$NFT" ]; then + warn "no $NFT -- iptables was built without nftables support" + note " add this to /etc/pkgusr/stacks/machine.conf:" + note " [iptables]" + note " configure_args = --enable-nftables" + note " then: packagemanager update iptables --reinstall --regenerate --run --yes" + exit 1 +fi + +# The generic names, and the per-family ones that go with them. ebtables and +# arptables have no legacy build here, so they are already nft and are left +# alone. +changed=0 +for n in iptables iptables-save iptables-restore \ + ip6tables ip6tables-save ip6tables-restore; do + t="/usr/sbin/$n" + if [ -L "$t" ] && [ "$(readlink "$t")" = "$(basename "$NFT")" ]; then + note "$n already uses the nft backend" + continue + fi + if [ -e "$t" ] && [ ! -L "$t" ]; then + # A REAL FILE IS NOT OURS TO REPLACE. Everything iptables installs + # under these names is a symlink into one of the two multi binaries; + # a regular file here is something else, and guessing is worse than + # stopping. + warn "$t is a real file, not a symlink -- left alone" + note " whatever installed it is not this iptables build; look at it" + note " before switching backends: packagemanager which-package $t" + continue + fi + if ln -sfn "$(basename "$NFT")" "$t"; then + note "$n -> $(basename "$NFT")" + changed=$((changed + 1)) + else + warn "could not point $n at the nft backend" + fi +done + +# CONFIRM YOUR OWN EFFECT: the whole point is what --version prints. +echo +got="$(iptables --version 2>/dev/null)" +case "$got" in + *nf_tables*) note "iptables now reports: $got" ;; + *) + warn "iptables still reports: ${got:-<nothing>}" + note " the links are set but the binary disagrees; check:" + note " ls -l /usr/sbin/iptables /usr/sbin/xtables-nft-multi" + ;; +esac + +if [ "$fail" = 0 ]; then + echo "# iptables: nft backend, $changed link(s) changed." + echo "# the legacy binary is still there and still usable by name:" + echo "# iptables-legacy --version" + echo "# to go back: ln -sfn xtables-legacy-multi /usr/sbin/iptables" + echo "# rules in /etc/rc.d/rc.iptables are read by either backend;" + echo "# the nft one needs CONFIG_NFT_COMPAT in the kernel." +else + echo "# fix the !! lines above" +fi +exit $fail diff --git a/stacks/kernel-sway.sh b/stacks/kernel-sway.sh index 2c61ca4..25c30d9 100755 --- a/stacks/kernel-sway.sh +++ b/stacks/kernel-sway.sh @@ -24,6 +24,29 @@ need CONFIG_SND "sound core" need CONFIG_SECCOMP "webkit's sandbox" need CONFIG_USER_NS "webkit's sandbox (bubblewrap)" need CONFIG_TMPFS "XDG_RUNTIME_DIR lives on tmpfs" +# THE SOFTWARE RENDERER NEEDS ITS OWN KERNEL OPTION. The book's mesa page: +# "For llvmpipe or softpipe, CONFIG_DRM_VGEM is required or X clients will +# fail". Without it the software fallback that exists to save you when the +# hardware path fails cannot be used either -- and the symptom is the same +# "failed to create dri2 screen". A note, not a failure: a machine that +# never falls back does not need it. +# `need` fails the stack; this one only reports, so it reads the config the +# same way rather than inventing a second reader. +want() { # want <config-option> <why> + local cfg="" opt="$1" why="$2" + for c in /proc/config.gz "/boot/config-$(uname -r)" /boot/config; do + [ -r "$c" ] && cfg="$c" && break + done + [ -n "$cfg" ] || return 0 + if case "$cfg" in + *.gz) zgrep -qE "^$opt=(y|m)" "$cfg" ;; + *) grep -qE "^$opt=(y|m)" "$cfg" ;; + esac + then echo "# $opt is set -- $why" + else echo "-> $opt is NOT set -- $why" + fi +} +want CONFIG_DRM_VGEM "the book: llvmpipe/softpipe need it or X clients fail" # runtime, not config: the render node must exist [ -e /dev/dri/renderD128 ] || { echo "!! no /dev/dri/renderD* render node"; fail=1; } [ "$fail" = 0 ] && echo "# kernel: everything sway + webkit need is present" diff --git a/stacks/machine.conf b/stacks/machine.conf index 91052e4..7a506c5 100644 --- a/stacks/machine.conf +++ b/stacks/machine.conf @@ -2,17 +2,103 @@ # scripts' `meson setup` lines as -Dkey=value the moment the script is born. # The stack file stays portable; this file changes per computer. +# HOW MANY JOBS, ONCE. make reads MAKEFLAGS, cmake reads +# CMAKE_BUILD_PARALLEL_LEVEL, cargo reads CARGO_BUILD_JOBS -- and ninja reads +# nothing at all, defaulting to nproc+2, which is why the number has to be +# patched into its command line. [all] covers every package here; a named +# section below overrides it (webkitgtk keeps 8 for RAM reasons). +# +# [all] +# jobs = 13 +# +# `jobs` is patched into `ninja`, `./x.py` and `cargo build|install` lines +# when the script is generated -- rust's page says plainly that only the +# COMBINATION of --jobs on x.py and CARGO_BUILD_JOBS limits it. A package +# built before you set this needs --regenerate to pick it up. +# +# ...and for make/cmake/cargo, one line in /etc/pkgusr/build.env: +# MAKEFLAGS=-j13 +# (lfs-helper derives the cmake and cargo variables from it, or from --jobs.) + +# A HARDWARE-ONLY MESA HAS NOTHING TO FALL BACK TO. +# +# `gallium-drivers = radeonsi` alone means that when the hardware path +# cannot be set up for some EGL platform, there is no software renderer +# behind it and the app gets +# libEGL warning: egl: failed to create dri2 screen +# and then nothing. The book says it twice, in its own words: for +# gallium-drivers "You should probably add the softpipe or llvmpipe option +# as a backup driver", and for vulkan-drivers "You should probably add the +# swrast option as a backup driver unless you don't need Vulkan at all". +# +# llvmpipe (not softpipe) because llvm is enabled below: it is the faster of +# the two software rasterisers and it is built from the same LLVM this +# machine already has. +# READ ONE PACKAGE FROM A DIFFERENT BOOK. +# +# The systemd book's linux-pam page lists systemd as REQUIRED; the 12.4 +# SysV book's page has no Required section at all -- it wants only Shadow. +# Same package, two books, two dependency graphs, and this machine is SysV. +# `book =` points one package at another book, for everything: versions, +# dependencies and the generated script. +# +# Import the book once, then uncomment: +# blfs import /path/to/BLFS-BOOK-12_4-nochunks.html +# +# [somepackage] +# book = /usr/share/blfs/books/BLFS-BOOK-12_4-nochunks.html +# +# AND linux-pam IS NOT THE CASE FOR IT, on the evidence. Both pages were +# compared command by command: +# 13.1: Linux-PAM-1.7.2; systemd is in the REQUIRED list and in NO command +# 12.4: Linux-PAM-1.7.1; its commands include `rm -rf /usr/lib/systemd` +# So the newer page needs nothing from systemd to build -- the requirement +# is metadata -- and since 1.14.234 `install` drops avoided packages from a +# plan anyway. Pointing linux-pam at 12.4 would cost a version (1.7.2 -> +# 1.7.1) and buy nothing. Use `book =` where the COMMANDS differ, not +# where only the dependency list does. +# +# NOT a merge of two pages: taking the dependency list from one book and the +# commands from another gives a script neither book has tested -- the +# versions differ (1.7.1 vs 1.7.2), the switches differ, the doc tarball +# differs. A whole page from the older book is something upstream shipped +# and someone ran. + +# SHADOW: --disable-logind, because there is no libsystemd here. +# +# shadow 4.20 auto-detects logind and the book's page (written for a systemd +# host, where libsystemd is simply present) passes no switch either way, so +# configure stops: +# checking for sd_session_get_remote_host in -lsystemd... no +# configure: error: libsystemd not found. If logind integration is not +# required, disable it using the --disable-logind option. +# configure names the flag itself. This is the case `book =` exists for -- +# a page whose COMMANDS do not suit this machine -- but the 12.4 page would +# also mean shadow 4.18.0, and one switch is cheaper than four versions +# back. +[shadow] +configure_args = --disable-logind + [mesa] platforms = x11,wayland -gallium-drivers = radeonsi -vulkan-drivers = amd +gallium-drivers = radeonsi,llvmpipe +vulkan-drivers = amd,swrast video-codecs = all -glvnd = disabled +# glvnd: the GL dispatcher (libglvnd, wayland.stack, before mesa). Enabled so +# pkg-config `opengl` exists (libdecor's demo, some games and toolkits ask +# for it by that name) and a second GL vendor could be added. Changing it +# means rebuilding libglvnd and mesa together -- see the libglvnd entry. +glvnd = enabled libunwind = disabled llvm = enabled valgrind = disabled egl = enabled opengl = true +# gbm: Mesa's default is auto, which on Linux means built -- and wlroots, +# sway and every Wayland compositor need it. Named so that a Mesa build +# which cannot provide it FAILS at configure, instead of finishing without +# it and leaving sway to fail at start. +gbm = enabled # Per-package build options. Same mechanism as [mesa]: every key is merged # into that package's generated `meson setup` line as -Dkey=value. @@ -23,6 +109,70 @@ opengl = true [iptables] configure_args = --enable-nftables +# GST_DEBUG=<level> ONLY WORKS WITH THIS. The book builds the core with +# -D gst_debug=false, which compiles the whole debug-log system out; this +# replaces that switch (the script's -D gst_debug=false becomes =true). +# Cost: a slightly larger library and one integer compare per log site while +# logging is off -- upstream's own default, what distributions ship. +# `gstreamer:gst_debug` is the same switch for a build that bundles the core +# as a meson SUBPROJECT; BLFS builds the core as its own package, so only +# this one applies (the subproject key would be dropped as unknown). +# +# THE PLUGINS MUST BE REBUILT AFTERWARDS. gstconfig.h carries +# GST_DISABLE_GST_DEBUG from the core's build, and each plugin package +# compiled against it has its own log calls compiled out. +[gstreamer10] +gst_debug = true + +# --------------------------------------------------------------------------- +# TURNING AUTODETECT INTO A REQUIREMENT. +# +# gst-plugins-bad decides at configure time whether to build each plugin, +# from what it can find. Built before libnice existed, it produced +# libgstwebrtc-1.0.so (the library, always built) and no webrtcbin element -- +# and said nothing, because nothing went wrong. `-Dwebrtc=enabled` makes the +# same situation a build FAILURE, which is the outcome worth having. +# +# COMMENTED OUT because these are option names, and an option name is +# machine-checkable rather than memorable. Confirm them against the source +# you actually have, then uncomment: +# +# grep -n '^option(' /usr/src/pkgusr/p_gst10-plugins-bad/src/*/meson_options.txt +# meson configure /usr/src/pkgusr/p_gst10-plugins-bad/src/*/build | less +# +# A wrong name is not dangerous -- meson stops with +# ERROR: Unknown option: "..." +# before anything is built -- but it costs you a run. +# +# # x264enc EXISTS ONLY WITH THIS. The book: "Without this switch, plugins +# with dependencies on (A)GPL licensed libraries are not built" -- and +# x264enc is the H.264 encoder every camera site wants for publishing. +[gst10-plugins-ugly] +gpl = enabled + +# alsa IS A -BASE OPTION, NOT A -BAD ONE. It reads perfectly well under +# [gst10-plugins-bad] and meson fails the whole build on it ("Unknown +# option: alsa"); the ALSA source/sink plugin lives in gst-plugins-base. +# (packagemanager checks this against the package's own meson.options now +# and drops a key the package does not have, with the near misses.) +[gst10-plugins-base] +alsa = enabled + +# CHECKED against gst-plugins-bad's own meson.options: webrtc, dtls, srtp, +# sctp and openh264 are all real options of THIS package. Enabled rather +# than left on auto, so a missing dependency fails the build instead of +# silently dropping the plugin. +[gst10-plugins-bad] +webrtc = enabled +dtls = enabled +srtp = enabled +sctp = enabled +openh264 = enabled +# +# [gst10-plugins-good] +# v4l2 = enabled +# vpx = enabled + # libdisplay-info builds v4l-utils as a SUBPROJECT for its edid-decode # comparison tests: it clones the repo and installs libdvbv5, # libv4lconvert and a dozen locale files -- every one of them already owned @@ -31,3 +181,93 @@ configure_args = --enable-nftables # downloads. nodownload blocks that. [libdisplay-info] wrap_mode = nodownload + +# WEBKITGTK: THE SWITCHES THE BOOK TURNS OFF. +# +# Each of these is off in the book's own cmake line, and each is invisible +# when missing -- no error, just a feature that does not work: +# +# USE_WOFF2=OFF a .woff2 icon font renders as a blank square +# ENABLE_WEB_RTC=OFF no camera or screen share in the browser +# ENABLE_MEDIA_STREAM the getUserMedia half of the same thing +# USE_GSTREAMER_WEBRTC which backend does the WebRTC +# +# woff2 must be INSTALLED first (media.stack builds it just before +# webkitgtk), or the switch turns on support for a library that is not there. +# +# jobs = 8 rather than [all]'s 13: webkitgtk's link step is the biggest +# memory consumer in the stack, and the OOM killer takes the compiler. +# USE_LIBRICE=ON, and media.stack builds librice just before webkitgtk. +# +# webkitgtk 2.52 can take its ICE from librice instead of libnice, and with +# the GStreamer WebRTC backend on it defaults to librice: +# Could NOT find Rice (missing: Io Proto) +# librice-{io,proto} is needed for USE_LIBRICE. +# The BLFS page never hits this -- it does not enable USE_GSTREAMER_WEBRTC, +# so librice is not in the book at all (checked: the 13.1 book does not +# mention it). libnice IS installed here and is what the rest of the +# WebRTC chain already uses, so this says "keep using it". +# +# To go the other way instead -- libnice, which the rest of the chain uses +# and which is already installed -- set this to OFF and nothing needs +# building. +# POPPLER WITHOUT THE SIGNATURE STACK. +# +# poppler's cmake calls these "soft mandatory": absent, they stop configure +# and name the switch themselves -- +# Could not find the 3.98 version of NSS3. If you're not interested in +# the features it provides set the cmake ENABLE_NSS3 option to OFF +# nss, boost and gpgmepp are all in avoid.stack AS POPPLER'S CHILDREN (they +# were avoided because poppler was), so each would fail the same way in +# turn. They serve PDF signature verification and the splash backend; +# zathura's poppler-glib backend needs none of it. +# +# OBSERVED: ENABLE_NSS3 -- the failure above. +# ANTICIPATED: ENABLE_GPGME and ENABLE_BOOST, from the same avoid list and +# the same "soft mandatory" mechanism. If a later poppler stops on one of +# these with a different spelling, its message will name the real one. +[poppler] +# ENABLE_GLIB=ON: the whole reason poppler is here is zathura's +# poppler-glib backend -- +# Run-time dependency poppler-glib found: NO +# -- and cmake autodetects that frontend. The thing a chain exists for +# should be asked for, not inferred (the gst-plugins-ugly gpl rule again). +ENABLE_GLIB = ON +ENABLE_NSS3 = OFF +ENABLE_GPGME = OFF +ENABLE_BOOST = OFF +ENABLE_QT6 = OFF +# +# FOUR OF THEM, FOUND ONE CONFIGURE AT A TIME. NSS3, then GPGME and BOOST +# (guessed from the avoid list), then QT6 -- each a full cmake run apart. +# The list is not a guessing game: poppler's own CMakeLists names every one +# of them, and on this machine that file is right there -- +# +# grep -n find_soft_mandatory_package \ +# /usr/src/pkgusr/p_poppler/src/poppler-*/CMakeLists.txt +# +# Read that once when poppler moves version, instead of discovering them +# one failed configure at a time. The book's page passes ENABLE_QT5=OFF +# and nothing else, because a BLFS system is expected to have qt6. + +[webkitgtk] +# ENABLE_JOURNALD_LOG=OFF, or the build stops thousands of files in: +# wtf/Assertions.h:64: fatal error: systemd/sd-journal.h: No such file +# webkit autodetects journald and there is no systemd here. This was in the +# machine's own conf and NOT in the section shipped at 1.14.203 -- my +# omission, and it cost a nine-thousand-file build. +ENABLE_JOURNALD_LOG = OFF +# and the sandbox: bubblewrap is installed, but it also wants xdg-dbus-proxy +# and a working /proc setup; OFF is what this machine had and what it built +# with before. Turn it ON deliberately, not by inheriting a default. +ENABLE_BUBBLEWRAP_SANDBOX = OFF +USE_GTK4 = ON +USE_LIBRICE = ON +USE_WOFF2 = ON +ENABLE_WEB_RTC = ON +ENABLE_MEDIA_STREAM = ON +USE_GSTREAMER_WEBRTC = ON +jobs = 8 + +[wlroots] +werror = false diff --git a/stacks/media.stack b/stacks/media.stack new file mode 100644 index 0000000..1e95d62 --- /dev/null +++ b/stacks/media.stack @@ -0,0 +1,258 @@ +# media.stack -- codecs, gstreamer, the WebRTC chain, webkitgtk, mpv, yt-dlp. +# stack-version: 1.14.321 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +# Without it, `stack media` planned 274 packages -- systemd, polkit, qt6, +# cups -- because those exclusions lived in sway.stack alone. +include avoid + +# THE DEVELOPMENT BOOK. Codecs, gstreamer and webkitgtk move faster than a +# stable book, so the book entries below read svn: versions, dependencies, +# scripts. Dependencies OUTSIDE this file stay on the default book (built if +# missing, never upgraded to svn's page). Per entry: from-book=default. +# `packagemanager stack media` refreshes the svn book once per run. +from-book svn + +# +# Everything that decides whether a browser can PLAY and PUBLISH: audio and +# video codecs, the gstreamer plugin sets, the four WebRTC pieces from git, +# H.264 both ways, the fonts webkit needs, and mpv. +# +# packagemanager stack media --run # just this chain +# packagemanager probe # did the features land +# +# WHY IT IS SEPARATE. This is the part that gets rebuilt: a codec appears, +# a plugin set was configured without a feature, webkitgtk moves a version. +# Replaying the whole desktop to redo it is an afternoon; replaying this is +# an evening at most, and usually minutes. +# +# ORDER MATTERS, AND THIS IS NOT THE FIRST STACK. It needs, from +# wayland.stack: alsa-lib and ffmpeg (entry 1b -- the audio floor and the +# codecs), llvm and mesa, wayland, and gtk4 with pango and libsoup3. +# webkitgtk links all of them. sway.stack pulls this file in with +# `include media` at exactly the right point, so the full desktop build is +# still one command and this list exists only once. +# +# WHAT IS *NOT* HERE, deliberately: +# * ffmpeg, alsa-lib -- the audio and codec floor: too many things below +# them in wayland.stack depend on them being early. +# * x264, x265, dav1d, libaom, svt-av1, fdk-aac, lame -- they arrive as +# ffmpeg's own dependencies, so the book's plan builds them. x264enc +# (the ELEMENT) comes from gst10-plugins-ugly, which IS here. +# * mesa, gtk4, wayland -- graphics and toolkit, not media. A codec +# rebuild should not touch them. +# * pipewire, wireplumber, alsa-utils, v4l-utils -- the audio SERVER and +# the camera tools. They are in sway.stack's section 7, and rebuilding +# a codec has no reason to restart your sound daemon. + +book opus +book libvpx +# openh264: NOT in the BLFS book (checked against 13.1), so this entry is +# ours. Cisco's H.264 codec, and the only source of openh264enc/openh264dec +# in gst-plugins-bad -- which autodetects it, so it must be installed BEFORE +# -bad, and a -bad built earlier needs rebuilding to pick it up. +# +# Tag v2.6.0 and the tarball were verified; the meson path was NOT built +# here (no meson in the authoring sandbox), only read out of its meson.build, +# which does generate the .pc file -bad looks for. It also ships a plain +# Makefile (PREFIX=/usr/local by default) if meson ever disappoints. +tar openh264 \ + url=https://github.com/cisco/openh264/archive/refs/tags/v2.6.0.tar.gz \ + pkg=openh264-2.6.0.tar.gz \ + have=/usr/lib/pkgconfig/openh264.pc \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" + +book gstreamer10 +# A PLUGIN IS NOT THE SAME THING AS A LIBRARY. gst-plugins-bad always +# builds libgstwebrtc-1.0.so; the webrtcbin ELEMENT is compiled only if +# libnice was there at configure time, and its absence shows up months later +# as a browser saying "GstWebRTC plugin not found". gst-inspect-1.0 answers +# it in one call, so each entry proves its own elements. +book gst10-plugins-base expect="gst-inspect-1.0 opusenc >/dev/null 2>&1" +book gst10-plugins-good expect="gst-inspect-1.0 v4l2src >/dev/null 2>&1 && gst-inspect-1.0 vp8enc >/dev/null 2>&1" +# --------------------------------------------------------------------- +# WebRTC and camera streaming. +# +# BLFS 13.0 carries NONE of these, so they are git entries. They must be +# built BEFORE gst10-plugins-bad and webkitgtk, because webrtcbin -- the +# element a browser uses for WebRTC -- is compiled into gst-plugins-bad only +# if libnice is present, and webkitgtk links the result. Adding them later +# means recompiling both (hours). +# +# What each is for: +# libnice ICE: the NAT traversal WebRTC cannot work without +# libsrtp the encrypted media transport +# usrsctp WebRTC data channels +# webrtc-audio-processing echo cancellation for a microphone (the log's +# "webrtc-audio-processing-2 found: NO") +# +# The camera side is already in this stack: v4l-utils for the device and +# pipewire for routing it. +# +# UNTESTED -- the build lines follow each project's README, and none of these +# has been built here. Expect to adjust them; that is what the stack file is +# for. +git libnice url=https://gitlab.freedesktop.org/libnice/libnice.git ref=0.1.22 have=/usr/lib/pkgconfig/nice.pc build="meson setup build --prefix=/usr --buildtype=release -Dgtk_doc=disabled; ninja -C build" install="ninja -C build install" +git libsrtp url=https://github.com/cisco/libsrtp.git ref=v2.6.0 have=/usr/lib/pkgconfig/libsrtp2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +# ref: 1.0.0 does not exist upstream (checked: the tags are 0.9.3.0, 0.9.4.0, +# 0.9.5.0). 0.9.5.0 has meson.build, so the build line below still applies. +git usrsctp url=https://github.com/sctplab/usrsctp.git ref=0.9.5.0 have=/usr/lib/pkgconfig/usrsctp.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +git webrtc-audio-processing url=https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing.git ref=v2.1 have=/usr/lib/pkgconfig/webrtc-audio-processing-2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" + +book gst10-plugins-bad \ + expect="gst-inspect-1.0 webrtcbin >/dev/null 2>&1 && gst-inspect-1.0 dtlssrtpenc >/dev/null 2>&1 && gst-inspect-1.0 srtpenc >/dev/null 2>&1" + +# gst10-libav: the ffmpeg-backed decoders (avdec_h264, avdec_aac and the +# rest). Without it a browser and mpv fall back to whatever else can decode +# the stream, which for H.264 video is usually nothing -- and the failure is +# a black frame, not an error. +# +# AFTER ffmpeg (entry 1b) and gst10-plugins-base, which it needs, and it is +# named here rather than left to a dependency plan so that stays true. The +# probe is an element again: the package installs libgstlibav.so either way, +# and whether avdec_h264 registers depends on what ffmpeg was built with -- +# which is exactly the thing that was wrong about ffmpeg and ALSA. +book gst10-libav expect="gst-inspect-1.0 avdec_h264 >/dev/null 2>&1" + +# gst-plugins-rs: the Rust plugin set. NOT a BLFS page, so this entry is +# ours -- which means the ref and the build line are ours to keep true. +# +# Worth having here for webrtcsink/webrtcsrc (the newer WebRTC senders, used +# by anything that publishes rather than just receives) and +# gtk4paintablesink. cargo-c is already installed, as a dependency of +# librsvg. +# +# THREE THINGS TO KNOW BEFORE RUNNING IT: +# - it downloads crates at build time; there is no offline mode. +# - --library-type cdylib. Without it cargo-c also installs a STATIC +# libgstrs*.a into /usr/lib/gstreamer-1.0, which nothing loads and which +# is what the first run died on. A plugin is a shared object. +# - the plugin list is a CHOICE. Building all of gst-plugins-rs is enormous +# and most of it is irrelevant here, so five are named. Add with another +# -p; the package names are the directory names under the checkout's +# audio/, video/, net/ and generic/ -- `ls */*/Cargo.toml` lists them, and +# a wrong one fails loudly ("package(s) not found") before anything builds. +# - audiornnoise (RNNoise suppression, for a microphone in a call) is in +# gst-plugin-AUDIOFX, not a plugin of its own. Its noise model is the +# nnnoiseless crate, vendored by cargo, so there is no system library to +# install first. +# - ref=main is not a pin. Upstream tags 0.14.x for GStreamer 1.28, but +# that pairing is unverified here, so this follows main rather than +# claiming a tag that may not match. Pin it once a build succeeds: +# the ref that worked is the one to write down. +git gst-plugins-rs \ + url=https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs.git \ + ref=main \ + have=/usr/lib/gstreamer-1.0/libgstrswebrtc.so \ + build="cargo cbuild --release --library-type cdylib --prefix=/usr --libdir=/usr/lib -p gst-plugin-webrtc -p gst-plugin-rtp -p gst-plugin-dav1d -p gst-plugin-gtk4 -p gst-plugin-audiofx" \ + install="cargo cinstall --release --library-type cdylib --prefix=/usr --libdir=/usr/lib -p gst-plugin-webrtc -p gst-plugin-rtp -p gst-plugin-dav1d -p gst-plugin-gtk4 -p gst-plugin-audiofx" \ + expect="gst-inspect-1.0 webrtcsink >/dev/null 2>&1 && gst-inspect-1.0 audiornnoise >/dev/null 2>&1" + +# H.264 ENCODING, WHICH IS A DIFFERENT PACKAGE FROM DECODING. +# +# avdec_h264 (decode) comes from gst10-libav above. Nothing in this stack +# produced an ENCODER, and every camera site that publishes video wants one: +# x264enc lives in gst-plugins-UGLY, openh264enc/dec in -bad but only if +# libopenh264 was there when -bad configured. +# +# The book's own words on the switch this needs: "Without this switch, +# plugins with dependencies on (A)GPL licensed libraries are not built" -- +# so `[gst10-plugins-ugly] gpl = enabled` in machine.conf is not optional +# here, it is the difference between having x264enc and not. +book gst10-plugins-ugly expect="gst-inspect-1.0 x264enc >/dev/null 2>&1" + +# 6b. does the whole chain actually exist? (reports, builds nothing) +check webrtc + +# No exclusions. The book's dependency graph is built as the book states +# it, recommendations included: a recommendation is there for a reason, and +# skipping one is how texlive came to build before FreeType. This costs +# build time (texlive, LibreOffice and their tails are hours) and buys a +# system that matches the book. +# librice, BEFORE webkitgtk. NOT a BLFS page (the 13.1 book does not +# mention it -- checked, zero occurrences). +# +# webkitgtk 2.52 takes its ICE from librice rather than libnice when +# USE_GSTREAMER_WEBRTC=ON, and says so: +# Checking for module 'rice-proto' -- Package 'rice-proto' not found +# librice-{io,proto} is needed for USE_LIBRICE. +# +# EVERYTHING BELOW WAS CHECKED against the repository, after a first attempt +# that guessed all three and got all three wrong: +# * the repo is github.com/ystreet/librice (the author's). +# gitlab.freedesktop.org/gstreamer/librice answers 401 -- it is not +# there -- which is what the failed clone was telling us. +# * newest tag v0.4.3 (0.3.1, 0.4.0-0.4.2 also exist). Pinned, not main. +# * the workspace members are **rice-proto** and **rice-io**, not +# librice-proto/librice-io. `-p librice-proto` would have failed after +# the clone finally worked. +# * both declare [package.metadata.capi], so cargo-c is the right tool and +# the .pc files it installs are rice-proto.pc and rice-io.pc -- exactly +# the modules webkit looks for. +# +# Why not `cargo install`: that builds BINARIES into ~/.cargo/bin. What +# webkit needs is a C library with a pkg-config file, which is what +# `cargo cinstall` (cargo-c) produces -- the same tool gst-plugins-rs uses +# here. A crates.io tarball would work too, but its API is unreachable from +# where this was written, so the git tag is the verifiable source. +git librice \ + url=https://github.com/ystreet/librice.git \ + ref=v0.4.3 \ + have=/usr/lib/pkgconfig/rice-proto.pc \ + build="cargo cbuild --release --library-type cdylib --prefix=/usr --libdir=/usr/lib -p rice-proto -p rice-io" \ + install="cargo cinstall --release --library-type cdylib --prefix=/usr --libdir=/usr/lib -p rice-proto -p rice-io" \ + expect="pkg-config --exists rice-proto && pkg-config --exists rice-io" + +# woff2, BEFORE webkitgtk. The book's webkitgtk page passes +# `-D USE_WOFF2=OFF` and that is what a plain build gets -- so an icon font +# served as .woff2, which is most of them, renders as nothing. The symptom +# is a blank square where a glyph should be, with no error anywhere. +# +# woff2 IS a book page (woff2-1.0.2), so its generated script carries the +# two fixes the page documents: a `#include <cstdint>` sed for gcc-15 and +# CMAKE_POLICY_VERSION_MINIMUM=3.5 for cmake 4. brotli is its required +# dependency and is already in this tree. +# +# Then `[webkitgtk] USE_WOFF2 = ON` in machine.conf, or this changes nothing. +book woff2 + +book webkitgtk \ + expect="ldd /usr/lib/libwebkitgtk-6.0.so.4 2>/dev/null | grep -q woff2" +book mpv + + +# yt-dlp -- NOT a BLFS page; upstream's release tarball. +# +# The RELEASE ASSET, not the git tree and not the GitHub "source code" +# archive: those lack what the wheel's shared-data section names (the man +# page, README.txt, the shell completions, lazy_extractors.py), all +# generated at release time -- hatchling stops on the first one missing. +# Checked here: this tarball builds with the book's own wheel recipe and +# `yt-dlp --version` answers 2026.08.19. +# +# hatchling is the build backend (>= 1.27) and is NOT an entry here. It is +# already on a machine that has built any python module, and a `book +# hatchling` line does not mean "make sure it is there": it asked the book, +# planned a chain of newer python modules (vcs_versioning, which wants a +# newer `packaging`) and stopped before yt-dlp ever started (1.14.314). If +# it IS missing, the build stops on `BackendUnavailable: Cannot import +# 'hatchling.build'`, which lfs-helper recognises and names the page for. +# ffmpeg is +# not a build dependency but yt-dlp needs it at RUNTIME to merge separate +# video and audio streams -- it comes from wayland.stack. With yt-dlp on PATH, +# mpv plays a URL directly: mpv https://www.youtube.com/watch?v=... +# +# YOUTUBE BREAKS OLD VERSIONS WITHIN WEEKS. A tag is pinned so a replay +# builds what was tested; move it when downloads start failing: +# git ls-remote --tags https://github.com/yt-dlp/yt-dlp | tail -3 +# then edit the two version strings below and +# packagemanager stack media --only yt-dlp --run +tar yt-dlp \ + url=https://github.com/yt-dlp/yt-dlp/releases/download/2026.08.19/yt-dlp.tar.gz \ + pkg=yt-dlp-2026.08.19.tar.gz \ + have=/usr/bin/yt-dlp \ + build="pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir ." \ + install="pip3 install --no-index --find-links=dist --no-cache-dir --no-user --no-deps --force-reinstall yt-dlp" \ + expect="yt-dlp --version >/dev/null 2>&1" diff --git a/stacks/net-wifi.sh b/stacks/net-wifi.sh new file mode 100644 index 0000000..157f59d --- /dev/null +++ b/stacks/net-wifi.sh @@ -0,0 +1,213 @@ +#!/bin/bash +# net-wifi.sh -- REPORT what is still missing to bring a wireless link up. +# Builds nothing, writes nothing. Every line names a file or a binary and +# says whether it is there. +# +# This exists because "wifi does not come up" has five separate causes on a +# SysV system and the error from `ifup` names at most one of them: no DHCP +# client, no ifconfig.<iface> file, a file named for the wrong interface, no +# per-SSID wpa_supplicant config, or a SERVICE= that no installed service +# script implements. +# +# THE SERVICE NAMES ARE READ OFF THE DISK, not remembered. Which variable +# the wpa service uses to chain to a DHCP client differs between bootscript +# releases, and guessing it is how you get a link with no address. +# TWO KINDS OF FINDING, AND ONLY ONE OF THEM IS THE STACK'S FAULT. +# +# This check stopped the whole stack at 8 of 9 because no +# /etc/sysconfig/ifconfig.wlan0 existed -- a file that holds the user's SSID +# and cannot be built by anything here. A step must not fail for work it +# was never going to do: +# warn something the stack should have installed -> exit 1 +# todo something only you can write -> printed, exit 0 +fail=0 +warn() { echo "!! $*"; fail=1; } +todo() { echo "-> $*"; todos=$((todos + 1)); } +note() { echo "# $*"; } +todos=0 + +# ---- 1. the binaries ------------------------------------------------------ +for b in /sbin/wpa_supplicant /usr/sbin/wpa_supplicant; do + [ -x "$b" ] && wpa="$b" && break +done +[ -n "${wpa:-}" ] || warn "no wpa_supplicant in /sbin or /usr/sbin" +command -v dhcpcd >/dev/null 2>&1 \ + || warn "no dhcpcd: a joined link will have no address (book: dhcpcd)" +command -v ip >/dev/null 2>&1 || command -v ifconfig >/dev/null 2>&1 \ + || warn "neither ip nor ifconfig is installed" + +# ---- 2. the interfaces the kernel actually has ---------------------------- +# The name matters: ifconfig.<name> must match the interface, and predictable +# names (wlp3s0) are not the wlan0/wifi0 that examples use. +ifaces="" +for d in /sys/class/net/*; do + [ -e "$d/wireless" ] || [ -e "$d/phy80211" ] || continue + ifaces="$ifaces $(basename "$d")" +done +if [ -z "$ifaces" ]; then + todo "the kernel has no wireless interface at all" + note "that is a driver or firmware problem, not a configuration one:" + note " lfs-kernel firmware --missing" + exit 0 +fi +note "wireless interface(s):$ifaces" + +# ---- 3. the service scripts that exist ------------------------------------ +svcdir=/lib/services +[ -d "$svcdir" ] || svcdir=/usr/lib/services +if [ -d "$svcdir" ]; then + have_svc="$(ls -1 "$svcdir" 2>/dev/null | tr '\n' ' ')" + note "services installed in $svcdir: $have_svc" + case " $have_svc " in + *" wpa "*) ;; + *) warn "no '$svcdir/wpa' service: SERVICE=wpa cannot work" + note "that script comes from blfs-bootscripts (services stack)" ;; + esac + case " $have_svc " in + *" dhcpcd "*) ;; + *) warn "no '$svcdir/dhcpcd' service: nothing will fetch an address" + note " the dhcpcd PACKAGE ships no service script -- it comes" + note " from blfs-bootscripts' install-service-dhcpcd target:" + note " packagemanager stack services --only blfs-bootscripts --run" ;; + esac + # which variable the wpa service reads to chain onward -- from the script + if [ -r "$svcdir/wpa" ]; then + chain="$(grep -oE '\$\{?[A-Z_]*SERVICE[A-Z_]*' "$svcdir/wpa" \ + | tr -d '${' | sort -u | grep -v '^SERVICE$' | tr '\n' ' ')" + [ -n "$chain" ] \ + && note "the wpa service hands off using: $chain" \ + || note "the wpa service names no hand-off variable; it may start" + [ -n "$chain" ] || note "dhcpcd itself, or expect a second ifconfig file" + fi +else + warn "no $svcdir: blfs-bootscripts is not installed" +fi + +# ---- 4. the per-interface configuration ----------------------------------- +for i in $ifaces; do + f="/etc/sysconfig/ifconfig.$i" + if [ ! -r "$f" ]; then + todo "write $f -- ifup $i has nothing to read" + # THIS FILE HOLDS NO SECRET. Only the wpa_supplicant conf does, so + # there is no reason to make the reader work out the syntax: print + # the file, with the interface that exists and the hand-off variable + # this machine's wpa service actually reads. + _chain_var="$(printf '%s' "${chain:-WPA_SERVICE}" | awk '{print $1}')" + note " it has no SSID and no key in it -- this is the whole file:" + echo " cat > $f << \"EOF\"" + echo " ONBOOT=\"yes\"" + echo " IFACE=\"$i\"" + echo " SERVICE=\"wpa\"" + echo " ${_chain_var:-WPA_SERVICE}=\"dhcpcd\"" + echo " DHCP_START=\"-b\"" + echo " DHCP_STOP=\"-k\"" + echo " EOF" + note "a file for a DIFFERENT interface will not do; these exist:" + # the || never fired: the pipeline's status was sed's, not ls's, so an + # empty listing printed nothing at all where it mattered most + others="$(ls -1 /etc/sysconfig/ifconfig.* 2>/dev/null)" + if [ -n "$others" ]; then + printf '%s\n' "$others" | sed 's/^/# /' + else + note " (none at all)" + fi + continue + fi + # SERVICE must name a script that is installed + for s in $(sed -n 's/^[[:space:]]*[A-Z_]*SERVICE[A-Z_]*=["'"'"']\{0,1\}\([a-z0-9-]*\).*/\1/p' "$f"); do + [ -n "$s" ] || continue + [ -x "$svcdir/$s" ] \ + && note "$f: SERVICE $s -> $svcdir/$s" \ + || warn "$f names service '$s', which is not in $svcdir" # ours + done + grep -qE '^[[:space:]]*ONBOOT=.?yes' "$f" \ + || note "$f: ONBOOT is not yes, so boot will not bring it up" +done + +# ---- 4b. the firewall: userspace and kernel must agree -------------------- +# A legacy iptables binary on an nft-only kernel fails on every rule with +# "Module ip_tables not found", and an nft binary on a legacy-only kernel +# fails the other way. Both halves are checkable, so check both rather than +# leaving it to the first boot that needs a rule. +if command -v iptables >/dev/null 2>&1; then + _ipt="$(iptables --version 2>/dev/null)" + case "$_ipt" in + *nf_tables*) _ipt_kind=nft ;; + *legacy*) _ipt_kind=legacy ;; + *) _ipt_kind=unknown ;; + esac + note "iptables: $_ipt -> $_ipt_kind" + # what the kernel offers. /proc/net/nf_tables exists when nf_tables is + # loaded or built in; the module names are the fallback. + _k_nft=no; _k_legacy=no + { [ -e /proc/net/nf_tables ] || grep -qw nf_tables /proc/modules 2>/dev/null || [ -e "/lib/modules/$(uname -r)/kernel/net/netfilter/nf_tables.ko" ] || [ -e "/lib/modules/$(uname -r)/kernel/net/netfilter/nf_tables.ko.xz" ]; } && _k_nft=yes + { grep -qw ip_tables /proc/modules 2>/dev/null || [ -e "/lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/ip_tables.ko" ] || [ -e "/lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/ip_tables.ko.xz" ] || [ -d /proc/net/ip_tables_names ] || [ -e /proc/net/ip_tables_names ]; } && _k_legacy=yes + note "kernel: nf_tables=$_k_nft ip_tables(legacy)=$_k_legacy" + if [ "$_ipt_kind" = legacy ] && [ "$_k_legacy" = no ]; then + warn "a LEGACY iptables on a kernel with no ip_tables: every rule will fail" + note " rebuild it against nftables -- machine.conf already asks for it:" + note " [iptables]" + note " configure_args = --enable-nftables" + note " packagemanager update iptables --reinstall --regenerate --run --yes" + elif [ "$_ipt_kind" = nft ] && [ "$_k_nft" = no ]; then + warn "an nft iptables on a kernel with no nf_tables: every rule will fail" + note " add CONFIG_NF_TABLES (and NFT_COMPAT) to the kernel, or build" + note " iptables legacy-only by dropping --enable-nftables" + elif [ "$_ipt_kind" = unknown ]; then + todo "cannot tell which iptables this is: $_ipt" + else + note "iptables and the kernel agree" + fi +fi +echo + +# ---- 5. the wpa_supplicant configuration ---------------------------------- +found_conf="" +for c in /etc/sysconfig/wpa_supplicant/wpa_supplicant-*.conf \ + /etc/wpa_supplicant/wpa_supplicant*.conf /etc/wpa_supplicant.conf; do + [ -r "$c" ] && found_conf="$found_conf $c" +done +if [ -z "$found_conf" ]; then + todo "write a wpa_supplicant configuration: there is no SSID to join" + note " wpa_passphrase '<SSID>' '<passphrase>' \\" + note " > /etc/sysconfig/wpa_supplicant/wpa_supplicant-<iface>.conf" + note " chmod 600 that file" +else + for c in $found_conf; do + # A PLACEHOLDER NAME IS A TEMPLATE. wpa_supplicant-__wifi0__.conf is + # something's example, not a network you configured, and reporting it + # as a broken config sends you to fix a file you should copy. + case "$(basename "$c")" in + *__*__*|*example*|*template*|*sample*) + note "template: $c (copy it to wpa_supplicant-<iface>.conf)" + continue ;; + esac + if grep -q 'psk\|key_mgmt=NONE' "$c" 2>/dev/null; then + note "config: $c" + # only a file that HOLDS a key can leak one. Saying both "has no + # psk" and "holds a key" about the same file, two lines apart, is + # a contradiction the reader has to resolve. + case "$(stat -c %a "$c" 2>/dev/null)" in + *[4567]) todo "chmod 600 $c -- it holds a key and is world-readable" ;; + esac + else + todo "$c has no psk and no key_mgmt=NONE: nothing to join with" + fi + done +fi + +echo +if [ "$fail" != 0 ]; then + echo "# the !! lines are this stack's to fix -- it stops here so they do" + echo "# not get lost in the next package's output" +elif [ "$todos" -gt 0 ]; then + echo "# $todos thing(s) left. The interface file above has no secret in" + echo "# it -- paste it as shown. Only the wpa_supplicant config holds a" + echo "# key, which is why nothing here writes that one for you." + echo "# The stack is otherwise complete." +else + echo "# wireless: the pieces are all in place; bring it up with" + echo "# ifup $(set -- $ifaces; echo "$1")" +fi +# -> lines do not fail the step; !! lines do +exit $fail diff --git a/stacks/network.stack b/stacks/network.stack index 7666a55..275f82d 100644 --- a/stacks/network.stack +++ b/stacks/network.stack @@ -1,4 +1,11 @@ # network.stack -- wired and wireless networking, for a SysV system. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +# Without it a dependency plan walks into systemd, qt6, cups and texlive. +include avoid + # # The sway stack builds a desktop; nothing in it brings up a network. This is # what `wifi` (wpa_cli / wpa_supplicant / ifup) needs, plus a firewall. @@ -8,6 +15,15 @@ # # Every entry here is a BLFS package, checked against the book. +# THE SYSTEM ACCOUNTS THIS MACHINE NEEDS, DECLARED ONCE. +# +# A package user cannot create an account, so every book "Configuring" +# section that says `useradd` is refused and recorded -- correctly, and then +# somebody types it from the failure box hours later, from memory, with the +# uid it happened to suggest. accounts.conf is where they are written down +# instead, and this step creates them BEFORE the package that wants them. +cfg accounts + book libnl # netlink, what wpa_supplicant talks to the kernel with book wpa_supplicant # wpa_supplicant, wpa_cli, wpa_passphrase book wireless_tools # iwconfig/iwlist, if you want them @@ -21,10 +37,43 @@ book libmnl # minimal netlink library; libnftnl needs it # upstream moves -- `--check-refs` asks git remotes and cannot see a tar url. tar libnftnl url=https://www.netfilter.org/pub/libnftnl/libnftnl-1.3.2.tar.xz pkg=libnftnl-1.3.2.tar.xz have=/usr/lib/libnftnl.so build="./configure --prefix=/usr --disable-static; make" install="make install" -book iptables # nft backend, via machine.conf (see below) +# THE BOOK DISABLES NFTABLES; THIS MACHINE'S KERNEL HAS NOTHING ELSE. +# +# BLFS configures iptables with --disable-nftables, so you get the legacy +# binaries only -- and a kernel built without CONFIG_IP_NF_IPTABLES answers +# modprobe: FATAL: Module ip_tables not found +# iptables: can't initialize iptables table `filter' +# for every rule. `[iptables] configure_args = --enable-nftables` in +# machine.conf fixes it (the merge drops the book's --disable-nftables when +# it adds the opposite), and the probe makes sure: with nftables support +# `iptables --version` says (nf_tables), without it (legacy). +book iptables expect="test -x /usr/sbin/xtables-nft-multi" + +# ...AND THEN SWITCH TO IT. `--enable-nftables` builds both backends and +# upstream's install leaves the plain names on the legacy one, so +# `iptables --version` still says (legacy) after a build whose log says +# "nftables support: yes". This step points iptables/ip6tables and their +# save/restore at xtables-nft-multi, and the probe checks the result rather +# than the build. +cfg iptables-nft # nft backend, via machine.conf (see below) book net-tools # ifconfig, route, netstat +# THE ADDRESS HAS TO COME FROM SOMEWHERE. `lfs config network dhcp` writes +# no static interface files on purpose and says a DHCP client comes from BLFS +# afterwards -- and this stack did not have one, so a joined wireless link sat +# there with no address and `ifup` had no service to hand off to. +book dhcpcd # the DHCP client; SERVICE=dhcpcd in ifconfig.* + +# openssh lives HERE, not in services.stack and not anywhere near sway: it +# is networking. Its bootscript comes from blfs-bootscripts +# (install-sshd), which the services stack installs afterwards -- so sshd is +# startable once both stacks have run, and `expect` below checks only what +# this entry can be responsible for. +book openssh expect="test -x /usr/sbin/sshd" + +check net-wifi # what is still missing to bring a link up + # --------------------------------------------------------------------------- # iptables-nft # diff --git a/stacks/power.sh b/stacks/power.sh new file mode 100644 index 0000000..00f86dc --- /dev/null +++ b/stacks/power.sh @@ -0,0 +1,195 @@ +#!/bin/bash +# power.sh -- delegate shutdown, reboot and suspend to a GROUP. +# +# A PACKAGE USER OWNS FILES, NOT IDENTITY. halt, poweroff and reboot have to +# become root and are therefore root's, with a group deciding who may run +# them -- the same shape as seatd-<user> in mytools. This step does only +# what a package user may not: the group, the membership, the setuid bits, +# and one boot script. +# +# WHAT UDEV CAN AND CANNOT DO. A rule like +# SUBSYSTEM=="power", KERNEL=="state", MODE="4550", GROUP="<g>" +# never fires: /sys/power/state is a kernel attribute, not a device with a +# uevent, so udev is never asked about it. And the mode is wrong twice -- +# setuid means nothing on a sysfs file, and 550 has no write bit, so a group +# member could not write `mem` to it even if the rule had applied. The boot +# script below is what actually does the work; it is not a fallback. +set -u + +fail=0 +warn() { echo "!! $*"; fail=1; } +note() { echo "# $*"; } + +# ---- 1. the group --------------------------------------------------------- +# Named with the collector prefix, like every other group this system makes, +# and READ from the config rather than defaulted: reporting a built-in +# default as though it were the configured value is a bug this project has +# already paid for once. +grp="${POWER_GROUP:-}" +if [ -z "$grp" ]; then + # packagemanager exports the RESOLVED prefix (lfs config, its own conf, + # or the collector groups already on the system). This parsed + # packagemanager.conf as JSON -- it is key=value -- and found nothing. + pfx="${LFS_COLLECTOR_PREFIX:-}" + [ -z "$pfx" ] && pfx="$(sed -n 's/^collector_prefix=\(.*\)$/\1/p' \ + /etc/pkgusr/packagemanager.conf 2>/dev/null | head -n1)" + if [ -z "$pfx" ]; then + warn "no collector prefix: not in the environment, not in /etc/pkgusr/packagemanager.conf" + note "set it once: packagemanager config --collector-prefix <prefix>" + note "or the group: POWER_GROUP=<name> packagemanager stack services --only power --run" + exit 1 + fi + grp="${pfx%_}_power" +fi +# IN THE COLLECTOR RANGE. A system-range groupadd took gid 997, the only +# collector-prefixed group outside 90000+. packagemanager places it. +if getent group "$grp" >/dev/null 2>&1; then + note "group $grp exists" +else + packagemanager sysgroup create "$grp" >/dev/null 2>&1 || true + getent group "$grp" >/dev/null 2>&1 \ + || { warn "could not create the group $grp"; exit 1; } + note "created group $grp (gid $(getent group "$grp" | cut -d: -f3))" +fi + +# ---- 2. who is in it ------------------------------------------------------ +u="${LFS_DESKTOP_USER:-}" +[ -z "$u" ] && [ -r /etc/pkgusr/desktop-user ] && u="$(head -n1 /etc/pkgusr/desktop-user)" +if [ -n "$u" ] && id "$u" >/dev/null 2>&1; then + if id -nG "$u" 2>/dev/null | tr ' ' '\n' | grep -qx "$grp"; then + note "$u is already in $grp" + else + usermod -aG "$grp" "$u" || warn "could not add $u to $grp" + # CONFIRM YOUR OWN EFFECT: usermod has returned 0 and changed nothing + # on this project before, three times. + id -nG "$u" 2>/dev/null | tr ' ' '\n' | grep -qx "$grp" \ + && note "added $u to $grp (they must log in again for it to count)" \ + || warn "usermod said nothing, and $u is still not in $grp" + fi +else + note "no desktop user known; add members by hand: usermod -aG $grp <name>" +fi + +# ---- 3. the binaries ------------------------------------------------------ +# 4550 root:<group>: setuid root, and only group members may execute it. +# Anything not installed is skipped -- pm-utils is not in the book, and +# shutdown is a symlink in some sysvinit builds. +# A SYMLINK IS ITS TARGET. poweroff and reboot are links to halt; stat +# without -L read the link (777 p_sysvinit) and called it a failure while +# halt, the file they point to, was already right. +did=0 +seen=" " +for b in /usr/bin/halt /usr/bin/poweroff /usr/bin/reboot /usr/bin/pm-suspend \ + /usr/bin/pm-hibernate /sbin/halt /sbin/poweroff /sbin/reboot; do + [ -f "$b" ] || continue + b="$(readlink -f "$b")" || continue + case "$seen" in *" $b "*) continue ;; esac + seen="$seen$b " + case "$(stat -c '%a %U:%G' "$b" 2>/dev/null)" in + "4550 root:$grp") note "$(basename "$b") already 4550 root:$grp"; continue ;; + esac + chown "root:$grp" "$b" && chmod 4550 "$b" || { warn "could not set up $b"; continue; } + got="$(stat -c '%a %U:%G' "$b" 2>/dev/null)" + if [ "$got" = "4550 root:$grp" ]; then + note "$b -> 4550 root:$grp" + did=$((did + 1)) + else + warn "$b reads '$got', not '4550 root:$grp'" + warn " (a nosuid mount cannot hold a setuid bit: findmnt -T $b)" + fi +done +[ "$did" = 0 ] && note "no binary needed changing" + +# ---- 4. the boot script --------------------------------------------------- +# /sys and /run are made fresh on every boot, so the ownership has to be +# re-applied on every boot. ONE file, symlinked into the runlevel -- a +# second copy under rc3.d is a second place to edit, and they drift. +init=/etc/rc.d/init.d/power-perms +if [ ! -d /etc/rc.d/init.d ]; then + warn "no /etc/rc.d/init.d: the bootscripts are not installed" + note "build them first: packagemanager stack services --run" +else + cat > "$init" << EOF +#!/bin/sh +# power-perms -- give $grp the kernel's power interface, every boot. +# +# Installed by stacks/power.sh. /sys and /run do not survive a reboot, so +# neither do these modes; this is the only thing that re-applies them. +### BEGIN INIT INFO +# Provides: power-perms +# Required-Start: +# Should-Start: \$local_fs +# Required-Stop: +# Default-Start: S +# Short-Description: group access to suspend and hibernate +### END INIT INFO + +GRP="$grp" +LOCKDIR=/run/pm-utils/locks +STORAGE=/run/pm-utils/pm-suspend/storage +LOGFILE=/var/log/pm-suspend.log + +case "\$1" in + start) + for d in "\$LOCKDIR" "\$STORAGE"; do + [ -d "\$d" ] || mkdir -p "\$d" || continue + chown "root:\$GRP" "\$d" && chmod 2775 "\$d" + done + # 0660, not 0550: writing "mem" to it IS the suspend. + for f in /sys/power/state /sys/power/disk; do + [ -e "\$f" ] || continue + chown "root:\$GRP" "\$f" && chmod 0660 "\$f" + done + if [ -f "\$LOGFILE" ]; then + chgrp "\$GRP" "\$LOGFILE" && chmod 0664 "\$LOGFILE" + fi + ;; + status) + for f in /sys/power/state /sys/power/disk "\$LOCKDIR" "\$STORAGE"; do + [ -e "\$f" ] && printf '%s %s\n' "\$(stat -c '%a %U:%G' "\$f")" "\$f" + done + ;; + stop|restart|reload) ;; + *) echo "Usage: \$0 {start|status}"; exit 1 ;; +esac +exit 0 +EOF + chmod 754 "$init" + note "wrote $init" + # The sysinit runlevel, because /sys/power/state exists from the first + # moment and a suspend attempt can come before any runlevel finishes. + # LFS's bootscripts call it rcS.d (inittab: `rc S`); rcsysinit.d is + # the old name, and the only one this looked for. + linked=0 + for d in /etc/rc.d/rcS.d /etc/rc.d/rcsysinit.d; do + [ -d "$d" ] || continue + ln -sf ../init.d/power-perms "$d/S99power-perms" + note "linked $d/S99power-perms" + linked=1 + break + done + [ "$linked" = 1 ] || warn "no /etc/rc.d/rcS.d (or rcsysinit.d) to link it into" + "$init" start && note "applied now, without waiting for a reboot" +fi + +# ---- 5. rules that cannot fire ------------------------------------------- +# Removed, not explained. /sys/power/state has no uevent, and /run/bootlog +# is a file the bootscripts write, not a device: neither line was ever +# applied. power-perms does the first one's work. +for r in /etc/udev/rules.d/*.rules; do + [ -r "$r" ] || continue + grep -qE 'KERNEL=="(state|bootlog)"' "$r" 2>/dev/null || continue + sed -i -E '/KERNEL=="(state|bootlog)"/d' "$r" + note "removed the state/bootlog lines from $r (they never matched)" + grep -q '[^[:space:]]' "$r" || { rm -f "$r"; note " $r was empty -- removed"; } +done + +echo +if [ "$fail" = 0 ]; then + echo "# power: $grp may halt, reboot and suspend" + echo "# check after a reboot: $init status" + [ -n "$u" ] && echo "# $u must log out and in before the group counts" +else + echo "# fix the !! lines above" +fi +exit $fail diff --git a/stacks/services.stack b/stacks/services.stack index c2ba73c..a16e234 100644 --- a/stacks/services.stack +++ b/stacks/services.stack @@ -1,4 +1,11 @@ # services.stack -- the boot-time services a SysV desktop needs. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +# Without it a dependency plan walks into systemd, qt6, cups and texlive. +include avoid + # # /etc/init.d after an LFS build holds only LFS's own scripts; nothing starts # dbus, alsa, bluetooth or a cron daemon. Those scripts live in @@ -34,12 +41,104 @@ git bash-completion url=https://github.com/scop/bash-completion ref=2.18.0 \ build="autoreconf -i && ./configure --prefix=/usr --sysconfdir=/etc" \ install="make install" -# coloured ls, the BLFS /etc/bashrc, and `su <tab>` over the package users. -# The Bash Shell Startup Files page is CONFIG, not a package, so no package -# stack ever ran it. -cfg shell-env +# coloured ls, the BLFS /etc/bashrc, `su <tab>` over the package users, and +# $LANG. The Bash Shell Startup Files page is CONFIG, not a package, so no +# package stack ever ran it. +# +# expect= is what this step keeps true, re-checked even when the entry is +# cached (see shadow-pam). Without it a machine that ran this step before +# the locale was part of it stays at `done` for ever -- and a terminal keeps +# saying `'C' is not a UTF-8 locale`, with nothing to say why. +cfg shell-env \ + expect='test -f /etc/bashrc && test -f /etc/locale.conf && { test ! -d /etc/profile.d || test -f /etc/profile.d/i18n.sh; }' book acpid # power/lid/button events -- what a laptop needs + +# ONE GROUP CARRIES THE PRIVILEGE. halt/poweroff/reboot must become root, so +# they are root's with a group deciding who may run them, and the kernel's +# power interface is re-grouped on every boot (/sys does not survive one). +# Set POWER_GROUP to override the <collector_prefix>_power default. +# linux-pam, BEFORE the accounts that need it. +# +# `su - u_claude` without a password works through pam_exec.so reading +# /etc/pam.d/su_u_user -- which needs Linux-PAM, and LFS installs neither pam +# nor a pam-aware shadow. Without it the accounts, groups, shared +# directories and launchers all still work; only PASSWORDLESS su does not. +# +# This entry installs the LIBRARY, which on its own changes nothing: until +# shadow is rebuilt against it, nothing reads /etc/pam.d at all. That +# rebuild is deliberately NOT here: +# +# packagemanager update shadow --reinstall --regenerate --run --yes +# +# It is the one step in this whole system that can leave you unable to `su` +# or log in, so it wants a root shell open, the book's Shadow page read, and +# a moment when nothing else is running. `cfg app-users` prints the same +# thing when it finds no /etc/pam.d. +# +# It was in avoid.stack until 1.14.235 -- as systemd's dependency, which is +# not why it is here now. With `install`/`update` honouring the avoid list +# (also 1.14.234), asking for it no longer drags systemd in behind it. +book linux-pam + +# ...AND THE REBUILD THAT MAKES IT MEAN ANYTHING, immediately after it. +# +# Linux-PAM above installs a library and changes nothing: until shadow is +# rebuilt against it, nothing reads /etc/pam.d, so `su - u_claude` keeps +# asking for a password however right su_u_user is. The book's Linux-PAM +# page ends with "You should now reinstall the Shadow package" -- so this +# step does, and then puts back the setuid bits that the rebuild strips +# (the BLFS Shadow page never sets them; the LFS one does). +# +# It is idempotent and checks its own effect: pam linkage with ldd, the +# modes by reading them back. With no Linux-PAM installed it says so and +# does nothing. +# +# expect= is what this step keeps true, and it is re-checked on every run even +# when the entry is cached: shadow's configure phase or a rebuild can undo it +# later, and a cached "done" then hides it (1.14.275). +cfg shadow-pam \ + expect='test -u /usr/bin/su && test "$(stat -c %U /usr/bin/su)" = root && { test ! -d /etc/pam.d || { test -f /etc/pam.d/login && test -f /etc/pam.d/passwd && test -f /etc/pam.d/system-auth; }; }' + +# THE APPLICATION ACCOUNTS, from a file rather than a command line. +# +# Each line of app-users.conf becomes a `packagemanager user setup` call, and +# this step also does the two things no per-account command can: merge +# /etc/pam.d/su_u_user (one file listing every account that may be entered +# without a password) and write /etc/profile.d/shared-dirs.sh (one file +# exporting SHARED_<NAME> for each link in the hub, +# /home/Shared -- your ~/Shared links there). +# +# Needs a main user: `packagemanager config --main-user <your-login>`. +# Nothing of yours is edited -- no .bashrc, no .bash_profile -- and the PAM +# file is backed up before it is touched. +# +# Re-checked on every run (see shadow-pam): the accounts IT MANAGES (the +# names in app-users.conf -- not every u_*: a hand-made u_test is not its +# business) are not directly in /home, su calls su_u_user, root owns it, and +# it checks the caller, not the target. And (1.14.312) your ~/Shared is +# the link to the hub, not a directory of links: a machine set up before the +# hub existed re-runs this step once and moves them. +cfg app-users \ + expect='test -x /usr/bin/xdg-runtime-share && ! awk -F"|" "\$1 !~ /^[[:space:]]*(#|name[[:space:]]|\$)/ { gsub(/[[:space:]]/, \"\", \$1); print \"u_\" \$1 }" /etc/pkgusr/stacks/app-users.conf | while read -r u; do getent passwd "$u" | cut -d: -f6; done | grep -qE "^/home/[^/]+\$" && { test ! -f /etc/pam.d/su || grep -qE "^[[:space:]]*auth[[:space:]]+sufficient[[:space:]]+pam_exec[.]so.*su_u_user" /etc/pam.d/su; } && { test ! -e /etc/pam.d/su_u_user || { test "$(stat -c %U:%a /etc/pam.d/su_u_user)" = root:755 && ! grep -qF -e "=~ \"\$PAM_USER\"" /etc/pam.d/su_u_user; }; } && { m="$(grep "^main_user=" /etc/pkgusr/packagemanager.conf 2>/dev/null | head -n1 | cut -d= -f2)"; h="$(getent passwd "$m" 2>/dev/null | cut -d: -f6)"; [ -z "$h" ] || [ ! -d "$h/Shared" ] || [ -L "$h/Shared" ]; }' + +# WHO MAY TURN THE MACHINE OFF. A package user owns files, not identity: +# halt, poweroff and reboot have to BECOME root, so they stay root's and a +# group decides who may run them -- 4550 root:<collector>_power, with the +# desktop user in that group. The same step writes the power-perms boot +# script, because /sys/power/state is made fresh at every boot and its group +# has to be re-applied there. +# +# expect= matters more here than almost anywhere else. Without it this entry +# was cached "done" for ever, and the modes it set are exactly the kind that +# get destroyed silently: a sysvinit rebuild reinstalls halt without the +# setuid bit, and a `chown` -- from anything, including this project's own +# ownership sweep before 1.14.308 -- clears setuid as a side effect the +# kernel never reports. Then poweroff stops working for the user and nothing +# says why. Re-checked on every run: the group, the mode and owner of halt, +# and the boot script. +cfg power \ + expect='g="${POWER_GROUP:-}"; if [ -z "$g" ]; then p="$(grep "^collector_prefix=" /etc/pkgusr/packagemanager.conf 2>/dev/null | head -n1 | cut -d= -f2)"; g="${p%_}_power"; fi; h=/usr/bin/halt; [ -f "$h" ] || h=/sbin/halt; getent group "$g" >/dev/null 2>&1 && test -f /etc/rc.d/init.d/power-perms && test "$(stat -Lc "%a %U:%G" "$h" 2>/dev/null)" = "4550 root:$g"' book fcron # the cron daemon; make-ca's weekly update needs one # blfs-bootscripts: 20250225 is the FINAL release (BLFS has stopped shipping @@ -55,9 +154,30 @@ book fcron # the cron daemon; make-ca's weekly update needs # iptables restores the firewall ruleset at boot # random seeds /dev/urandom from the saved pool # service-wpa the `wpa` helper `ifup` uses for a wireless interface +# install-service-dhcpcd WAS MISSING. The dhcpcd PACKAGE installs no service +# script; /lib/services/dhcpcd comes from here, and `SERVICE=dhcpcd` (or a +# wpa hand-off to it) silently has nothing to run without it. Adding a +# package to the network stack was only half the job. tar blfs-bootscripts \ url=https://anduin.linuxfromscratch.org/BLFS/blfs-bootscripts/blfs-bootscripts-20250225.tar.xz \ pkg=blfs-bootscripts-20250225.tar.xz \ have=/etc/rc.d/init.d/dbus \ build="true" \ - install="make install-dbus install-alsa install-bluetooth install-acpid install-fcron install-iptables install-random install-service-wpa" + install="make install-dbus install-alsa install-bluetooth install-acpid install-fcron install-iptables install-random install-service-wpa install-service-dhcpcd install-sshd" + +# ...AND THE SAME LIST AS A MAPPING, so it cannot go stale again. +# +# The install= line above is a list written on one day. Add a package to a +# stack afterwards and its bootscript is simply not installed: the daemon is +# there, nothing starts it, and nothing says so. `cfg bootscripts` reads +# bootscripts.conf -- package | proof | targets | creates -- and installs the +# targets whose package IS here and whose script is NOT, then checks the disk +# rather than make's exit status. It also writes the one thing a bootscript +# cannot supply: /etc/sysconfig/ifconfig.<iface> for a WIRED interface, which +# is what actually makes dhcpcd run at boot (there is no init.d/dhcpcd in +# BLFS at all). Wireless is left to `check net-wifi`: that file needs an SSID. +# +# expect= is re-checked even when the entry is cached, so a package installed +# later is noticed on the next run. +cfg bootscripts \ + expect='! command -v dhcpcd >/dev/null 2>&1 || { { test -e /lib/services/dhcpcd || test -e /usr/lib/services/dhcpcd; } && ls /etc/sysconfig/ifconfig.* >/dev/null 2>&1; }' diff --git a/stacks/setid.conf b/stacks/setid.conf new file mode 100644 index 0000000..56b9a7a --- /dev/null +++ b/stacks/setid.conf @@ -0,0 +1,68 @@ +# setid.conf -- files that MUST carry a setuid or setgid bit. +# +# A package user cannot set these bits: chmod 4711 as p_shadow is refused by +# the kernel, and `lfs-helper setid` exists to apply the ones a build ASKED +# for and was refused. These are different -- the build never asks. The +# LFS Shadow page sets 4711 on /usr/bin/su and the BLFS page does not, so a +# BLFS-driven reinstall of shadow leaves su non-setuid and `su` stops +# working, with nothing recorded anywhere to notice it. +# +# So they are declared here, and `cfg shadow-pam` (and any install that +# touches these paths) applies them. Nothing here is guessed: each line is +# what LFS or BLFS sets on that file. +# +# path | mode | owner | group | package +# +# GROUP IS KEPT AS THE PACKAGE USER where that is how the file arrived: +# ownership follows the installer in this system, and only the USER needs to +# be root for a setuid bit to mean root. `-` keeps whatever is there. +# +# PACKAGE is the 5th column and it is a lock, not documentation (1.14.310). +# The lease hands a root-owned file to a package user for the length of a +# build phase, and "the file is setuid" used to be reason enough. It is not: +# a manifest is built by TIME WINDOW, so a webkitgtk build that happened to +# be running while `cfg shadow-pam` touched /usr/bin/su recorded su as its +# own -- and the next webkitgtk build leased it, chowned it (which clears the +# setuid bit silently) and left `su` as `p_webkitgtk 0711`. Three times. +# Now only the package named here may be handed the file. A line without a +# package is leased to nobody. + +# Shadow: su must be setuid root or nobody can become another account. +# The other four are setuid root on every distribution and in LFS. +/usr/bin/su | 4711 | root | - | shadow +/usr/bin/passwd | 4711 | root | - | shadow +/usr/bin/chage | 4711 | root | - | shadow +/usr/bin/gpasswd | 4711 | root | - | shadow +/usr/bin/newgrp | 4711 | root | - | shadow +# ...AND THE FOUR THE FIRST LIST MISSED. shadow's own install asked for +# nine setuid bits and setid.conf declared five of them; these are the rest, +# read off that request list rather than guessed. +/usr/bin/chfn | 4711 | root | - | shadow +/usr/bin/chsh | 4711 | root | - | shadow +/usr/bin/newgidmap | 4711 | root | - | shadow +/usr/bin/newuidmap | 4711 | root | - | shadow + +# Linux-PAM: its own page sets this one, and its install runs as the +# package user, so the chmod there is refused the same way. +/usr/sbin/unix_chkpwd | 4755 | root | - | linux-pam + +# D-Bus: BLFS installs the launch helper 4750 root:messagebus. Here the +# GROUP is the point -- only messagebus may run it. (This line lived only +# inside lfs-helper until 1.14.263; setid.conf is the one table now.) +/usr/libexec/dbus-daemon-launch-helper | 4750 | root | messagebus | dbus + +# NOT HERE, AND ON PURPOSE: halt, poweroff and reboot. +# +# They carry 4550 root:<collector>_power -- setuid root so they can turn the +# machine off, and executable ONLY by that group, which is what makes "the +# desktop user may reboot" a membership rather than a password. The group's +# NAME is configured (collector_prefix, or POWER_GROUP), so it cannot be +# written down here: a line saying `| root | -` would keep whatever group the +# file happens to have, and on a machine where `cfg power` has not run yet +# that is p_sysvinit -- handing shutdown to whoever builds sysvinit. +# +# `cfg power` is the one place that knows the group, and since 1.14.309 it +# carries an expect= that re-checks the mode and owner on every stack run: +# packagemanager stack services --only power --run +# A `chown` clears setuid without a word (see 1.14.308), so this is checked +# rather than assumed. diff --git a/stacks/shadow-pam.sh b/stacks/shadow-pam.sh new file mode 100644 index 0000000..a3c53df --- /dev/null +++ b/stacks/shadow-pam.sh @@ -0,0 +1,354 @@ +#!/bin/bash +# shadow-pam.sh -- make `su` actually use PAM, and put the setuid bits back. +# +# TWO THINGS THAT ONLY MAKE SENSE TOGETHER. +# +# Linux-PAM installs a library and changes nothing: until shadow is REBUILT +# against it, nothing reads /etc/pam.d at all -- so `su - u_claude` keeps +# asking for a password however right /etc/pam.d/su_u_user is. The book's +# own Linux-PAM page ends with "You should now reinstall the Shadow +# package", and that is this step. +# +# And rebuilding shadow strips the setuid bit from /usr/bin/su, because the +# BLFS page never sets it (the LFS page does). Warning about that and +# leaving it is no good: `su` is broken either way, and the fix is a mode. +# So the bits come from setid.conf and are APPLIED here. +# +# Both halves are idempotent and both check their own effect: pam linkage by +# `ldd`, the modes by reading them back. +set -u + +CONF="${SETID_CONF:-/etc/pkgusr/stacks/setid.conf}" +PM="${SHADOW_PAM_PM:-packagemanager}" +SU="${SHADOW_PAM_SU:-/usr/bin/su}" +fail=0 +note() { echo "# $*"; } +warn() { echo "!! $*"; fail=1; } + +# Defined before BOTH steps use them: step 1 asked _pam_installed forty +# lines above where it was defined, which under `set -u` is a command that +# does not exist yet -- the config would silently never be written. +_su_has_pam() { ldd "$SU" 2>/dev/null | grep -q 'libpam\.so'; } +_pam_installed() { [ -e /usr/lib/libpam.so ] || [ -e /lib/libpam.so ]; } + +# CONFIG BEFORE REBUILD, AND THAT ORDER IS THE WHOLE POINT. +# +# The first version rebuilt shadow and THEN wrote /etc/pam.d. Between the +# two, su links libpam and has no service file -- and every build in this +# system runs as its package user through `su - p_<pkg>`, so that window is +# one where NOTHING can be built, including the very configure phases that +# would write the missing files. Root's own `su` survives it only because +# pam_rootok is sufficient. +# +# So: write the config while su still works, then rebuild. A system that +# stops midway is then left with pam config and a non-pam su -- harmless -- +# rather than a pam su and no config. + +# ---- 1. the pam config files su will read -------------------------- +# A PAM-LINKED su WITH NO /etc/pam.d/su IS WORSE THAN A NON-PAM ONE: +# +# su: pam_start: error 26 (PAM_ABORT -- no service file) +# +# `update` runs install, not configure, and those files are written by the +# CONFIGURE phase of linux-pam (/etc/pam.d/system-auth and friends) and +# shadow (/etc/pam.d/su, login, passwd, chage...). So a rebuild leaves su +# linked against a library with nothing to read. +# THE SET THE BOOK WRITES, NOT A SAMPLE OF IT. +# +# This checked su and system-auth. Both existed, so it said "in place" +# while login, passwd, chage and twelve admin tools had no file: +# passwd: pam_start() failed, error 26 +# login: PAM Failure, aborting +# Two files out of sixteen. The set now comes from the same place the files +# do: the package's configure_pkg, generated from the book. Every +# /etc/pam.d/<n> it names is expected, and a `for VAR in a b c` loop that +# writes /etc/pam.d/${VAR} contributes a, b and c. Commented (skipped) +# lines are not the book's commands and are not counted. +PAM_PKGS="${SHADOW_PAM_PKGS:-linux-pam shadow}" +PAM_DIR="${SHADOW_PAM_DIR:-/etc/pam.d}" + +HELPER="${SHADOW_PAM_HELPER:-lfs-helper}" +# ONE READER OF WHAT A SCRIPT WRITES. 1.14.262 had its own awk here; the +# lease in lfs-helper (1.14.263) needs the same list for every package, so +# this asks it. Names under /etc/pam.d, one per line. +_book_pam_set() { # $1 = generated install script + $HELPER script-targets "$1" configure_pkg 2>/dev/null \ + | sed -n 's|^/etc/pam\.d/||p' | sort -u +} + +# THE SCRIPT ON DISK MAY BE THE PROBLEM. Shadow's was generated before +# blfs read sub-sections, so its configure phase is prose and a bare `:` -- +# it names no pam file, and `script configure` would run it again and write +# nothing. So: read the script that is there; if it names no pam file, +# regenerate it (an edited copy is kept as .edited) and read that. A script +# that is already right is never regenerated just because this step ran. +_fetch_script() { # $1 = package, $2 = optional --regenerate -> path + $PM script fetch "$1" ${2:-} 2>/dev/null \ + | sed 's/\x1b\[[0-9;]*m//g' \ + | sed -n 's/^script: //p; s/^already there: //p' | tail -1 +} + +_missing_pam="" +_want_pam="" +_scripts="" +if _pam_installed; then + for _cfgpkg in $PAM_PKGS; do + _sp_path="$(_fetch_script "$_cfgpkg")" + _set="" + [ -f "$_sp_path" ] && _set="$(_book_pam_set "$_sp_path" | tr '\n' ' ')" + if [ -z "$(printf '%s' "$_set" | tr -d ' ')" ]; then + note "$_cfgpkg's script names no $PAM_DIR file -- regenerating it from the book" + _sp_path="$(_fetch_script "$_cfgpkg" --regenerate)" + [ -f "$_sp_path" ] && _set="$(_book_pam_set "$_sp_path" | tr '\n' ' ')" + fi + if [ -z "$_sp_path" ] || [ ! -f "$_sp_path" ]; then + warn "could not get $_cfgpkg's install script -- cannot tell what it writes" + note " by hand: packagemanager script fetch $_cfgpkg --regenerate" + continue + fi + _scripts="$_scripts $_cfgpkg=$_sp_path" + if [ -z "$(printf '%s' "$_set" | tr -d ' ')" ]; then + # a pam package whose configure names no pam file is the bug + # this check exists for, not an empty requirement + warn "$_cfgpkg's configure phase writes no $PAM_DIR file" + note " the generator dropped them -- look at configure_pkg in $_sp_path" + continue + fi + _want_pam="$_want_pam $_set" + for _n in $_set; do + [ -f "$PAM_DIR/$_n" ] || _missing_pam="$_missing_pam $_n" + done + done +fi + +if _pam_installed && [ -n "$_want_pam" ]; then + if [ -z "$_missing_pam" ]; then + note "all $(echo $_want_pam | wc -w) $PAM_DIR files the book writes are in place" + else + note "missing from $PAM_DIR:$_missing_pam" + note " running the CONFIGURE phases that write them (update installs;" + note " it does not configure)" + # THE su TRAP. Shadow's page writes $PAM_DIR/su too, WITHOUT the + # pam_exec line that makes application users passwordless -- so this + # step silently undoes `cfg app-users`. app-users is the next entry + # in services.stack and puts it back (the suite asserts that order); + # say so here, so `su - u_<n>` asking for a password is not a mystery. + _had_exec=0 + grep -q 'pam_exec\.so' "$PAM_DIR/su" 2>/dev/null && _had_exec=1 + for _cfgpkg in $PAM_PKGS; do + case "$_scripts" in *" $_cfgpkg="*) ;; *) continue ;; esac + if $PM script configure "$_cfgpkg" >/dev/null 2>&1; then + note " configured $_cfgpkg" + else + warn "could not run $_cfgpkg's configure phase" + note " by hand: packagemanager script configure $_cfgpkg" + fi + done + # CONFIRM IT -- every file, by name. + _still="" + for _n in $_want_pam; do + [ -f "$PAM_DIR/$_n" ] || _still="$_still $_n" + done + if [ -z "$_still" ]; then + note " all $(echo $_want_pam | wc -w) written" + else + warn "still missing from $PAM_DIR:$_still" + note " a missing service file is \"pam_start: error 26\" for that program" + fi + if [ "$_had_exec" = 1 ] && ! grep -q 'pam_exec\.so' "$PAM_DIR/su" 2>/dev/null; then + note " shadow's configure rewrote $PAM_DIR/su without the pam_exec line:" + note " application users need a password until app-users runs again." + note " services.stack runs it next; on its own:" + note " packagemanager stack services --only app-users --run" + fi + fi +elif _pam_installed; then + warn "could not work out which $PAM_DIR files the book writes -- nothing checked" +fi +echo + +# ---- 2. is su PAM-aware? ------------------------------------------------- + +if ! _pam_installed; then + note "Linux-PAM is not installed -- nothing to rebuild shadow against." + note " passwordless su needs it: packagemanager stack services --only linux-pam --run" +elif _su_has_pam; then + note "$SU already links libpam -- no rebuild needed" +else + note "$SU does not link libpam: shadow was built before PAM existed." + note " rebuilding it now (this is the step the book's Linux-PAM page asks for)" + # REGENERATE, because the script on disk was written before pam was + # here and its configure line has no --with-libpam to gain. + if $PM update shadow --reinstall --regenerate --run --yes; then + if _su_has_pam; then + note " $SU now links libpam" + else + warn "shadow rebuilt and $SU still does not link libpam" + note " check the configure line: grep -n configure \\" + note " /usr/src/pkgusr/p_shadow/install_Shadow-*" + note " the book's page may need --with-libpam for this system." + fi + else + warn "rebuilding shadow failed -- su is unchanged, which is the safe outcome" + note " logs: packagemanager errors" + fi +fi +echo + +# ---- 3. the setuid bits, applied ---------------------------------------- +# A rebuild of shadow resets these every time, so this runs whether or not +# the rebuild above did anything. +[ -r "$CONF" ] || { warn "no $CONF -- cannot restore the setuid bits"; exit 1; } + +applied=0 checked=0 +while IFS='|' read -r path mode owner group; do + path="$(printf '%s' "${path:-}" | tr -d '[:space:]')" + case "$path" in ""|\#*) continue ;; esac + mode="$(printf '%s' "${mode:-}" | tr -d '[:space:]')" + owner="$(printf '%s' "${owner:-}" | tr -d '[:space:]')" + group="$(printf '%s' "${group:-}" | tr -d '[:space:]')" + [ -n "$mode" ] || { warn "$path: no mode in $CONF"; continue; } + if [ ! -e "$path" ]; then + # NOT AN ERROR. Not every package here installs every file -- + # unix_chkpwd only exists once Linux-PAM is in. + note "$path is not installed -- skipped" + continue + fi + checked=$((checked + 1)) + now="$(stat -c '%a %U %G' "$path" 2>/dev/null)" + want_owner="${owner:--}" + [ "$want_owner" = "-" ] && want_owner="$(printf '%s' "$now" | awk '{print $2}')" + want_group="${group:--}" + [ "$want_group" = "-" ] && want_group="$(printf '%s' "$now" | awk '{print $3}')" + if [ "$now" = "$mode $want_owner $want_group" ]; then + continue + fi + note "$path is $now, should be $mode $want_owner $want_group" + # ORDER MATTERS: chown CLEARS the setuid bit, so chmod goes last. + chown "$want_owner:$want_group" "$path" 2>/dev/null \ + || warn " could not chown $path" + if chmod "$mode" "$path" 2>/dev/null; then + # CONFIRM IT. A setuid bit that did not take is the whole failure + # this step exists to end. + after="$(stat -c '%a %U %G' "$path" 2>/dev/null)" + if [ "$after" = "$mode $want_owner $want_group" ]; then + note " -> $after" + applied=$((applied + 1)) + else + warn " $path is $after after chmod $mode -- not what was asked" + fi + else + warn " could not chmod $mode $path" + fi +done < "$CONF" + +echo + +# ---- 4. accounts pam can actually look up -------------------------------- +# EVERY BUILD IS A `su - p_<pkg>`, so this line is not cosmetic: +# +# su: Authentication service cannot retrieve authentication info +# (Ignored) +# +# pam_unix reads /etc/shadow. `add_package_user` writes /etc/passwd and +# leaves no shadow entry, so pam_unix cannot find the account and says so -- +# on every package build, in every log, for ever. It is "(Ignored)" only +# because pam_rootok already succeeded for root; for any non-root caller it +# is a real failure. +# +# `pwconv` is shadow's own tool for exactly this: it adds a shadow entry for +# every passwd entry that lacks one, with the password field locked. It +# rewrites /etc/shadow, so: back it up, run it, read it back. +# +# UNVERIFIED FROM WHERE THIS WAS WRITTEN: the detection, the backup and the +# read-back were exercised against a fixture; `pwconv` itself was not (it +# edits the real /etc/shadow and there is no safe way to run it in that +# sandbox). If it turns out not to cover these accounts, the read-back +# below says so by name and the backup is beside the file -- which is why +# both are here. +_no_shadow() { + # accounts in passwd with no /etc/shadow line + while IFS=: read -r _n _rest; do + case "$_n" in ""|\#*) continue ;; esac + grep -q "^$_n:" /etc/shadow 2>/dev/null || printf '%s\n' "$_n" + done < /etc/passwd +} + +# NO /etc/shadow AT ALL IS THE CASE THIS MISSED. +# +# The guard was `[ -r /etc/passwd ] && [ -r /etc/shadow ]`, so on a system +# with no shadow file it skipped in silence -- which is exactly the system +# that needs it most: pam_unix has nowhere to look for ANY account, and +# every `su - p_<pkg>` says "cannot retrieve authentication info". +# +# pwconv CREATES /etc/shadow from /etc/passwd, moving each password hash out +# of passwd and leaving `x` behind. That is a change to how this machine +# authenticates, so: +# * /etc/passwd is backed up FIRST (it is the file that changes), +# * root's own line is checked afterwards, by name, +# * and if anything is wrong, passwd is put back and shadow removed. +if [ ! -r /etc/passwd ]; then + warn "cannot read /etc/passwd -- skipping the shadow check" +elif [ ! -e /etc/shadow ]; then + note "there is no /etc/shadow at all: pam_unix cannot look up ANY account," + note " which is the \"cannot retrieve authentication info\" on every build." + note " pwconv creates it from /etc/passwd (hashes move; passwd keeps 'x')." + _pwbak="/etc/passwd.bak-$(date +%Y%m%d-%H%M%S)" + if ! cp -p /etc/passwd "$_pwbak"; then + warn "could not back up /etc/passwd -- not running pwconv" + else + note " backed up /etc/passwd -> $(basename "$_pwbak")" + if pwconv && [ -s /etc/shadow ] && grep -q '^root:' /etc/shadow \ + && grep -q '^root:' /etc/passwd; then + chmod 600 /etc/shadow 2>/dev/null + note " /etc/shadow created, root is in it, $(grep -c . /etc/shadow) entries" + note " VERIFY BEFORE LOGGING OUT, from another terminal:" + note " su - $(awk -F: '$3>=1000 && $3<60000 {print $1; exit}' /etc/passwd) -c id" + note " if login is broken: cp -p $_pwbak /etc/passwd && rm -f /etc/shadow" + else + warn "pwconv did not produce a usable /etc/shadow -- putting" + warn " /etc/passwd back and removing the half-made file" + cp -p "$_pwbak" /etc/passwd + rm -f /etc/shadow + note " nothing changed; the backup is still at $_pwbak" + fi + fi +elif [ -r /etc/shadow ]; then + _missing_sh="$(_no_shadow | tr '\n' ' ')" + if [ -z "$(printf '%s' "$_missing_sh" | tr -d ' ')" ]; then + note "every account has an /etc/shadow entry (pam_unix can read them)" + else + note "accounts with no /etc/shadow entry:$_missing_sh" + note " pam_unix cannot look those up -- that is the" + note " \"cannot retrieve authentication info\" line on every build." + _shbak="/etc/shadow.bak-$(date +%Y%m%d-%H%M%S)" + if cp -p /etc/shadow "$_shbak"; then + note " backed up /etc/shadow -> $(basename "$_shbak")" + if pwconv; then + _still="$(_no_shadow | tr '\n' ' ')" + if [ -z "$(printf '%s' "$_still" | tr -d ' ')" ]; then + note " pwconv added the missing entries (passwords locked)" + else + warn "still without a shadow entry:$_still" + note " pwconv ran and did not cover them; look at /etc/passwd" + fi + else + warn "pwconv failed -- /etc/shadow is unchanged" + note " the backup is $_shbak if anything looks wrong" + fi + else + warn "could not back up /etc/shadow -- not running pwconv" + fi + fi +fi + +echo +if [ "$fail" = 0 ]; then + echo "# shadow-pam: $checked file(s) checked, $applied corrected." + echo "# try it, in ANOTHER terminal, before closing this one:" + echo "# su - <an application user> -c id" + echo "# if su is broken, this is the file to look at: /etc/pam.d/su" +else + echo "# the !! lines above need attention" +fi +exit $fail diff --git a/stacks/shell-env.sh b/stacks/shell-env.sh index c254b33..1d736be 100755 --- a/stacks/shell-env.sh +++ b/stacks/shell-env.sh @@ -1,6 +1,6 @@ #!/bin/bash -# shell-env.sh -- interactive shell environment: coloured ls, bash -# completion, and username completion for su (root step, idempotent). +# shell-env.sh -- interactive shell environment: the system locale, coloured +# ls, bash completion, and username completion for su (root step, idempotent). # # BLFS's "The Bash Shell Startup Files" page writes /etc/profile and # /etc/bashrc, and /etc/bashrc is what makes ls colourful -- it evals @@ -11,10 +11,23 @@ # is a `tar` entry in the stack beside this script. set -u +# ROOT WROTE IT, SO ROOT OWNS IT. +# +# On a package-user system a file's owner records which package installed it, +# and anything written while a build window is open is attributed to that +# package user -- /etc/profile.d/pkgusr-system.sh owned by p_gstreamer10, +# 00-path-functions.sh by p_gettext, /etc/profile.d itself by p_libnice. +# For a file a PACKAGE installed that is correct and useful. For a config +# file root writes it is wrong twice over: it says a package owns something +# no package installed, and every login shell sources it, so whoever can +# write it can run code as everyone who logs in. Re-asserted on every run. +own_root() { [ -e "$1" ] && chown root:root "$1" 2>/dev/null; return 0; } + # 1. /etc/dircolors -- the database ls colours come from if [ ! -f /etc/dircolors ]; then if command -v dircolors >/dev/null 2>&1; then dircolors -p > /etc/dircolors + own_root /etc/dircolors echo "# wrote /etc/dircolors (from dircolors -p)" fi else @@ -69,6 +82,7 @@ complete -A user su sudo passwd chage groups id # End /etc/bashrc BASHRC chmod 644 "$_dest" +own_root "$_dest" echo "# wrote $_dest" # 3. make sure /etc/profile actually sources it, and profile.d @@ -85,4 +99,141 @@ if [ -f /etc/pkgusr/bashrc ] && ! grep -q '/etc/bashrc' /etc/pkgusr/bashrc; then echo "# package users now source /etc/bashrc too" fi + +# 5. THE SYSTEM LOCALE -- $LANG, and why a terminal says the locale is "C" +# +# warning: foot: 'C' is not a UTF-8 locale, falling back to 'C.UTF-8' +# +# Nothing is compiled wrong when that appears: LANG is simply not in the +# environment. LFS's own /etc/profile reads /etc/locale.conf, but BLFS's +# "The Bash Shell Startup Files" page REPLACES /etc/profile with one that +# reads /etc/profile.d/*.sh instead -- and its i18n.sh, the piece that +# exports LANG, is a separate block on that page that is easy to walk past. +# Take either route: this writes both files, and neither profile minds the +# other's. +# +# Without it every program falls back to the C locale: sorting, dates, +# translated messages, and anything that has to decide whether a byte is a +# character. +LOCALE_CONF="${SHELL_ENV_LOCALE_CONF:-/etc/locale.conf}" # overridable, for tests +I18N_SH="${PROFILE_D:-/etc/profile.d}/i18n.sh" +LFS_CONFIG_JSON="${LFS_STORE:-/usr/share/lfs}/config.json" +_norm() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | sed 's/utf-8/utf8/; s/-//g'; } +_have_locale() { + [ -n "$1" ] || return 1 + locale -a 2>/dev/null | while read -r l; do + [ "$(_norm "$l")" = "$(_norm "$1")" ] && echo yes + done | grep -q yes +} + +# WHAT THIS MACHINE ALREADY SAYS IT WANTS, in order. Nothing is guessed +# from the keyboard or the timezone: a wrong locale is worse than none. +conf_lang="$(sed -n 's/^[[:space:]]*LANG=//p' "$LOCALE_CONF" 2>/dev/null \ + | tr -d '"'"'"' ' | head -n1)" +asked="${LFS_LOCALE:-$conf_lang}" +[ -n "$asked" ] || asked="$(sed -n 's/.*"locale"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ + "$LFS_CONFIG_JSON" 2>/dev/null | head -n1)" +# A NAME IS NOT A LOCALE UNTIL glibc HAS BUILT IT. locale.conf may well name +# one nobody ever generated, and then LANG is set to something that does not +# exist -- which looks exactly like not setting it at all. +want="" +for _c in "$asked" en_US.UTF-8 C.UTF-8; do + if _have_locale "$_c"; then want="$_c"; break; fi +done +if [ -n "$asked" ] && [ "$(_norm "$asked")" != "$(_norm "${want:-}")" ]; then + echo "!! $asked is asked for but not installed (see: locale -a)" + echo "# make it as glibc's package user: localedef -i ${asked%%.*} -f UTF-8 $asked" + [ -n "$want" ] && echo "# using $want until then" +fi + +if [ -z "$want" ]; then + echo "!! no UTF-8 locale is installed -- \$LANG cannot be set to one" + echo "# glibc builds them; the archive is glibc's, so make one as its" + echo "# package user (p_glibc here) rather than as root:" + echo "# localedef -i en_US -f UTF-8 en_US.UTF-8" + echo "# then run this step again." +else + case "$(_norm "$want")" in + *utf8) ;; + *) echo "!! $want is not a UTF-8 locale -- a terminal will still complain" ;; + esac + if [ -f "$LOCALE_CONF" ] && [ "$(_norm "$conf_lang")" = "$(_norm "$want")" ]; then + echo "# $LOCALE_CONF already says LANG=$conf_lang" + elif [ -f "$LOCALE_CONF" ]; then + echo "# $LOCALE_CONF says LANG=${conf_lang:-nothing} -- left alone (it is yours)" + echo "# $want is what works here; change it there if you want it" + else + printf 'LANG=%s\n' "$want" > "$LOCALE_CONF" + chmod 644 "$LOCALE_CONF"; own_root "$LOCALE_CONF" + echo "# wrote $LOCALE_CONF (LANG=$want)" + fi + # i18n.sh: JUDGED BY WHAT IT PRODUCES, NOT BY WHO WROTE IT. + # + # BLFS's page ships an i18n.sh of its own, and it is usually already + # there -- `. /etc/locale.conf` and export what came out. "Exists and is + # not ours, left alone" would have been the end of it, and $LANG would + # still be empty: the file was never the problem, the missing + # /etc/locale.conf above was. So source it in an empty environment and + # look at the LANG that comes out. TERM is set to something that is not + # `linux`, because both the LFS and BLFS versions special-case the + # console with a hard-coded C.UTF-8. + _i18n_gives() { + env -i HOME=/root TERM=xterm bash --noprofile --norc -c \ + ". \"\$1\" >/dev/null 2>&1; printf '%s' \"\${LANG:-}\"" _ "$1" 2>/dev/null + } + # WHOSE FILE IS IT. Three writers put an i18n.sh here and all three can + # leave LANG unset, so all three are ours to replace: + # this script -- its own marker + # the book -- `Begin /etc/profile.d/i18n.sh`, placeholder + # LANG that was never filled in + # lfs-helper -- its marker, and before 1.14.298 an i18n.sh + # with the export COMMENTED OUT: + # : # export LANG=en_US.UTF-8 + # which is what is on a machine built with it + _ours_i18n() { + grep -q '^# written by stacks/shell-env.sh' "$1" \ + || grep -q '^# written by lfs-helper' "$1" \ + || grep -q 'Begin /etc/profile.d/i18n.sh' "$1" \ + || grep -q '^[[:space:]]*:[[:space:]]*#[[:space:]]*export LANG=' "$1" + } + got="" + [ -f "$I18N_SH" ] && got="$(_i18n_gives "$I18N_SH")" + if [ -f "$I18N_SH" ] && [ "$(_norm "$got")" = "$(_norm "$want")" ]; then + echo "# $I18N_SH already gives LANG=$got -- left alone" + elif [ -f "$I18N_SH" ] && ! _ours_i18n "$I18N_SH"; then + # not ours, and not the book's either: somebody wrote this on purpose + echo "# $I18N_SH is hand-written and gives LANG=${got:-nothing} -- left alone" + echo "# $want is what this machine has; set it there, or delete the file" + else + if [ -f "$I18N_SH" ]; then + bak="$I18N_SH.bak-$(date +%Y%m%d-%H%M%S)" + cp -p "$I18N_SH" "$bak" \ + && echo "# $I18N_SH gave LANG=${got:-nothing} -- replaced (backup: $(basename "$bak"))" + fi + cat > "$I18N_SH" <<'I18N' +# written by stacks/shell-env.sh -- safe to delete, regenerated +# Begin /etc/profile.d/i18n.sh +# +# $LANG for every login shell, from /etc/locale.conf. BLFS's /etc/profile +# reads this directory; LFS's reads /etc/locale.conf itself. Either way the +# locale comes from one file. +if [ -r /etc/locale.conf ]; then + . /etc/locale.conf + export LANG + if [ -n "${LC_ALL:-}" ]; then export LC_ALL; fi +fi +# less(1) needs telling separately -- it reads this, not the locale. +export LESSCHARSET="${LESSCHARSET:-utf-8}" +# End /etc/profile.d/i18n.sh +: +I18N + chmod 644 "$I18N_SH"; own_root "$I18N_SH" + echo "# wrote $I18N_SH (exports LANG from $LOCALE_CONF)" + fi + if [ "$(_norm "${LANG:-C}")" != "$(_norm "$want")" ]; then + echo "# this shell still has LANG=${LANG:-unset}: /etc/profile.d is read at" + echo "# LOGIN, so log in again -- or now: . $I18N_SH" + fi +fi + echo "# done -- open a new shell, or: . /etc/bashrc" diff --git a/stacks/sway-session.sh b/stacks/sway-session.sh index 189420e..e54478d 100755 --- a/stacks/sway-session.sh +++ b/stacks/sway-session.sh @@ -11,7 +11,7 @@ HOME_USER="${LFS_DESKTOP_USER:-${LFS_HOME_USER:-}}" [ -n "$HOME_USER" ] && printf '%s\n' "$HOME_USER" > /etc/pkgusr/desktop-user HOME_DIR="/home/${HOME_USER:-nobody-set}" -# 1. one launcher: a dbus session around sway. +# 1. one launcher: the checks sway needs, then sway. # # The seat comes from seatd, not elogind: elogind only registers a session # through pam_elogind, and Linux-PAM is not on this system, so a tty login @@ -19,7 +19,8 @@ HOME_DIR="/home/${HOME_USER:-nobody-set}" # means libseat does not try logind first and fail with a puzzling message. cat > /usr/bin/sway-session <<'LAUNCH' #!/bin/bash -# start sway inside a dbus session; run from a plain tty login +# start sway from a plain tty login. The session bus and XDG_RUNTIME_DIR +# are NOT set up here -- that is yours to arrange. export XDG_CURRENT_DESKTOP=sway export LIBSEAT_BACKEND=seatd @@ -64,11 +65,10 @@ if [ -n "${XDG_VTNR:-}" ] && [ -r /sys/class/tty/tty0/active ]; then fi fi -if [ -z "${XDG_RUNTIME_DIR:-}" ]; then - export XDG_RUNTIME_DIR="/run/user/$(id -u)" - mkdir -p "$XDG_RUNTIME_DIR" 2>/dev/null && chmod 700 "$XDG_RUNTIME_DIR" -fi -exec dbus-run-session -- sway "$@" +# THE SESSION BUS AND XDG_RUNTIME_DIR ARE DELIBERATELY NOT HANDLED HERE. +# They are yours; this launcher only checks the seat, udev and the VT, and +# then starts the compositor with whatever environment you gave it. +exec sway "$@" LAUNCH chmod 755 /usr/bin/sway-session @@ -78,7 +78,7 @@ cat > /usr/bin/pipewire-start <<'PW' # start pipewire + wireplumber for this session, if not already up pgrep -x pipewire >/dev/null || { pipewire & disown; } pgrep -x wireplumber >/dev/null || { wireplumber & disown; } -echo "# pipewire is up (pipewire-pulse socket in $XDG_RUNTIME_DIR)" +echo "# pipewire is up (its sockets go in \${XDG_RUNTIME_DIR:-<unset>})" PW chmod 755 /usr/bin/pipewire-start diff --git a/stacks/sway.stack b/stacks/sway.stack index 2e4a652..fdb5d5a 100644 --- a/stacks/sway.stack +++ b/stacks/sway.stack @@ -1,295 +1,193 @@ -# sway.stack -- a working Sway desktop with a minimal WebKit browser. -# stack-version: 1.13.67 +# sway.stack -- the Sway desktop: the compositor, its tools, your programs. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +include avoid + # Replay with: packagemanager stack sway --run # Entries run top to bottom; book deps are resolved from the BLFS book, # installed entries are skipped, the first failure stops the run. # git refs are pinned to releases -- true them up if a clone fails. - -# Never built by this stack, however they are reached. harfbuzz RECOMMENDS -# texlive and LibreOffice; a sway desktop with a webkit browser wants -# neither, and texlive alone is hours of build time and a /opt tree. -# NOT BUILT BY THIS STACK. The book's recommendations are followed -- -# a recommendation the build actually needs (curl for cmake, icu for -# libxml2) must not be dropped by a blanket rule. These are the branches -# this desktop does not want, named one at a time, and everything they alone -# pull in goes with them. -avoid libreoffice -avoid asymptote -avoid biber -avoid qt6 -avoid okular -avoid phonon -avoid phonon-backend-vlc -avoid vlc -avoid cups -avoid openjdk -avoid java -avoid apache-ant -avoid postgresql -avoid openldap -avoid unixodbc -avoid graphviz -avoid gegl -# the KDE leaves okular pulls in: named individually, because a package can -# have several parents and pruning by one of them drops shared dependencies -# (graphite2 was lost that way) -avoid plasma-activities -avoid libkexiv2 -avoid kwindowsystem -avoid libfm-qt -avoid xdg-desktop-portal-lxqt -avoid xdg-desktop-portal-gnome -avoid gnome-desktop - -# The desktop-portal tail. The book makes xdg-desktop-portal REQUIRE all -# four backends -- gtk, wlr, gnome AND lxqt -- so one recommendation from -# gtk4 drags in the whole of xfce and lxqt, gnome-desktop and libfm-qt. -# Sway needs none of it to run; screen sharing wants xdg-desktop-portal-wlr, -# which can be built on its own later: -# packagemanager install xdg-desktop-portal-wlr --run -avoid xdg-desktop-portal -avoid xdg-desktop-portal-gtk -avoid xdg-desktop-portal-wlr -avoid xdg-desktop-portal-gnome -avoid xdg-desktop-portal-lxqt -avoid xfce4-panel -avoid xfce4-notifyd -avoid xfce4-dev-tools -avoid xfconf -avoid libxfce4util -avoid libxfce4ui -avoid libxfce4windowing -avoid exo -avoid garcon -avoid liblxqt -avoid lxqt-build-tools -avoid lxqt-notificationd -avoid lxqt-kwindowsystem -avoid libqtxdg -avoid polkit-qt -avoid libfm-qt -avoid menu-cache -avoid libwnck -avoid notification-daemon - -# Not this desktop either: a systemd package on a SysV system, phone modem -# stacks, a mail/ssh server, spell-checking dictionaries. -avoid systemd -avoid ModemManager -avoid libmbim -avoid libqmi -avoid geoclue2 -avoid openssh -avoid ruby -avoid aspell -avoid enchant -avoid gnome-keyring -avoid gcr -avoid startup-notification - -# Tails of things already avoided. Each of these is in the plan ONLY to -# serve a package this stack does not build, and the tools cannot yet work -# that out for themselves (see 1.12.83: pruning by the single parent that -# `blfs order` reports dropped shared dependencies, and broke harfbuzz). -# Checked one by one against the plan's parent column: -avoid libcupsfilters # cups (avoided) -avoid libppd # cups -avoid cups-filters # cups -avoid libkexiv2 # okular (avoided) -avoid exiv2 # libkexiv2 -avoid clucene # libreoffice (avoided) -avoid redland # libreoffice -avoid rasqal # redland -avoid raptor # rasqal -avoid jasper # qt6 (avoided) -avoid libmng # qt6 -avoid double-conversion # qt6 -avoid poppler # libcupsfilters -- its ONLY parent, avoided -avoid mupdf # libcupsfilters -avoid gs # libcupsfilters -avoid qpdf # libcupsfilters -avoid nss # poppler -avoid boost # poppler -avoid pdfio # libcupsfilters -avoid gpgmepp # poppler -# Checked against EVERY parent, not just the one the plan prints: these have -# no parent left in the plan at all. -avoid colord # cups (avoided) -- no live parent -avoid linux-pam # systemd (avoided) -- no live parent -avoid xdg-utils # cups -- no live parent -avoid freeglut # mupdf (avoided) -- no live parent -avoid polkit # colord + systemd (both avoided) -- no live parent -avoid libgusb # colord -# NOT avoided, though the plan labels them with an avoided parent: the label -# shows only the FIRST parent. gtk3 is really wanted by adwaita-icon-theme, -# ffmpeg by mpv, polkit by colord... and pulseaudio by pipewire/sdl3/gst. -# (rust and llvm STAY: glycin genuinely requires rust, and llvm is a stack -# entry in its own right for mesa.) - -# 0. verify the kernel (DRM/amdgpu, evdev, snd, seccomp, user ns, render node) -check kernel-sway - -# 1. seat and session (SysV: elogind is the logind; seatd is not in the book) -book dbus -book elogind url=https://github.com/elogind/elogind ref=v255.17 prog=loginctl build="meson setup build --prefix=/usr --buildtype=release -Dcgroup-controller=elogind -Ddbuspolicydir=/etc/dbus-1/system.d; ninja -C build" install="ninja -C build install" - -# 2. graphics -- llvm feeds mesa; mesa options come from the machine -# (machine.conf wiring pending; book defaults until then) -# cmake (pulled in by llvm and others) has NO required dependencies and four -# recommended ones -- and its own build turns on the system libraries, so a -# missing curl is a hard failure: -# CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! -# This is the trade-off, and the answer is the same one: name -# what the build actually needs. -book nghttp2 -book curl -book libarchive -book libuv -book llvm -# glslang provides glslangValidator, which mesa needs ONLY because -# machine.conf turns on the Vulkan drivers -- the book does not list it as a -# mesa dependency, so no dependency order can place it. A machine option -# that creates a dependency has to name it here. -book glslang -# The font stack goes BEFORE mesa. -# -# libass (built inside mesa's plan, via ffmpeg) hard-requires harfbuzz in its -# configure, and the book's libass page does not list it -- so no dependency -# order can place it. What CAN place it is entry order: anything an earlier -# entry installed is simply there. -book harfbuzz # text shaping: freetype recommends it, -book graphite2 # and webkit/pango need both -book fontconfig - -book mesa -book xwayland -book libinput -book libxkbcommon - -# 3. wayland core, then the compositor from git (not in the book) -book wayland -book wayland-protocols - -# 4. a terminal (foot, wayland-native) and its two small deps - -# 5. fonts and 2D -# -# From here on the entries are the RECOMMENDATIONS worth having, named -# instead of inherited: with the book's suggestions no longer -# arrive by themselves, which is the point -- LibreOffice, OpenJDK, CUPS, -# Qt6/okular/VLC, texlive and the xfce/lxqt portals are not part of this -# desktop. Anything missed shows up as a clean "not found" at configure -# time and becomes one more line here. - -# 6. GTK4 + WebKitGTK (6.0 API), gstreamer (bad included -- wanted), mpv -book pango # GTK4 text layout -book gdk-pixbuf # GTK4 image loading -book hicolor-icon-theme -book adwaita-icon-theme # cursors and the default icon set -book gtk4 -book glib-networking # TLS for libsoup, or the browser has no https -book libsoup3 -book gstreamer10 -book gst10-plugins-base -book gst10-plugins-good -# --------------------------------------------------------------------- -# WebRTC and camera streaming. # -# BLFS 13.0 carries NONE of these, so they are git entries. They must be -# built BEFORE gst10-plugins-bad and webkitgtk, because webrtcbin -- the -# element a browser uses for WebRTC -- is compiled into gst-plugins-bad only -# if libnice is present, and webkitgtk links the result. Adding them later -# means recompiling both (hours). +# THREE FILES, ONE COMMAND (1.14.320). This stack begins with the two it +# stands on, in the order they have to be built: # -# What each is for: -# libnice ICE: the NAT traversal WebRTC cannot work without -# libsrtp the encrypted media transport -# usrsctp WebRTC data channels -# webrtc-audio-processing echo cancellation for a microphone (the log's -# "webrtc-audio-processing-2 found: NO") +# wayland.stack session, graphics, wayland, GTK4, seatd, wlroots +# media.stack codecs, gstreamer, WebRTC, webkitgtk, mpv, yt-dlp # -# The camera side is already in this stack: v4l-utils for the device and -# pipewire for routing it. +# and then does what only a Sway desktop needs. The applications that used +# to live here (zathura, imv) are apps.stack now, and jq is in tools.stack: +# none of them is part of the desktop, and none needs rebuilding with it. +include wayland + +# 6b. the media chain, in its own stack so it can be rebuilt on its own. # -# UNTESTED -- the build lines follow each project's README, and none of these -# has been built here. Expect to adjust them; that is what the stack file is -# for. -git libnice url=https://gitlab.freedesktop.org/libnice/libnice.git ref=0.1.22 have=/usr/lib/pkgconfig/nice.pc build="meson setup build --prefix=/usr --buildtype=release -Dgtk_doc=disabled; ninja -C build" install="ninja -C build install" -git libsrtp url=https://github.com/cisco/libsrtp.git ref=v2.6.0 have=/usr/lib/pkgconfig/libsrtp2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" -# ref: 1.0.0 does not exist upstream (checked: the tags are 0.9.3.0, 0.9.4.0, -# 0.9.5.0). 0.9.5.0 has meson.build, so the build line below still applies. -git usrsctp url=https://github.com/sctplab/usrsctp.git ref=0.9.5.0 have=/usr/lib/pkgconfig/usrsctp.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" -git webrtc-audio-processing url=https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing.git ref=v2.1 have=/usr/lib/pkgconfig/webrtc-audio-processing-2.pc build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +# It sits HERE because webkitgtk needs gtk4, pango and libsoup3 from +# wayland.stack, and mesa and ffmpeg from further up. +include media -book gst10-plugins-bad -# No exclusions. The book's dependency graph is built as the book states -# it, recommendations included: a recommendation is there for a reason, and -# skipping one is how texlive came to build before FreeType. This costs -# build time (texlive, LibreOffice and their tails are hours) and buys a -# system that matches the book. -book webkitgtk -book mpv -# 7. audio: ALSA is the floor; pipewire installed, started on demand -book alsa-lib +# 7. audio: alsa-lib is wayland.stack's entry 1b (it has to be early); +# the server comes after media.stack, so pipewire finds gstreamer book alsa-utils book pipewire book wireplumber -# 8. SDL3 for the Sw* tools -book sdl3 - -# 9. the nimbin2 family: make all && make PREFIX=/usr install - -# 10. wire the session together (root step) - - -# TeX is NOT built by this stack. +# 8. SDL3 for the Sw* tools. # -# Putting `book texlive` last only moved the ENTRY: texlive is harfbuzz's -# recommendation, so it was still pulled into an early entry's plan and -# failed there (its configure wants freetype-config, which FreeType removed -# years ago -- an upstream problem, not this desktop's). Nothing here needs -# TeX: the book's `make ... pdf ps` manual builds are commented out (1.12.87) -# and asymptote/biber are avoided. -# -# To build it later, on its own: packagemanager install texlive --run -avoid texlive +# SDL3 dlopens libasound rather than linking it, so ldd says nothing; the +# name it will dlopen is in the binary, and that is what to look for. +# the glob matters: SDL3 installs libSDL3.so.0.<n> and a bare libSDL3.so +# symlink only when the -devel side is there, so naming one path makes the +# probe fail for the wrong reason +book sdl3 expect="strings /usr/lib/libSDL3.so* 2>/dev/null | grep -q libasound" # --------------------------------------------------------------------- -# Everything from here needs GIT, or is a plain tarball: kept to the END. -# -# A git entry failed with "git: command not found" -- git is itself a BLFS -# package, so anything cloned has to come after it is built. Grouping them -# here also means the desktop is finished before the extras start, and -# `--no-git` can skip the lot. +# THE COMPOSITOR AND ITS TOOLS # --------------------------------------------------------------------- -book git - -# -Dserver=enabled: seatd-user execs /usr/bin/seatd, so the BINARY must -# exist even though no daemon runs at boot. -Dlibseat-seatd=enabled is what -# libseat needs for LIBSEAT_BACKEND=seatd; logind stays enabled as a fallback -# but is unusable here (no pam_elogind). -git seatd url=https://git.sr.ht/~kennylevinsen/seatd ref=0.9.1 prog=seatd build="meson setup build --prefix=/usr --buildtype=release -Dserver=enabled -Dlibseat-seatd=enabled -Dlibseat-logind=elogind; ninja -C build" install="ninja -C build install" -# wlroots' DRM backend needs libdisplay-info; without it meson DISABLES drm -# at configure time and says so only in a summary line, and sway then dies at -# runtime with "Cannot create DRM backend: disabled at compile-time". -book libdisplay-info - -# -Dbackends=drm,libinput,... is explicit on purpose: an auto-detected -# backend list is a silent dependency, and this is the dependency that was -# silently missing. -git wlroots url=https://gitlab.freedesktop.org/wlroots/wlroots.git ref=0.19.2 have=/usr/lib/pkgconfig/wlroots-0.19.pc build="meson setup build --prefix=/usr --buildtype=release -Dbackends=drm,libinput,x11 -Dsession=enabled; ninja -C build" install="ninja -C build install" +# scdoc: the man-page generator every sway-family project uses. Not in +# BLFS; SLFS has it (scdoc-1.11.5, from git.sr.ht -- the host seatd already +# comes from). It was assumed present because sway's line below asks for +# -Dman-pages=enabled; it was not, and grim stopped on +# doc/meson.build:1:8: ERROR: Program 'scdoc' not found or not executable +# Before sway, so a replay of this stack meets it in order. No dependencies. +git scdoc url=https://git.sr.ht/~sircmpwn/scdoc ref=1.11.5 prog=scdoc build="make PREFIX=/usr" install="make PREFIX=/usr install" # needs=: book packages a git entry depends on -- nothing in the book orders # anything before a git build, so its dependencies must be named here. # the author's own options: man pages, swaybar, swaynag, bash completions git sway url=https://github.com/swaywm/sway ref=1.11 prog=sway needs=json-c,cairo,pango build="meson setup build --prefix=/usr --buildtype=release -Dman-pages=enabled -Dswaybar=true -Dswaynag=true -Dbash-completions=true; ninja -C build" install="ninja -C build install" git swaybg url=https://github.com/swaywm/swaybg ref=v1.2.1 prog=swaybg build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" + +# SCREENSHOTS AND IDLE: grim, slurp, swayidle. None is a BLFS page; grim and +# slurp are SLFS pages (the Sway book), swayidle is swaywm's own. Refs checked: +# grim v1.5.0 -- SLFS 1.5.0 fetches it from gitlab.freedesktop.org. +# NOT github.com/emersion/grim: that mirror stopped at +# v1.4.0 in 2022 when grim moved away from GitHub. +# slurp v1.5.0 -- git ls-remote on github.com/emersion/slurp; SLFS agrees +# swayidle v1.9.0 -- git ls-remote on github.com/swaywm/swayidle +# needs=libjpeg, not libjpeg-turbo: that is the book's ANCHOR for +# libjpeg-turbo-3.2.0 (`packagemanager search jpeg` shows both). +# Their options are asked for, not autodetected: jpeg (grim -t jpeg) and man +# pages (scdoc is here already: sway builds its man pages with it). +# +# grim -g "$(slurp)" ~/shot.png a region, picked with the mouse +# +# swayidle -Dlogind=disabled: its logind half (before-sleep, lock on +# `loginctl lock-session`) needs a logind SESSION, and sway here gets its +# seat from seatd-<user>, not elogind -- so auto-detection would find +# libelogind, build it in, and have nothing to talk to at runtime. Timeouts +# (`swayidle timeout 300 'swaymsg "output * power off"' resume ...`) are the +# part that works without it. +git grim url=https://gitlab.freedesktop.org/emersion/grim.git ref=v1.5.0 prog=grim needs=libpng,pixman,libjpeg build="meson setup build --prefix=/usr --buildtype=release -Djpeg=enabled -Dman-pages=enabled -Dbash-completions=true; ninja -C build" install="ninja -C build install" +git slurp url=https://github.com/emersion/slurp ref=v1.5.0 prog=slurp needs=cairo,libxkbcommon build="meson setup build --prefix=/usr --buildtype=release -Dman-pages=enabled; ninja -C build" install="ninja -C build install" +git swayidle url=https://github.com/swaywm/swayidle ref=v1.9.0 prog=swayidle build="meson setup build --prefix=/usr --buildtype=release -Dlogind=disabled -Dman-pages=enabled -Dbash-completions=true -Dzsh-completions=false -Dfish-completions=false; ninja -C build" install="ninja -C build install" +# wl-clipboard: wl-copy and wl-paste. Not in the book either, and sway is +# unusable without it once anything wants a clipboard from a script. +git wl-clipboard url=https://github.com/bugaevc/wl-clipboard ref=v2.2.1 \ + prog=wl-copy \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" \ + expect="wl-paste --version >/dev/null 2>&1" + +# wf-recorder: screen recording for a wlroots compositor. Not in the book. +# +# CHECKED against the repo at v0.6.0 (newest tag; master and two feature +# branches also exist): +# +# * audio is CONDITIONAL. meson_options.txt has `pulse` and `pipewire` +# as auto features and config.h.in has `#mesondefine HAVE_AUDIO`, which +# gates src/audio.cpp -- so with neither library found at configure +# time you get a recorder that cannot record sound. +# * and `--help` lists `-a, --audio` EITHER WAY: that text is not inside +# the #ifdef. So the probe checks what the binary LINKS, which is the +# only honest answer to "was audio built in". +# * `meson build` (as in the by-hand script) still works, but `meson +# setup build` is the spelling meson documents now. +# +# Its codecs come from ffmpeg, which is entry 1b, and the default is +# libx264 -- so gst10-plugins-ugly's x264 is not what it uses; ffmpeg's own +# libx264 support is. +git wf-recorder url=https://github.com/ammen99/wf-recorder.git \ + ref=v0.6.0 prog=wf-recorder \ + build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" \ + expect="ldd /usr/bin/wf-recorder 2>/dev/null | grep -qE 'libpulse|libpipewire'" + +# light: backlight and keyboard brightness. Not in the book. +# +# THE UPSTREAM MOVED. haikarainen/light on GitHub is gone -- checked, and +# `git ls-remote` on it fails while three other GitHub URLs in this stack +# answer, so it was the path and not the network. GitHub returns 401 for a +# repository it does not have, which is why the clone asked for a username. +# +# gitlab.com/dpeukert/light is where it continued. Checked on the machine +# this runs on: the URL answers and `git ls-remote --tags` prints NOTHING -- +# the fork carries no tags at all, only `main`. So `ref=main` is not +# laziness here, it is the only ref there is, and it means this entry +# follows upstream: the commit that built is the one worth writing down if +# a build ever breaks. +# +# git ls-remote --heads https://gitlab.com/dpeukert/light.git +# +# CFLAGS='-fcommon' IS NOT A WORKAROUND FOR A BROKEN PACKAGE. light's +# helpers.h declares `light_loglevel` without `extern`, which every C +# compiler accepted until GCC 10 made -fno-common the default; each object +# then defines it and the link fails with +# multiple definition of `light_loglevel' +# -fcommon restores the pre-GCC-10 behaviour, which is what this code was +# written against. The alternative is patching the header, and that is +# upstream's call, not this stack's. +# +# IT HAS TO BE IN THE ENVIRONMENT, NOT ON THE CONFIGURE LINE. autogen.sh +# here is not just autoreconf: it ends by running ./config.status --recheck +# and then `make`, so the compile that fails happens BEFORE the ./configure +# on this line is ever reached. Passing CFLAGS to that configure fixed +# nothing and the link failed exactly as before. Exported, it reaches +# autogen.sh's own configure run, which is the one that writes the Makefile. +# +# And `rm -f config.status` first: autogen.sh re-runs whatever config.status +# is lying around, so a tree configured by hand earlier keeps its old flags +# no matter what this line says. +# +# If that fails too, one of these: +# - the new URL, if light moved: just fix url= below. +# - the source you built from last time, as a tar entry: +# tar light url=file:///path/to/light-1.2.2.tar.gz pkg=light-1.2.2.tar.gz \ +# have=/usr/bin/light build="./autogen.sh; ./configure --prefix=/usr \ +# --with-udev; make" install="make install" +# - brightnessctl, which does the same job and DOES resolve (tag 0.5.1 +# verified from here). It uses the same udev/video-group approach: +# git brightnessctl url=https://github.com/Hummer12007/brightnessctl.git \ +# ref=0.5.1 prog=brightnessctl build="make" \ +# install="make install" expect="brightnessctl --version >/dev/null 2>&1" +# +# It writes to /sys/class/backlight/*/brightness, which root owns, so the +# binary needs either setuid root or a udev rule putting those files in a +# group you are in. --with-udev installs 90-backlight.rules, which chgrps +# them to `video`; that is the one to prefer, and it means the desktop user +# must be in `video` (cfg power puts them in the power group, not this one). +git light url=https://gitlab.com/dpeukert/light.git ref=main prog=light \ + build="export CFLAGS='-O2 -fcommon'; rm -f config.status; ./autogen.sh; ./configure --prefix=/usr --with-udev; make" \ + install="make install" \ + expect="light -V >/dev/null 2>&1" + +# foot, the terminal, and its two small libraries git tllist url=https://codeberg.org/dnkl/tllist ref=1.1.0 have=/usr/include/tllist.h build="meson setup build --prefix=/usr" install="ninja -C build install" git fcft url=https://codeberg.org/dnkl/fcft ref=3.3.2 have=/usr/include/fcft build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" git foot url=https://codeberg.org/dnkl/foot ref=1.28.0 prog=foot build="meson setup build --prefix=/usr --buildtype=release; ninja -C build" install="ninja -C build install" +# the rest of the fonts: icons, a patched terminal font, and Noto for +# everything DejaVu does not cover. One step, because they are all "unpack +# an archive into /usr/share/fonts and run fc-cache". +cfg fonts + tar dejavu-fonts url=https://downloads.sourceforge.net/dejavu/dejavu-fonts-ttf-2.37.tar.bz2 pkg=dejavu-fonts-ttf-2.37.tar.bz2 have=/usr/share/fonts/dejavu build="true" install="install -v -d -m755 /usr/share/fonts/dejavu; install -v -m644 ttf/*.ttf /usr/share/fonts/dejavu; fc-cache /usr/share/fonts/dejavu" + +# --------------------------------------------------------------------- +# YOUR OWN PROGRAMS: the Sw* tools and the browser. SDL3's image and font +# libraries are here because only these use them. +# --------------------------------------------------------------------- git sdl3-image url=https://github.com/libsdl-org/SDL_image ref=release-3.2.4 have=/usr/lib/libSDL3_image.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" git sdl3-ttf url=https://github.com/libsdl-org/SDL_ttf ref=release-3.2.2 have=/usr/lib/libSDL3_ttf.so build="cmake -B build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release; cmake --build build" install="cmake --install build" # Browser builds two programs, browser-mini and browser-big; the entry keeps diff --git a/stacks/tools.stack b/stacks/tools.stack new file mode 100644 index 0000000..1a89858 --- /dev/null +++ b/stacks/tools.stack @@ -0,0 +1,54 @@ +# tools.stack -- command-line tools for everyday use. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +include avoid + +# +# packagemanager stack tools --run +# +# WHY ITS OWN STACK. Every stack here answers one question (see dev.stack). +# This one is "the small programs you type": nothing boots with them, the +# desktop does not need them, and nothing depends on them -- safe to build +# last, safe to leave out. + +# TRANSLATE SHELL (`trans`): Google, Bing, Yandex and others from a terminal. +# trans :de "good evening" -> German +# trans -b en:de "text" just the translation +# trans -shell interactive +# +# Not a BLFS page; upstream's tag. v0.9.7.1 is the newest (checked with git +# ls-remote); built and installed here with the two lines below: `make` +# runs gawk over the sources and writes ONE script, build/trans, and +# `make PREFIX=/usr install` copies it and trans.1. Nothing is compiled. +# +# gawk is its interpreter (LFS has it). curl is what it fetches with -- it +# warns "curl is not found" and returns nothing without it -- hence needs=. +# Optional at runtime: mpv for -speak (media.stack), rlwrap for -shell history. +git translate-shell \ + url=https://github.com/soimort/translate-shell \ + ref=v0.9.7.1 \ + prog=trans \ + needs=curl \ + build="make" \ + install="make PREFIX=/usr install" \ + expect="test -x /usr/bin/trans" + +# jq -- moved here from sway.stack in 1.14.320. Not in the book, so this entry is ours -- and it was BUILT to write +# it, not guessed: +# +# - jq-1.8.2 is the newest tag (jq-1.8.2rc1 exists too; the release +# tarball is the non-rc one), md5 e288a84cf2957ea2cfc643aa9e59b473 +# - the release tarball ships `configure`, so no autogen.sh step +# - oniguruma is NOT on the system and does not need to be: configure says +# "Oniguruma was not found. Will use the packaged oniguruma", builds the +# copy in vendor/, and `jq -n '"abc"|test("b")'` answers true. That is +# the probe below -- `jq --version` would pass on a jq with no regex, +# which is the half of jq most scripts use. +tar jq url=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-1.8.2.tar.gz \ + pkg=jq-1.8.2.tar.gz prog=jq \ + build="./configure --prefix=/usr; make" \ + install="make install" \ + expect="jq -n '\"abc\" | test(\"b\")' >/dev/null 2>&1" + diff --git a/stacks/vision.stack b/stacks/vision.stack new file mode 100644 index 0000000..85c2477 --- /dev/null +++ b/stacks/vision.stack @@ -0,0 +1,88 @@ +# vision.stack -- computer vision, kept out of the desktop stack. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +# Without it a dependency plan walks into systemd, qt6, cups and texlive. +include avoid + +# +# Replay with: packagemanager stack vision --run +# +# WHY ITS OWN STACK. OpenCV is hours of compiling, it is nobody's +# dependency here, and a desktop works perfectly without it -- so it has no +# business in sway.stack, where every entry is something the desktop needs. +# Anything else of this shape (a big optional toolkit, built because you use +# it rather than because something requires it) belongs here too. +# +# None of this is in the BLFS book. That means the ref, the build line and +# the probe are ours to keep true, and the entry says what was checked. + +# opencv 5. +# +# CHECKED: `git ls-remote` lists branch 5.x and tag 5.0.0 (and a +# 5.0.0-alpha). The tag is pinned rather than the branch -- 5.x moves, and +# a build that worked is worth being able to repeat. To follow development +# instead, ref=5.x, and then write down the commit that built. +# +# The build is the same one you would run by hand: +# +# cmake -DOPENCV_GENERATE_PKGCONFIG=ON -DCMAKE_INSTALL_PREFIX=/usr ../opencv +# make -j16 && make install +# +# ...with two differences, both deliberate: +# +# * `cmake -S . -B build` instead of a sibling build directory. The +# clone lands in opencv-git and the phases resume inside it, so the +# build directory has to be under it -- a `../opencv` layout would put +# it outside the tree the package user owns. +# * `cmake --build build` instead of `make -j16`. It reads +# CMAKE_BUILD_PARALLEL_LEVEL, which lfs-helper exports from `jobs` in +# machine.conf or from --jobs, so the count lives in one place instead +# of being typed into this line. +# +# OPENCV_GENERATE_PKGCONFIG=ON is kept: upstream stopped generating the .pc +# file by default at 4.x, and everything that links OpenCV outside cmake +# needs it. +# +# Expect it to want collector groups on the way: the python bindings go into +# python's site-packages and the .pc into pkgconfig's directory, both owned +# by other packages. The repair asks which group to share and retries. +git opencv \ + url=https://github.com/opencv/opencv.git \ + ref=5.0.0 \ + prog=opencv_version \ + have=/usr/bin/opencv_version \ + build="cmake -S . -B build -D OPENCV_GENERATE_PKGCONFIG=ON -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_BUILD_TYPE=Release -D CMAKE_SKIP_INSTALL_RPATH=ON; cmake --build build" \ + install="cmake --install build" \ + expect="opencv_version | grep -q '^5'" + +# WHAT THE FIRST REAL BUILD SHOWED (5.0.0, this machine): +# +# Unavailable: java python3 <- no numpy, so no cv2 for python +# Eigen: NO <- Posegraph optimization stays off +# OpenEXR: NO +# Lapack: built-in libclapack <- no system BLAS/LAPACK +# FFMPEG: YES, GStreamer: YES, v4l2: YES, GTK3 highgui, OpenCL: YES +# +# The python bindings are the one most people miss: cmake reports +# ModuleNotFoundError: No module named 'numpy' +# and then lists python3 as Unavailable -- a full build with no cv2 at the +# end of it. If you want them, numpy first -- WITH --recursive, because +# numpy's own Required is `cython, meson_python` and a plain install builds +# one package: +# packagemanager install numpy --run --recursive +# ...then rebuild opencv, because bindings are decided at configure time. +# +# It also DOWNLOADS during configure: ippicv (Intel IPP) and a unicode font, +# from raw.githubusercontent.com. So this entry needs the network at build +# time, not just at fetch time, and a machine with no route to github cannot +# configure it at all. +# +# opencv_contrib is a SECOND repository whose modules are built by pointing +# opencv's OPENCV_EXTRA_MODULES_PATH at it, not by building it on its own. +# Nothing here needs it, so it is not an entry: if you want it, clone it +# beside opencv and add +# -D OPENCV_EXTRA_MODULES_PATH=<path>/modules +# to the build line above. Written down rather than added, because an entry +# that builds nothing on its own would be a lie about how that repo works. diff --git a/stacks/wayland.stack b/stacks/wayland.stack new file mode 100644 index 0000000..482e608 --- /dev/null +++ b/stacks/wayland.stack @@ -0,0 +1,204 @@ +# wayland.stack -- the base a Wayland desktop stands on, whichever compositor. +# stack-version: 1.14.320 +# +# WHAT THIS MACHINE NEVER BUILDS is in avoid.stack, included by every stack +# here: the exclusions are a property of the machine, not of one stack. +include avoid + +# packagemanager stack wayland --run +# +# The kernel check, the session (dbus, elogind), the audio floor, the +# graphics stack (llvm, glvnd, mesa), wayland itself, GTK4 with libsoup3, +# and -- after git -- seatd and wlroots. Nothing here is sway's: another +# wlroots compositor would stand on exactly this. sway.stack includes it, +# so the full desktop is still one command. +# +# WHY IT WAS SPLIT OUT (1.14.320). sway.stack had grown to 477 lines and +# answered four questions at once: the base, the compositor, a set of +# applications, and one person's own programs. Rebuilding a viewer meant +# reading past mesa's options to find it. + +# Never built by this stack, however they are reached. harfbuzz RECOMMENDS +# texlive and LibreOffice; a sway desktop with a webkit browser wants +# neither, and texlive alone is hours of build time and a /opt tree. +# NOT BUILT BY THIS STACK. The book's recommendations are followed -- +# a recommendation the build actually needs (curl for cmake, icu for +# libxml2) must not be dropped by a blanket rule. These are the branches +# this desktop does not want, named one at a time, and everything they alone +# pull in goes with them. +# the KDE leaves okular pulls in: named individually, because a package can +# have several parents and pruning by one of them drops shared dependencies +# (graphite2 was lost that way) + +# The desktop-portal tail. The book makes xdg-desktop-portal REQUIRE all +# four backends -- gtk, wlr, gnome AND lxqt -- so one recommendation from +# gtk4 drags in the whole of xfce and lxqt, gnome-desktop and libfm-qt. +# Sway needs none of it to run; screen sharing wants xdg-desktop-portal-wlr, +# which can be built on its own later: +# packagemanager install xdg-desktop-portal-wlr --run + +# Not this desktop either: a systemd package on a SysV system, phone modem +# stacks, a mail/ssh server, spell-checking dictionaries. +# openssh is AVOIDED here and INSTALLED by network.stack, where it belongs: +# it is networking, not desktop. Letting a dependency plan pull it in +# mid-sway would put its system-account question in the middle of a graphics +# build. + +# Tails of things already avoided. Each of these is in the plan ONLY to +# serve a package this stack does not build, and the tools cannot yet work +# that out for themselves (see 1.12.83: pruning by the single parent that +# `blfs order` reports dropped shared dependencies, and broke harfbuzz). +# Checked one by one against the plan's parent column: +# Checked against EVERY parent, not just the one the plan prints: these have +# no parent left in the plan at all. +# NOT avoided, though the plan labels them with an avoided parent: the label +# shows only the FIRST parent. gtk3 is really wanted by adwaita-icon-theme, +# ffmpeg by mpv, polkit by colord... and pulseaudio by pipewire/sdl3/gst. +# (rust and llvm STAY: glycin genuinely requires rust, and llvm is a stack +# entry in its own right for mesa.) + +# 0. verify the kernel (DRM/amdgpu, evdev, snd, seccomp, user ns, render node) +check kernel-sway + +# 1. seat and session (SysV: elogind is the logind; seatd is not in the book) +book dbus +book elogind url=https://github.com/elogind/elogind ref=v255.17 prog=loginctl build="meson setup build --prefix=/usr --buildtype=release -Dcgroup-controller=elogind -Ddbuspolicydir=/etc/dbus-1/system.d; ninja -C build" install="ninja -C build install" + +# 1b. THE AUDIO FLOOR, BEFORE ANYTHING THAT LOOKS FOR IT. +# +# alsa-lib used to sit at entry 7, after mpv -- and ffmpeg is built inside +# mesa's plan (entry 2) and SDL3 inside gstreamer's, both of which autodetect +# ALSA. They configured cleanly, installed cleanly, and had no ALSA support, +# because the library did not exist yet. Nothing failed, so nothing said so. +# +# This is the same rule the harfbuzz comment below states for libass: what an +# earlier entry installed is simply there. An optional dependency of +# something built inside a plan can only be placed by entry order. +book alsa-lib + +# ffmpeg and SDL3 are named here, before mesa and gstreamer, so they are +# built in THIS order rather than wherever a dependency plan reaches them -- +# and expect= proves the feature afterwards instead of trusting the order. +book ffmpeg expect="ffmpeg -hide_banner -devices 2>/dev/null | grep -qw alsa" + +# 2. graphics -- llvm feeds mesa; mesa options come from the machine +# (machine.conf wiring pending; book defaults until then) +# cmake (pulled in by llvm and others) has NO required dependencies and four +# recommended ones -- and its own build turns on the system libraries, so a +# missing curl is a hard failure: +# CMAKE_USE_SYSTEM_CURL is ON but a curl is not found! +# This is the trade-off, and the answer is the same one: name +# what the build actually needs. +book nghttp2 +book curl +book libarchive +book libuv +book llvm +# glslang provides glslangValidator, which mesa needs ONLY because +# machine.conf turns on the Vulkan drivers -- the book does not list it as a +# mesa dependency, so no dependency order can place it. A machine option +# that creates a dependency has to name it here. +book glslang +# The font stack goes BEFORE mesa. +# +# libass (built inside mesa's plan, via ffmpeg) hard-requires harfbuzz in its +# configure, and the book's libass page does not list it -- so no dependency +# order can place it. What CAN place it is entry order: anything an earlier +# entry installed is simply there. +book harfbuzz # text shaping: freetype recommends it, +book graphite2 # and webkit/pango need both +book fontconfig + +# LIBGLVND, BEFORE MESA -- because machine.conf builds Mesa with +# glvnd=enabled, and Mesa's configure needs it there. +# +# glvnd is the GL dispatcher: libGL, libEGL and libGLES are its, and +# Mesa installs only its vendor libraries behind it (libGLX_mesa, +# libEGL_mesa + 50_mesa.json). It is what provides pkg-config `opengl`, +# and what lets another GL driver (NVIDIA's) sit beside Mesa. +# +# NOT a BLFS page -- the 13.1 book only links to GLFS for it. The canonical +# repo is gitlab.freedesktop.org/glvnd/libglvnd; the NVIDIA GitHub copy is +# the one reachable from where this was written. CHECKED there: newest tag +# v1.7.0, and the option names (all left at their defaults). +# +# SWITCHING AN EXISTING MACHINE: libglvnd and mesa must be done in ONE go. +# Between the two, libGL.so.1 is glvnd's with no vendor behind it and every +# GL program fails: +# packagemanager stack wayland --only libglvnd,mesa --run +# Programs already built keep working: the sonames (libGL.so.1, libEGL.so.1, +# libGLESv2.so.2) do not change, only who provides them. +git libglvnd \ + url=https://gitlab.freedesktop.org/glvnd/libglvnd.git \ + ref=v1.7.0 \ + needs=xorg7-lib \ + have=/usr/lib/pkgconfig/opengl.pc \ + build="rm -rf build; meson setup build --prefix=/usr --buildtype=release; ninja -C build" \ + install="ninja -C build install" \ + expect="pkg-config --exists opengl && pkg-config --exists glx && test -e /usr/lib/libGLdispatch.so.0" + +# THE SOFTWARE RENDERER IS A SEPARATE DRIVER, and a hardware-only mesa +# fails with "egl: failed to create dri2 screen" and no fallback. The probe +# wants both: radeonsi for the GPU, swrast_dri.so for llvmpipe. (llvmpipe +# installs as swrast_dri.so -- the file is named after the interface, not +# the rasteriser, which is why looking for llvmpipe_dri.so finds nothing.) +book mesa expect="test -e /usr/lib/dri/swrast_dri.so -o -e /usr/lib/dri/kms_swrast_dri.so && test -e /usr/share/glvnd/egl_vendor.d/50_mesa.json && test -e /usr/lib/libGLX_mesa.so.0" +book xwayland +book libinput +book libxkbcommon + +# 3. wayland core +book wayland +book wayland-protocols + +# 5. fonts and 2D +# +# From here on the entries are the RECOMMENDATIONS worth having, named +# instead of inherited: with the book's suggestions no longer +# arrive by themselves, which is the point -- LibreOffice, OpenJDK, CUPS, +# Qt6/okular/VLC, texlive and the xfce/lxqt portals are not part of this +# desktop. Anything missed shows up as a clean "not found" at configure +# time and becomes one more line here. + +# 6. GTK4 and libsoup3. WebKitGTK, gstreamer and mpv are media.stack's. +book pango # GTK4 text layout +book gdk-pixbuf # GTK4 image loading +book hicolor-icon-theme +book adwaita-icon-theme # cursors and the default icon set +book gtk4 +book glib-networking # TLS for libsoup, or the browser has no https +book libsoup3 + +# --------------------------------------------------------------------- +# Everything from here needs GIT: kept to the end. +# +# A git entry failed with "git: command not found" -- git is itself a BLFS +# package, so anything cloned has to come after it is built. +# --------------------------------------------------------------------- +book git + +# -Dserver=enabled: seatd-user execs /usr/bin/seatd, so the BINARY must +# exist even though no daemon runs at boot. -Dlibseat-seatd=enabled is what +# libseat needs for LIBSEAT_BACKEND=seatd; logind stays enabled as a fallback +# but is unusable here (no pam_elogind). +git seatd url=https://git.sr.ht/~kennylevinsen/seatd ref=0.9.1 prog=seatd build="meson setup build --prefix=/usr --buildtype=release -Dserver=enabled -Dlibseat-seatd=enabled -Dlibseat-logind=elogind; ninja -C build" install="ninja -C build install" +# wlroots' DRM backend needs libdisplay-info; without it meson DISABLES drm +# at configure time and says so only in a summary line, and sway then dies at +# runtime with "Cannot create DRM backend: disabled at compile-time". +book libdisplay-info + +# -Dbackends=drm,libinput,... is explicit on purpose: an auto-detected +# backend list is a silent dependency, and this is the dependency that was +# silently missing. +git wlroots url=https://gitlab.freedesktop.org/wlroots/wlroots.git ref=0.19.2 have=/usr/lib/pkgconfig/wlroots-0.19.pc build="meson setup build --prefix=/usr --buildtype=release -Dbackends=drm,libinput,x11 -Dsession=enabled; ninja -C build" install="ninja -C build install" + +# TeX is NOT built by this stack. +# +# Putting `book texlive` last only moved the ENTRY: texlive is harfbuzz's +# recommendation, so it was still pulled into an early entry's plan and +# failed there (its configure wants freetype-config, which FreeType removed +# years ago -- an upstream problem, not this desktop's). Nothing here needs +# TeX: the book's `make ... pdf ps` manual builds are commented out (1.12.87) +# and asymptote/biber are avoided. +# +# To build it later, on its own: packagemanager install texlive --run diff --git a/stacks/webrtc.sh b/stacks/webrtc.sh new file mode 100644 index 0000000..10efb61 --- /dev/null +++ b/stacks/webrtc.sh @@ -0,0 +1,152 @@ +#!/bin/bash +# webrtc.sh -- REPORT what the browser needs for a call and a camera. +# Builds nothing, writes nothing. Every line names a thing and says whether +# it is there. +# +# WHY THIS EXISTS. gst-plugins-bad always builds libgstwebrtc-1.0.so, so +# `packagemanager verify` is clean and the file list looks complete -- while +# the webrtcbin ELEMENT, which is what a browser loads, is compiled only if +# libnice was installed at configure time. The browser says +# GstWebRTC plugin not found. Make sure to install gst-plugins-bad +# >= 1.20 with the webrtc plugin enabled. +# and nothing on the system disagrees with it. A library present is not a +# feature present, and only the element registry knows the difference. +set -u + +fail=0; todos=0 +warn() { echo "!! $*"; fail=1; } +todo() { echo "-> $*"; todos=$((todos + 1)); } +note() { echo "# $*"; } + +have_el() { gst-inspect-1.0 "$1" >/dev/null 2>&1; } + +if ! command -v gst-inspect-1.0 >/dev/null 2>&1; then + warn "no gst-inspect-1.0: gstreamer itself is not installed" + exit 1 +fi +note "gstreamer: $(gst-inspect-1.0 --version 2>/dev/null | sed -n 2p)" +echo + +# ---- 1. the elements a call is made of ----------------------------------- +# element|what it is for|which package builds it|which library it needs +while IFS='|' read -r el what pkg lib; do + [ -n "$el" ] || continue + if have_el "$el"; then + note "$el ($what)" + continue + fi + warn "$el is MISSING -- $what" + note " it comes from $pkg, and only if $lib was installed first" + note " rebuild: packagemanager stack sway --only $pkg --run" +done <<'ELEMENTS' +webrtcbin|the WebRTC session itself|gst10-plugins-bad|libnice +dtlssrtpenc|DTLS key exchange|gst10-plugins-bad|openssl +srtpenc|encrypted media transport|gst10-plugins-bad|libsrtp +sctpenc|WebRTC data channels|gst10-plugins-bad|usrsctp +opusenc|the audio codec every call negotiates|gst10-plugins-base|opus +vp8enc|the video codec every call negotiates|gst10-plugins-good|libvpx +avdec_h264|decoding H.264 (most video on the web)|gst10-libav|ffmpeg +x264enc|ENCODING H.264 -- what a camera site needs to publish|gst10-plugins-ugly|x264 and -D gpl=enabled +openh264enc|the other H.264 encoder, used by some sites|gst10-plugins-bad|openh264 (install it, then REBUILD -bad) +v4l2src|the camera|gst10-plugins-good|v4l-utils +alsasrc|the microphone|gst10-plugins-base|alsa-lib +ELEMENTS +echo + +# ---- 1b. nice to have, not required ------------------------------------- +# gst-plugins-rs is not a BLFS page and not everyone builds it, so a missing +# element here is a note, never a failure: the chain works without it. +for el in webrtcsink webrtcsrc gtk4paintablesink audiornnoise; do + if have_el "$el"; then + note "$el (gst-plugins-rs)" + else + todo "$el is absent -- gst-plugins-rs is not built (optional)" + [ "$el" = audiornnoise ] && note " (noise suppression for a microphone in a call)" + fi +done +echo + +# ---- 2. the libraries, so a missing element says WHY --------------------- +for pc in nice libsrtp2 usrsctp vpx opus webrtc-audio-processing-2 \ + gstreamer-webrtc-1.0; do + if pkg-config --exists "$pc" 2>/dev/null; then + note "$pc $(pkg-config --modversion "$pc" 2>/dev/null)" + else + todo "$pc is not installed (pkg-config cannot see it)" + fi +done +echo + +# ---- 3. the camera itself ------------------------------------------------- +# A device node that exists is not a device you may read: on this system the +# video group is what decides, and a browser is run by a person. +cams="$(ls -1 /dev/video* 2>/dev/null)" +if [ -z "$cams" ]; then + todo "no /dev/video* -- no camera, or its driver has no firmware" + note " check: lfs-kernel firmware --missing" +else + for c in $cams; do + note "$c $(stat -c '%a %U:%G' "$c")" + done + u="${LFS_DESKTOP_USER:-}" + [ -z "$u" ] && [ -r /etc/pkgusr/desktop-user ] && u="$(head -n1 /etc/pkgusr/desktop-user)" + if [ -n "$u" ]; then + if id -nG "$u" 2>/dev/null | tr ' ' '\n' | grep -qx video; then + note "$u is in the video group" + else + todo "$u is not in the video group: usermod -aG video $u" + fi + fi +fi +echo + +# ---- 4. what the browser was built with ---------------------------------- +# webkitgtk's own switches are not introspectable at run time, so read the +# generated script that built it -- the record of what was asked for. +wk="$(ls -1d /usr/src/pkgusr/p_webkitgtk/install_last 2>/dev/null | head -n1)" +if [ -r "$wk" ]; then + for opt in ENABLE_WEB_RTC ENABLE_MEDIA_STREAM USE_GSTREAMER_WEBRTC; do + if grep -q -- "$opt=ON" "$wk" 2>/dev/null; then + note "webkitgtk: $opt=ON" + else + todo "webkitgtk was not built with $opt=ON" + note " add it under [webkitgtk] in /etc/pkgusr/stacks/machine.conf," + note " then: packagemanager stack sway --only webkitgtk --run (hours)" + fi + done +else + note "no /usr/src/pkgusr/p_webkitgtk/install_last: cannot say what the" + note " browser was built with" +fi + +# WHEN WAS IT BUILT, AND WITH WHAT AROUND IT. A script generated by an older +# blfs than the libraries beside it was configured before they existed -- +# which is exactly how this package came to have no webrtc element. +for pkgdir in /usr/src/pkgusr/p_gst10-plugins-bad /usr/src/pkgusr/p_webkitgtk; do + sc="$pkgdir/install_last" + [ -r "$sc" ] || continue + gen="$(sed -n 's/.*generated by blfs \([0-9.]*\).*/\1/p' "$sc" | head -n1)" + [ -n "$gen" ] || continue + if [ -e /usr/lib/pkgconfig/nice.pc ] \ + && [ "$sc" -ot /usr/lib/pkgconfig/nice.pc ]; then + warn "$(basename "$pkgdir") was built BEFORE libnice was installed" + warn " (its script is older than nice.pc), so its optional" + warn " features were detected as absent. Rebuild it." + else + note "$(basename "$pkgdir"): script from blfs $gen" + fi +done + +echo +if [ "$fail" != 0 ]; then + echo "# the !! lines are missing features, not missing files: the package" + echo "# is installed and was configured without them. Rebuilding it now" + echo "# picks them up, because the libraries are there this time." +elif [ "$todos" -gt 0 ]; then + echo "# $todos optional piece(s) absent; a call will work without some of" + echo "# them (data channels, echo cancellation) and not without others." + echo "# The stack is otherwise complete." +else + echo "# the whole chain is here: camera, microphone, codecs, transport." +fi +exit $fail diff --git a/test_lfs_crosschain.sh b/test_lfs_crosschain.sh index 2a61229..668e422 100644 --- a/test_lfs_crosschain.sh +++ b/test_lfs_crosschain.sh @@ -6,9 +6,148 @@ # Then checks the failure paths -- because "it printed done" is not evidence # that anything worked. # -# Usage: bash test_lfs_crosschain.sh [path-to-lfs-script] +# Usage: bash test_lfs_crosschain.sh [path-to-lfs-script] [path-to-book] +# +# RUNNING PART OF IT. The suite outgrew one execution slot, so it can run in +# pieces. Blocks are the `# ---- <title>` sections, numbered in file order: +# +# --list number and name every block +# --from N [--to M] run that range (M defaults to the last block) +# --only N,M,... run just those blocks +# --only <text> run every block whose title contains <text> +# +# A selection re-executes a generated subset: the preamble, the chosen +# blocks, and the summary. The preamble is what every block needs; a block +# that ALSO needs something an earlier block computed cannot run alone, and +# says so with bash's own unbound-variable message naming the variable -- +# rather than passing because its assertions never ran. set -u +_SELF="${BASH_SOURCE[0]}" +_SEL_FROM=""; _SEL_TO=""; _SEL_ONLY=""; _SEL_LIST=0 +_args=() +while [ $# -gt 0 ]; do + case "$1" in + --list) _SEL_LIST=1; shift ;; + --from) _SEL_FROM="${2:-}"; shift 2 ;; + --to) _SEL_TO="${2:-}"; shift 2 ;; + --only) _SEL_ONLY="${2:-}"; shift 2 ;; + --from=*) _SEL_FROM="${1#*=}"; shift ;; + --to=*) _SEL_TO="${1#*=}"; shift ;; + --only=*) _SEL_ONLY="${1#*=}"; shift ;; + -h|--help) + sed -n '3,22p' "$_SELF" | sed 's/^# \{0,1\}//'; exit 0 ;; + --*) echo "!! unknown option: $1" >&2 + echo " try: --list --from N --to M --only N,M|<text>" >&2 + exit 2 ;; + *) _args+=("$1"); shift ;; + esac +done +set -- ${_args[@]+"${_args[@]}"} + +# The splitter is ONE piece of code with two jobs (list, extract), so a +# numbering the list prints cannot disagree with the numbering a range uses. +_blocks() { # _blocks list | _blocks emit <from> <to> <only> + python3 - "$_SELF" "$@" <<'PYSPLIT' +import re, sys +path, mode = sys.argv[1], sys.argv[2] +lines = open(path, encoding="utf-8", errors="replace").read().split("\n") + +# HEREDOC BODIES ARE NOT CODE. Every python test in this suite arrives as a +# quoted heredoc, and their bodies contain lines starting with `# ---- ` and +# with `name=`. Counting those as block markers renumbered the blocks and +# hoisted python assignments into bash. +heredoc = [False] * len(lines) +tag = None +for i, ln in enumerate(lines): + if tag is None: + m = re.search(r"<<-?\s*[\"']?([A-Za-z_][A-Za-z0-9_]*)[\"']?\s*$", ln) + if m: + tag = m.group(1) + continue + heredoc[i] = True + if ln.strip() == tag: + tag = None + +marks = [i for i, ln in enumerate(lines) + if ln.startswith("# ---- ") and not heredoc[i]] +summary = next((i for i, ln in enumerate(lines) + if ln.startswith("#@@ SUMMARY @@")), len(lines)) +bounds = [] +for n, start in enumerate(marks): + end = marks[n + 1] if n + 1 < len(marks) else summary + bounds.append((start, min(end, summary))) + +def title(i): + return re.sub(r"^# ---+ | *-* *#$| -+ #$", "", + lines[i]).strip(" -#").strip() + +if mode == "list": + for n, (start, _end) in enumerate(bounds, 1): + print("%4d %s" % (n, title(start))) + print("\n%d blocks. Run some: --from N --to M | --only N,M |" + " --only <text>" % len(bounds)) + sys.exit(0) + +frm, to, only = sys.argv[3], sys.argv[4], sys.argv[5] +sel = [] +if only: + if re.fullmatch(r"[0-9,\s]+", only): + sel = [int(x) for x in re.split(r"[,\s]+", only.strip()) if x] + else: + sel = [n for n, (s, _e) in enumerate(bounds, 1) + if only.lower() in title(s).lower()] + if not sel: + sys.stderr.write("!! no block title contains %r\n" % only) + sys.exit(2) +else: + a = int(frm) if frm else 1 + b = int(to) if to else len(bounds) + sel = list(range(a, b + 1)) +sel = [n for n in sorted(set(sel)) if 1 <= n <= len(bounds)] +if not sel: + sys.stderr.write("!! that selection is empty (1-%d)\n" % len(bounds)) + sys.exit(2) + +out = lines[:marks[0]] +# What a later block may still need from an earlier one: the tool paths and +# the empty accumulators, which are plain assignments and run nothing. A +# value produced by a command substitution other than `dirname` is NOT +# hoisted -- copying it would silently re-run work. +safe = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=(\"\"|''|)$" + r"|^[A-Za-z_][A-Za-z0-9_]*=\"\$\(dirname [^\n]*\)[^$`]*\"$") +first = bounds[sel[0] - 1][0] +for i in range(marks[0], first): + if not heredoc[i] and safe.match(lines[i].strip()): + out.append(lines[i]) +for n in sel: + s, e = bounds[n - 1] + out.append("") + out.append("#== block %d/%d ==" % (n, len(bounds))) + out.extend(lines[s:e]) +out.extend(lines[summary:]) +sys.stderr.write("running %d of %d blocks: %s\n" + % (len(sel), len(bounds), + ",".join(str(n) for n in sel) if len(sel) < 12 + else "%d-%d" % (sel[0], sel[-1]))) +print("\n".join(out)) +PYSPLIT +} + +if [ "$_SEL_LIST" = 1 ]; then + _blocks list + exit 0 +fi +if [ -n "$_SEL_FROM$_SEL_TO$_SEL_ONLY" ]; then + _sub="$(mktemp /tmp/lfstest-subset.XXXXXX.sh)" + _blocks emit "$_SEL_FROM" "$_SEL_TO" "$_SEL_ONLY" > "$_sub" || exit $? + # exec, so the subset's exit status is the suite's. The temp file is the + # thing that actually ran: keep it, and say where, because a subset that + # dies on an unbound variable is only diagnosable by reading it. + echo " (subset script: $_sub)" >&2 + exec bash "$_sub" ${1+"$@"} +fi + LFS_TOOL="${1:-./lfs}" # A bare name is resolved off $PATH; anything unreadable stops NOW with one # message. `bash test_lfs_crosschain.sh lfs` used to produce hundreds of @@ -46,6 +185,15 @@ LFS_BOOK="${2:-$(_find_book)}" PHASES_LIB="$(cd "$(dirname "$LFS_TOOL")" && pwd)/lfs-phases" export PKGUSR_LIB="$PHASES_LIB" T=$(mktemp -d /tmp/lfstest.XXXXXX) +# THE DESKTOP IS FIVE FILES (1.14.320): wayland, media and sway build as one +# (sway.stack includes the other two), apps and tools beside them. A check +# for an ENTRY reads them all, in build order, with the include lines taken +# out so the concatenation parses on its own in $T. +_DESKTOP_STACK="$T/desktop-all.stack" +for _dsf in wayland media sway apps tools; do + [ -f "$(dirname "${LFS_TOOL:-$0}")/stacks/$_dsf.stack" ] \ + && grep -vE '^include ' "$(dirname "${LFS_TOOL:-$0}")/stacks/$_dsf.stack" +done > "$_DESKTOP_STACK" 2>/dev/null PASS=0; FAIL=0 # Count in FILES, not shell variables. # @@ -54,13 +202,26 @@ PASS=0; FAIL=0 # incremented there dies with the subshell, so the suite printed 446 PASS # lines and reported 433. A wrong count is worse than no count: it can hide a # regression behind a number that still looks healthy. -_COUNTDIR="${TMPDIR:-/tmp}/lfstest-count.$$" +# ...AND IT MUST NOT BE NAMED LIKE THE THING THAT GETS SWEPT UP. This was +# ${TMPDIR:-/tmp}/lfstest-count.$$, beside the /tmp/lfstest.$$ trees the suite +# builds -- and something a block runs sweeps `lfstest*`, so the counters +# vanished mid-run: every later ok() printed a shell error instead of counting, +# and the final line read "PASS: FAIL: SKIP: (final)". A name nothing +# globs for, and a mkdir before each write so a sweep cannot silence it again. +_COUNTDIR="$(mktemp -d "${TMPDIR:-/tmp}/lfscount.XXXXXX")" mkdir -p "$_COUNTDIR" : > "$_COUNTDIR/pass" : > "$_COUNTDIR/fail" +: > "$_COUNTDIR/skip" -ok() { echo " PASS $1"; echo x >> "$_COUNTDIR/pass"; } -bad() { echo " FAIL $1"; echo x >> "$_COUNTDIR/fail"; } +_tick() { [ -d "$_COUNTDIR" ] || mkdir -p "$_COUNTDIR"; echo x >> "$_COUNTDIR/$1"; } +ok() { echo " PASS $1"; _tick pass; } +bad() { echo " FAIL $1"; _tick fail; } +# A check whose PRECONDITION is absent is neither pass nor fail. The rc.site +# check needs a SysV book on disk and reported FAIL without one, so a clean +# run looked like a broken one and the real failures had to be counted by +# hand every time. +skip() { echo " SKIP $1"; _tick skip; } # Lift ONE function out of a bash source file, to exercise it in isolation. # @@ -87,16 +248,50 @@ _slice_fn() { # _slice_fn <file> <fn> ' "$1" } +# _setid_table <lfs-helper>: what _setid_known PRINTS, from the repo's +# setid.conf. Since 1.14.263 the table is that file, not text in the +# function, so the function's body no longer names any path. +_setid_table() { + ( warn() { :; }; PKGUSR_ETC=/nonexistent + eval "$(_slice_fn "$1" _setid_conf)" + eval "$(_slice_fn "$1" _setid_known)" + SETID_CONF="$(dirname "$LFS_TOOL")/stacks/setid.conf" _setid_known ) +} + +_slice_def() { # _slice_def <file> <python def> + # _slice_fn matches SHELL functions. A python def needs its own slice, + # and a slice that returns nothing makes every assertion on it pass + # vacuously -- which is worse than any of them failing. + awk -v fn="$2" 'BEGIN{s="^def " fn "\\("} + !i && $0 ~ s { i=1; print; next } + i && /^[a-zA-Z@]/ { i=0 } i { print }' "$1" +} + # the python-heredoc blocks report their own verdict; fold them in the same way +# (77 = the block printed SKIP: its precondition is not on this machine) _count_rc() { - if [ "$1" -eq 0 ]; then echo x >> "$_COUNTDIR/pass" - else echo x >> "$_COUNTDIR/fail"; fi + if [ "$1" -eq 0 ]; then _tick pass + elif [ "$1" -eq 77 ]; then _tick skip + else _tick fail; fi } -_pass_total() { wc -l < "$_COUNTDIR/pass" | tr -d ' '; } -_fail_total() { wc -l < "$_COUNTDIR/fail" | tr -d ' '; } +_pass_total() { wc -l < "$_COUNTDIR/pass" 2>/dev/null | tr -d ' ' || echo 0; } +_fail_total() { wc -l < "$_COUNTDIR/fail" 2>/dev/null | tr -d ' ' || echo 0; } +_skip_total() { wc -l < "$_COUNTDIR/skip" 2>/dev/null | tr -d ' ' || echo 0; } -cleanup() { rm -rf "$T" "$_COUNTDIR"; } +# ONLY THE SHELL THAT SET IT MAY CLEAN UP. +# +# `trap cleanup EXIT` is inherited by every SUBSHELL, and this suite runs +# blocks inside ( ... ) and $( ... ) constantly. Each of those exits, fires +# the trap, and removes $T and the counters out from under the run that is +# still going: from that point every ok() printed a shell error instead of +# counting and the final line read "PASS: FAIL: SKIP: (final)". $$ is +# the same in a subshell; $BASHPID is not. +_MAIN_PID="${BASHPID:-$$}" +cleanup() { + [ "${BASHPID:-$$}" = "$_MAIN_PID" ] || return 0 + rm -rf "$T" "$_COUNTDIR" +} trap cleanup EXIT export LFS="$T/lfs" @@ -3322,6 +3517,2008 @@ print(" PASS entering the chroot refreshes the tools first") PY52 _count_rc $? +# ---- a test block the book words differently (1.14.211) -------------------- # +# The generator moves a test block to the `test` phase by matching the +# sentence that introduces it, and the pattern was anchored: `^to test`, +# `^to check`, `^the test suite`. numpy's book page says +# "The installation can be tested with the following commands:" +# so a venv, `pip3 install hypothesis attrs` and `numpy.test()` were +# generated into INSTALL. numpy installed -- 1320 files -- and then the +# package failed on `No module named 'pytest'`: a test dependency, in an +# install. +_p="" +_bl="$(dirname "$LFS_TOOL")/blfs" +if [ -f "$_bl" ]; then + grep -q 'can be tested' "$_bl" \ + || _p="$_p;the book's 'can be tested' wording is still read as an install" + grep -q '_TEST_BLOCK_RE.search' "$_bl" \ + || _p="$_p;the test intro is matched only at the start of the sentence" + # THE COMMANDS SAY WHAT THEY ARE. Widening the SENTENCE match was not + # enough: the regenerated script still had numpy's venv and + # `numpy.test()` in install, because the book on that machine (13.0) + # words or nests it differently again. Chasing prose is a losing game + # (1.14.212). + grep -q '_TEST_CONTENT_RE' "$_bl" \ + || _p="$_p;a test block is classified only by the prose around it" + grep -q '_by_content = _test_content_reason' "$_bl" \ + || _p="$_p;the content check is defined and not used first" + _tc="$(python3 - "$_bl" <<'PYTC' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r'_TEST_CONTENT_RE = re\.compile\(\s*r"""(.*?)"""\s*\)', src, re.S) +rx = re.compile(m.group(1), re.M | re.X) +cases = [ + ('python3 -m venv --system-site-packages testenv &&\n' + 'python3 -c "import numpy, sys; sys.exit(numpy.test() is False)"', True), + ('ninja test', True), ('make check', True), ('make -k check', True), + ('meson test --print-errorlogs', True), ('ctest', True), + ('pytest', True), ('python3 -m pytest', True), + ('pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir $PWD', False), + ('pip3 install --no-index --find-links dist --no-user numpy', False), + ('ninja install', False), ('make install', False), + ('install -v -m644 README.md /usr/share/doc/x', False), + ('cmake -D CMAKE_INSTALL_PREFIX=/usr ..', False)] +bad = [t.splitlines()[0] for t, w in cases if bool(rx.search(t)) != w] +print("BAD: " + "; ".join(bad) if bad else "OK") +PYTC +)" + [ "$_tc" = OK ] || _p="$_p;test-by-content is wrong for: $_tc" + # END TO END, THROUGH THE REAL GENERATOR. Two releases in a row shipped + # a regex that passed its own unit check while the GENERATED SCRIPT was + # unchanged -- 1.14.211 widened a sentence match and the script still + # had numpy's venv in install. A pattern test cannot see that; only + # generating a script can. This fixture is numpy's page shape, cut down + # to what the parser requires (a sect2 with div.package, an a.ulink + # download in an itemizedlist, and pre/kbd class="command" blocks). + _fx2="$T/fixbook"; rm -rf "$_fx2"; mkdir -p "$_fx2" + cat > "$_fx2/book.html" <<'FIXBOOK' +<html><body> +<div class="sect2" lang="en"> +<h3 class="sect2"><a id="zzdemo" name="zzdemo"></a>zzdemo-1.2.3</h3> +<div class="package" lang="en"> +<div class="titlepage"><div><div><h4 class="title">Introduction to zzdemo</h4></div></div></div> +<div class="itemizedlist"><ul><li><p>Download (HTTP): <a class="ulink" + href="https://files.pythonhosted.org/packages/source/z/zzdemo/zzdemo-1.2.3.tar.gz" + >zzdemo-1.2.3.tar.gz</a></p></li></ul></div> +<p>Download MD5 sum: 00000000000000000000000000000000</p> +</div> +<div class="installation" lang="en"> +<div class="titlepage"><div><div><h4 class="title">Installation of zzdemo</h4></div></div></div> +<p>Build the module:</p> +<pre class="userinput"><kbd class="command">pip3 wheel -w dist --no-build-isolation --no-deps --no-cache-dir $PWD</kbd></pre> +<p>Now, as the <code class="systemitem">root</code> user:</p> +<pre class="root"><kbd class="command">pip3 install --no-index --find-links dist --no-user zzdemo</kbd></pre> +<p>The installation can be tested with the following commands:</p> +<pre class="userinput"><kbd class="command">mkdir -p test && +cd test && +python3 -m venv --system-site-packages testenv && +source testenv/bin/activate && +pip3 install hypothesis attrs && +python3 -c "import zzdemo, sys; sys.exit(zzdemo.test() is False)" +deactivate</kbd></pre> +</div></div></body></html> +FIXBOOK + if PKGUSR_BASE="$_fx2" PKGUSR_CONFIG=/dev/null \ + python3 "$_bl" --book-file "$_fx2/book.html" script zzdemo -o "$_fx2" \ + >"$_fx2/out" 2>&1; then + _ip="$(sed -n '/^install_pkg()/,/^}/p' "$_fx2/install_zzdemo-1.2.3")" + printf '%s' "$_ip" | grep -q 'pip3 install .*--find-links dist' \ + || _p="$_p;the generated install phase lost the actual install command" + printf '%s' "$_ip" | grep -q '^## SKIPPED' \ + || _p="$_p;the generated install phase does not skip the test block" + printf '%s' "$_ip" | grep -qE '^[^#]*numpy\.test|^[^#]*zzdemo\.test' \ + && _p="$_p;the test call is still live in the install phase" + printf '%s' "$_ip" | grep -qE '^[^#]*python3 -m venv' \ + && _p="$_p;the test venv is still built during install" + else + _p="$_p;the generator could not read a page shaped like numpy's: $(tail -2 "$_fx2/out" | tr '\n' ' ')" + fi + # and a skipped block must not claim the book said something it did not + grep -q 'OPTIONAL, skipped -- the book says: " + _g' "$_bl" \ + && _p="$_p;a content-detected test is attributed to the book" + # the real sentences, from the book + _tb="$(python3 - "$_bl" <<'PYTB' +import re, sys +src = open(sys.argv[1]).read() +m = re.search(r"_TEST_BLOCK_RE = re\.compile\((.*?), re\.I\)\n", src, re.S) +rx = re.compile(eval("(" + m.group(1) + ")"), re.I) +cases = [("The installation can be tested with the following commands:", True), + ("To test the results, issue: ninja test.", True), + ("To run the tests, issue: make check.", True), + ("The test suite requires a graphical session.", True), + ("This package does not come with a test suite.", False), + ("Now, as the root user:", False), + ("Build the module:", False), + ("Install WOFF2 by running the following commands:", False)] +bad = [t for t, want in cases if bool(rx.search(t)) != want] +print("BAD: " + "; ".join(bad) if bad else "OK") +PYTB +)" + [ "$_tb" = OK ] || _p="$_p;test-intro matching is wrong for: $_tb" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a test block is recognised however the book introduces it" +fi + +# ---- configure commands one sub-section down (1.14.262) -------------------- # +# Shadow's configure phase was prose and a bare `:` -- no /etc/pam.d/login, +# passwd, chage or the ten admin tools, so `passwd` said "pam_start() failed, +# error 26" and login "PAM Failure, aborting". The blocks are on the page one +# level down (div.configuration > div.sect3 > div.sect4 > pre.root), and the +# walk read only sect3's children. The fixture is the RENDERED book's shape, +# cut down. BLFS_STORE is fresh: the index cache is what hid three earlier +# extractor fixes, so this must not read one. +_p="" +if [ -f "$_bl" ]; then + _fx3="$T/fixconf"; rm -rf "$_fx3"; mkdir -p "$_fx3/store" + cat > "$_fx3/book.html" <<'FIXCONF' +<html><body> +<div class="titlepage"><div><div><h2 class="title"><a id="zzshadow"></a>zzshadow-4.2.0</h2></div></div></div><div class="sect1" lang="en"> +<div class="package" lang="en"><h3 class="sect2">Introduction to zzshadow</h3> +<div class="itemizedlist"><ul class="compact"><li class="listitem"><p>Download (HTTP): <a class="ulink" + href="https://example.org/zzshadow-4.2.0.tar.xz">https://example.org/zzshadow-4.2.0.tar.xz</a></p></li> +<li class="listitem"><p>Download MD5 sum: 00000000000000000000000000000000</p></li></ul></div></div> +<div class="installation" lang="en"><h3 class="sect2">Installation of zzshadow</h3> +<pre class="userinput"><kbd class="command">./configure --prefix=/usr && make</kbd></pre> +<p>Now, as the root user:</p> +<pre class="root"><kbd class="command">make install</kbd></pre></div> +<div class="configuration" lang="en"><h3 class="sect2">Configuring zzshadow</h3> +<div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title">Configuration Information</h4></div></div></div> +<p>Top level, as the root user:</p> +<pre class="root"><kbd class="command">touch /etc/zz-top-level</kbd></pre> +<div class="sect4"><div class="titlepage"><div><div><h5 class="title">'login'</h5></div></div></div><pre class="root"><kbd class="command">cat > /etc/pam.d/login << "EOF"</kbd> +<code class="literal">auth include system-auth</code> +EOF</pre></div> +<div class="sect4"><div class="titlepage"><div><div><h5 class="title">'chage'</h5></div></div></div><pre class="root"><kbd class="command">cat > /etc/pam.d/chage << "EOF"</kbd> +<code class="literal">auth sufficient pam_rootok.so</code> +EOF</pre></div> +<div class="sect4"><div class="titlepage"><div><div><h5 class="title">Other utilities</h5></div></div></div><pre class="root"><kbd class="command">for PROGRAM in chfn \ + useradd +do + install -v -m644 /etc/pam.d/chage /etc/pam.d/${PROGRAM} +done</kbd></pre></div> +</div></div></div></body></html> +FIXCONF + if BLFS_STORE="$_fx3/store" PKGUSR_BASE="$_fx3" PKGUSR_CONFIG=/dev/null \ + python3 "$_bl" --book-file "$_fx3/book.html" script zzshadow -o "$_fx3" \ + >"$_fx3/out" 2>&1; then + _cp="$(sed -n '/^configure_pkg()/,/^}/p' "$_fx3"/install_zzshadow-*)" + printf '%s' "$_cp" | grep -q '^touch /etc/zz-top-level' \ + || _p="$_p;a top-level configure block was lost" + printf '%s' "$_cp" | grep -q '^cat > /etc/pam.d/login' \ + || _p="$_p;a block inside a sub-section of the configuration is dropped" + printf '%s' "$_cp" | grep -q '^for PROGRAM in chfn' \ + || _p="$_p;the loop in the last sub-section is dropped" + printf '%s' "$_cp" | grep -q "^## 'login'" \ + || _p="$_p;a sub-section's title is not kept as a comment" + # the top level is unchanged: its own title was never emitted + printf '%s' "$_cp" | grep -q 'Configuration Information' \ + && _p="$_p;the top-level titlepage is now emitted (the top level changed)" + # THE SET, DERIVED. shadow-pam checks what the configure phase + # writes, read from this same function -- so run it on this output. + _spf="$(dirname "$LFS_TOOL")/stacks/shadow-pam.sh" + if [ -f "$_spf" ]; then + _set="$( (HELPER="bash $(dirname "$LFS_TOOL")/lfs-helper" + eval "$(sed -n '/^_book_pam_set() {/,/^}/p' "$_spf")" + _book_pam_set "$(ls "$_fx3"/install_zzshadow-*)") | tr '\n' ' ')" + [ "$_set" = "chage chfn login useradd " ] \ + || _p="$_p;the pam.d set read from configure_pkg is wrong: '$_set'" + fi + else + _p="$_p;the generator could not read a shadow-shaped page: $(tail -2 "$_fx3/out" | tr '\n' ' ')" + fi + # THE CACHE KEY INCLUDES THE CODE. It was book size+mtime+a hand-bumped + # number, so an edited extractor rendered from the old index and three + # fixes were reverted as "did not fire" (1.14.262). + grep -q '_build_id()}"$' "$_bl" \ + || _p="$_p;the index cache is not keyed on blfs's own build id" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "configure blocks in sub-sections are generated, and the cache cannot hide it" +fi + +# ---- the lease: hand over before, restore on failure, settle on success ---- # +# (1.14.263) Shadow's configure died on `/etc/pam.d/su: Permission denied`: +# a bash redirect no log parser knew, so the repair granted the DIRECTORY and +# stopped. The lease hands such files to the installer BEFORE the phase, +# puts them back if it fails, and applies setid.conf if it succeeds. +# Exercised with real chown on real accounts, in a sticky directory, without +# su: the phase is simulated by writing between lease and settle. +_p="" +_hlL="$(dirname "$LFS_TOOL")/lfs-helper" +if [ "$(id -u)" != 0 ]; then + skip "the lease (needs root for chown)" +elif [ -f "$_hlL" ]; then + _lz="$( + # NOT UNDER /tmp: script-targets ignores /tmp, /dev, /proc, /sys and + # /run on purpose, and $T lives in /tmp -- every configure target in + # this fixture was silently dropped until it moved. + R="$(mktemp -d /var/tmp/lfslease.XXXXXX)"; trap 'rm -rf "$R"' EXIT + mkdir -p "$R/sbin" "$R/etc" "$R/state/manifests" + # _setid_conf reads $PKGUSR_ETC; under `set -u` an unset one killed + # the lookup, no policy was found, and 4711 came out 711 + PKGUSR_ETC="$R/no-etc" + chmod 1777 "$R/sbin" "$R/etc" + say() { :; }; note() { :; }; warn() { :; }; detail() { :; }; ok() { :; } + unprefix_pkg_user() { echo "$1"; } + user_exists() { id "$1" >/dev/null 2>&1; } + # two real non-root accounts stand in for package users + _a="$(awk -F: '$3>0 && $3<1000 {print $1}' /etc/passwd | sed -n 1p)" + _b="$(awk -F: '$3>0 && $3<1000 {print $1}' /etc/passwd | sed -n 2p)" + pkg_users_list() { printf '%s\n%s\n' "$_a" "$_b"; } + STATE="$R/state"; MANIFESTS="$R/state/manifests"; LEASES="$R/state/leases" + for _f in script_write_targets _setid_conf _setid_policy_for _setid_pkg_for \ + _lease_dir \ + _lease_candidates lease_files lease_restore lease_settle \ + _setid_state _setid_apply; do + eval "$(_slice_fn "$_hlL" "$_f")" + done + printf 'old\n' > "$R/sbin/tool"; chown "root:$_a" "$R/sbin/tool"; chmod 4711 "$R/sbin/tool" + printf 'other\n' > "$R/etc/shared"; chown "$_b:$_b" "$R/etc/shared"; chmod 644 "$R/etc/shared" + printf 'secret\n' > "$R/etc/rootonly"; chmod 600 "$R/etc/rootonly" + printf '%s\n' "$R/sbin/tool" > "$MANIFESTS/$_a.files" + # the 5th column is the package that installs it (1.14.310): without + # it the lease refuses, which is the point of the column + printf '%s | 4711 | root | - | %s\n' "$R/sbin/tool" "$_a" > "$R/setid.conf" + export SETID_CONF="$R/setid.conf" + printf 'configure_pkg() {\ncat > %s << "X"\nnew\nX\necho x >> %s\n}\n' \ + "$R/etc/shared" "$R/etc/rootonly" > "$R/script" + out="" + # 1. a failing phase: both leased files rewritten, then restored + lease_files "$_a" "$R/script" all + [ "$(stat -c %U "$R/sbin/tool")" = "$_a" ] || out="$out;the manifest's root file was not handed over" + [ "$(stat -c %U "$R/etc/shared")" = "$_a" ] || out="$out;another package's configure target was not handed over" + [ "$(stat -c %U "$R/etc/rootonly")" = root ] || out="$out;a root file no manifest claims was handed over" + printf 'broken\n' > "$R/sbin/tool"; printf 'broken\n' > "$R/etc/shared" + lease_restore "$_a" + [ "$(cat "$R/sbin/tool")" = old ] && [ "$(stat -c '%a %U:%G' "$R/sbin/tool")" = "4711 root:$_a" ] \ + || out="$out;a failed phase did not restore the root file ($(stat -c '%a %U:%G' "$R/sbin/tool"))" + [ "$(cat "$R/etc/shared")" = other ] && [ "$(stat -c %U "$R/etc/shared")" = "$_b" ] \ + || out="$out;a failed phase did not restore the other package's file" + [ -e "$LEASES/$_a" ] && out="$out;a restored lease left its ledger behind" + # 2. a succeeding phase: setid.conf applies, a rewrite belongs to the installer + lease_files "$_a" "$R/script" all + printf 'new\n' > "$R/sbin/tool"; printf 'new\n' > "$R/etc/shared" + lease_settle "$_a" + [ "$(stat -c '%a %U:%G' "$R/sbin/tool")" = "4711 root:$_a" ] \ + || out="$out;setid.conf was not applied after success ($(stat -c '%a %U:%G' "$R/sbin/tool"))" + [ "$(stat -c %U "$R/etc/shared")" = "$_a" ] || out="$out;a rewritten file did not stay with its installer" + # 3. a leased file the phase did not change goes back to its owner + chown "$_b:$_b" "$R/etc/shared" + lease_files "$_a" "$R/script" configure + lease_settle "$_a" + [ "$(stat -c %U "$R/etc/shared")" = "$_b" ] || out="$out;an unchanged file was not given back" + # 4. an interrupted lease is restored before the next one + lease_files "$_a" "$R/script" configure + printf 'crashed\n' > "$R/etc/shared" + lease_files "$_a" "$R/script" configure + [ "$(cat "$R/etc/shared")" = new ] || out="$out;an interrupted lease was not restored first" + lease_restore "$_a" + # 5. its OWN file, taken by another package, is reclaimed on install + printf 'mine\n' > "$R/etc/own"; chown "$_b:$_b" "$R/etc/own" + printf '%s\n%s\n' "$R/sbin/tool" "$R/etc/own" > "$MANIFESTS/$_a.files" + lease_files "$_a" "$R/script" install + [ "$(stat -c %U "$R/etc/own")" = "$_a" ] \ + || out="$out;a manifest file another package holds was not leased on install" + printf 'reinstalled\n' > "$R/etc/own" + lease_settle "$_a" + [ "$(stat -c %U "$R/etc/own")" = "$_a" ] \ + || out="$out;a package could not reclaim its own file" + # 7. a root file a manifest claims only by time window is NOT leased + # (1.14.278: LLVM's manifest held /etc/pam.d/su_u_user) + printf 'x\n' > "$R/etc/rootonly"; chown root:root "$R/etc/rootonly"; chmod 755 "$R/etc/rootonly" + printf '%s\n' "$R/etc/rootonly" > "$MANIFESTS/$_a.files" + lease_files "$_a" "$R/script" install + [ "$(stat -c %U "$R/etc/rootonly")" = root ] \ + || out="$out;a root file that is neither setid nor configured was leased" + lease_restore "$_a" 2>/dev/null + # 8. A SETUID FILE IS LEASED ONLY TO THE PACKAGE THAT INSTALLS IT + # (1.14.310). /usr/bin/su landed in webkitgtk's time-window + # manifest because `cfg shadow-pam` ran during its build; the + # lease then chowned it, which clears the setuid bit, and `su` + # was p_webkitgtk 0711 until the phase ended. Three times. + printf 'x\n' > "$R/sbin/theirs"; chown root:root "$R/sbin/theirs"; chmod 4711 "$R/sbin/theirs" + printf '%s | 4711 | root | - | %s\n' "$R/sbin/tool" "$_a" > "$R/setid.conf" + printf '%s | 4711 | root | - | someone-else\n' "$R/sbin/theirs" >> "$R/setid.conf" + printf '%s\n' "$R/sbin/theirs" > "$MANIFESTS/$_a.files" + lease_files "$_a" "$R/script" install + [ "$(stat -c '%a %U' "$R/sbin/theirs")" = "4711 root" ] \ + || out="$out;another package's setuid file was leased ($(stat -c '%a %U' "$R/sbin/theirs"))" + lease_restore "$_a" 2>/dev/null + # ...and a setid line with NO package is leased to nobody + printf '%s | 4711 | root | -\n' "$R/sbin/theirs" > "$R/setid.conf" + lease_files "$_a" "$R/script" install + [ "$(stat -c '%a %U' "$R/sbin/theirs")" = "4711 root" ] \ + || out="$out;a setuid file with no package in setid.conf was leased" + lease_restore "$_a" 2>/dev/null + printf '%s | 4711 | root | - | %s\n' "$R/sbin/tool" "$_a" > "$R/setid.conf" + # 9. A LEASED SYMLINK THE PHASE DID NOT TOUCH GOES BACK (1.14.310). + # `cp -a` copies a RELATIVE link as a relative link, so the copy + # in the lease directory dangles, `cmp` cannot open it, and the + # failed compare read as "the phase rewrote it". Six + # /usr/sbin/iptables* links ended up owned by p_webkitgtk and + # written into its pkg.lst that way. + printf 'real\n' > "$R/sbin/xtables-nft-multi"; chown "$_b:$_b" "$R/sbin/xtables-nft-multi" + ln -sf xtables-nft-multi "$R/sbin/iptables"; chown -h "$_b:$_b" "$R/sbin/iptables" + printf '%s\n' "$R/sbin/iptables" > "$MANIFESTS/$_a.files" + lease_files "$_a" "$R/script" install + [ "$(stat -c %U "$R/sbin/iptables")" = "$_a" ] \ + || out="$out;a manifest symlink another package holds was not leased" + lease_settle "$_a" + [ "$(stat -c %U "$R/sbin/iptables")" = "$_b" ] \ + || out="$out;an untouched symlink was kept by the installer ($(stat -c %U "$R/sbin/iptables"))" + # ...and one the phase DID repoint stays with the installer + lease_files "$_a" "$R/script" install + ln -sfn xtables-legacy-multi "$R/sbin/iptables"; chown -h "$_a:$_a" "$R/sbin/iptables" + lease_settle "$_a" + [ "$(stat -c %U "$R/sbin/iptables")" = "$_b" ] \ + && out="$out;a repointed symlink was given back to the old owner" + # 10. the manifest filter drops what provenance says is not ours + eval "$(_slice_fn "$_hlL" _drop_other_packages_files)" + is_never_claimed() { return 1; } + printf '%s | 4711 | root | - | someone-else\n' "$R/sbin/theirs" >> "$R/setid.conf" + ln -sfn xtables-nft-multi "$R/sbin/iptables"; chown -h "$_a:$_a" "$R/sbin/iptables" + _kept="$(printf '%s\n%s\n%s\n' "$R/sbin/theirs" "$R/sbin/iptables" "$R/sbin/plainfile" \ + | _drop_other_packages_files "$_a")" + printf '%s\n' "$_kept" | grep -qxF "$R/sbin/theirs" \ + && out="$out;the manifest kept a setuid file setid.conf gives to another package" + printf '%s\n' "$_kept" | grep -qxF "$R/sbin/iptables" \ + && out="$out;the manifest kept a symlink into another package's file" + chown -h "$_b:$_b" "$R/sbin/iptables" + # 6. the earlier-stages claim must not strip a setuid bit (1.14.265) + eval "$(_slice_fn "$_hlL" _claim_candidates)" + printf 'x\n' > "$R/sbin/suid"; chmod 4711 "$R/sbin/suid" + printf 'x\n' > "$R/sbin/plain"; chmod 755 "$R/sbin/plain" + printf '%s\n%s\n' "$R/sbin/suid" "$R/sbin/plain" > "$R/claim.files" + _cc="$(LFS_HOST_MOUNT="" SNAP_ROOT="" _claim_candidates "$_a" "$R/claim.files")" + printf '%s\n' "$_cc" | grep -qxF "$R/sbin/suid" \ + && out="$out;the earlier-stages claim takes a setuid root file (chown strips the bit)" + printf '%s\n' "$_cc" | grep -qxF "$R/sbin/plain" \ + || out="$out;the earlier-stages claim no longer takes a plain root file" + rm -rf "$R" + echo "${out:-OK}" )" + [ "$_lz" = OK ] || _p="$_p${_lz#OK}" + # it is wired into the ONE runner, and settles after the log check + _rpa="$(_slice_fn "$_hlL" run_phase_as)" + printf '%s' "$_rpa" | grep -q 'lease_files "$owner" "$script" "$phase"' \ + || _p="$_p;run_phase_as does not lease" + _i_sw="$(printf '%s\n' "$_rpa" | grep -n '_swallowed=' | head -1 | cut -d: -f1)" + _i_st="$(printf '%s\n' "$_rpa" | grep -n 'lease_settle "$owner"' | head -1 | cut -d: -f1)" + [ -n "$_i_sw" ] && [ -n "$_i_st" ] && [ "$_i_sw" -lt "$_i_st" ] \ + || _p="$_p;the lease settles before the log is checked for a swallowed failure" + grep -q ' script-targets) shift; cmd_script_targets' "$_hlL" \ + || _p="$_p;script-targets is not dispatched" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +elif [ "$(id -u)" = 0 ]; then + ok "a leased file is restored on failure and settled on success" +fi + +# ---- one package, another book (1.14.237) ---------------------------------- # +# The systemd book's linux-pam page lists systemd as REQUIRED -- which is +# how `install linux-pam --recursive` came to plan Systemd-261.2 on a SysV +# machine. The 12.4 SysV book's page has NO Required section at all (only +# Shadow, as a recommendation): verified by generating it, `required= +# ('Shadow-4.18.0')`. So a package can be pointed at another book. +_p="" +if [ -f "$pm_src" ]; then + _bf="$(_slice_def "$pm_src" _blfs_book_flags)" + printf '%s' "$_bf" | grep -q '_book_override' \ + || _p="$_p;a package cannot be read from a different book" + _bov="$(_slice_def "$pm_src" _book_override)" + printf '%s' "$_bov" | grep -q '_machine_opts_for' \ + || _p="$_p;the override does not come from machine.conf" + # AND THE CASE IT IS ACTUALLY FOR: shadow 4.20 auto-detects logind, the + # book's page (written for a systemd host) passes no switch, and + # configure stops with "libsystemd not found ... use --disable-logind". + # One switch beats four versions back (1.14.239). + _mc6="$(dirname "$LFS_TOOL")/stacks/machine.conf" + grep -A2 '^\[shadow\]' "$_mc6" 2>/dev/null | grep -q -- '--disable-logind' \ + || _p="$_p;shadow will stop at configure for want of libsystemd" + grep -B10 '^\[shadow\]' "$_mc6" 2>/dev/null | grep -q 'libsystemd not found' \ + || _p="$_p;nothing records the error that switch answers" + # A KEY THAT ADDRESSES THE TOOL MUST NOT REACH THE BUILD LINE. + # `[linux-pam] book = ...` went onto the meson line as -Dbook=... and + # meson stopped with `ERROR: Unknown option: "book"` (1.14.240). + # _warn_unknown_meson_opts cannot catch it: at script-generation time + # the source is not unpacked and there is no meson.options to check. + _am2="$(_slice_def "$pm_src" apply_machine_opts)" + printf '%s' "$_am2" | grep -q 'for _reserved in' \ + || _p="$_p;a tool-level machine.conf key can still reach the build line" + _bko="$(python3 - "$pm_src" "$T" 2>&1 <<'PYBK' +import sys, types, os, io, contextlib +d = os.path.join(sys.argv[2], "bk"); os.makedirs(d, exist_ok=True) +open(os.path.join(d, "machine.conf"), "w").write( + "[zzpam]\nbook = /somewhere/BOOK.html\njobs = 4\n") +os.environ.update(PM_MACHINE_CONF=os.path.join(d, "machine.conf"), + PM_PACKAGE_FIXES=os.path.join(d, "none.conf"), + PKGUSR_BASE=d, PKGUSR_CONFIG="/dev/null") +sp = os.path.join(d, "install_ZzPam-1.0") +open(sp, "w").write("build_pkg() {\nmeson setup .. --prefix=/usr &&\nninja\n}\n") +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + mod.apply_machine_opts("zzpam", sp) +body = open(sp).read() +bad = [] +if "-Dbook" in body or "book=" in body: + bad.append("book reached the build line") +if "ninja -j4" not in body: + bad.append("jobs no longer applied") +print("BAD: " + "; ".join(bad) if bad else "OK") +PYBK +)" + _bko="$(printf '%s\n' "$_bko" | tail -1)" + [ "$_bko" = OK ] || _p="$_p;reserved machine.conf keys: $_bko" + + # NOT A MERGE, and the reason has to be written down + grep -q 'NOT a merge' "$pm_src" \ + || _p="$_p;nothing says why two pages are not merged" + grep -q 'NOT a merge' "$(dirname "$LFS_TOOL")/stacks/machine.conf" \ + || _p="$_p;machine.conf does not explain the whole-page rule" + _boo="$(PM_MACHINE_CONF=$T/bo.conf PM_PACKAGE_FIXES=$T/bo-none.conf \ + PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYBO' +import sys, types, os +open(os.environ["PM_MACHINE_CONF"], "w").write( + "[linux-pam]\nbook = /usr/share/blfs/books/zz-sysv.html\n") +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +one = mod._blfs_book_flags("linux-pam") +other = mod._blfs_book_flags("webkitgtk") +print("OK" if one and "zz-sysv.html" in one[-1] and not other + else "BAD %r %r" % (one, other)) +PYBO +)" + _boo="$(printf '%s\n' "$_boo" | tail -1)" + [ "$_boo" = OK ] || _p="$_p;the per-package book override is wrong: $_boo" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "one package can be read from another book, whole pages only" +fi + +# ---- cmake will not overwrite a symlink (1.14.242) ------------------------- # +# tiff's round 1 handed /usr/lib/pkgconfig/libtiff-4.pc from p_libtiff to +# p_tiff and the retry then died on +# file INSTALL cannot duplicate symlink ... at /usr/lib/libtiff.so.6 +# because: File exists +# Ownership was not the problem: cmake refuses to REPLACE a symlink at all, +# so handing it over changes nothing -- it has to go. And the paths in that +# message are NOT quoted, unlike the copy-file one, which is why the +# extractor saw nothing. +_p="" +_hl12="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl12" ]; then + _tk="$(_slice_fn "$_hl12" _take_cmake_dest)" + printf '%s' "$_tk" | grep -q 'duplicate symlink' \ + || _p="$_p;a symlink cmake refuses to replace is not recognised" + printf '%s' "$_tk" | grep -q 'rm -f "$dest"' \ + || _p="$_p;it hands the symlink over instead of removing it" + printf '%s' "$_tk" | grep -q 'root|""' \ + || _p="$_p;root's symlink could be removed" + # ROOT'S FILES ARE NEVER TAKEN HERE (1.14.263). 1.14.261 handed a + # manifest-claimed root file back AFTER the install failed on it; the + # lease now does that BEFORE the phase, and restores it on failure. + printf '%s' "$_tk" | grep -q "Operation not permitted: '" \ + || _p="$_p;the gate does not know meson's wording" + printf '%s' "$_tk" | sed -n '/root|"")/,/;;/p' | grep -q 'chown' \ + && _p="$_p;the repair loop still takes root's files" + _tbo="$( + R="$T/tb"; rm -rf "$R"; mkdir -p "$R/sbin" "$R/man" + say() { :; }; note() { :; }; warn() { :; } + unprefix_pkg_user() { echo "${1#p_}"; } + MANIFESTS="$R/man" + : > "$R/sbin/ours" + _asuser="$(awk -F: '$3>0 && $3<1000 {print $1; exit}' /etc/passwd)" + [ -n "$_asuser" ] || _asuser=nobody + printf '%s\n' "$R/sbin/ours" > "$MANIFESTS/$_asuser.files" + printf "Operation not permitted: '%s'\n" "$R/sbin/ours" > "$R/log" + eval "$(_slice_fn "$_hl12" _take_cmake_dest)" + _take_cmake_dest "$R/log" "$_asuser" >/dev/null 2>&1 + stat -c '%U' "$R/sbin/ours" )" + [ "$_tbo" = root ] || _p="$_p;the repair loop took a root file ($_tbo)" + # the unquoted form must extract + printf '%s\n' 'CMake Error at x/cmake_install.cmake:57 (file):' \ + ' file INSTALL cannot duplicate symlink' '' ' /src/b/libtiff.so.6' \ + '' ' at' '' ' /usr/lib/libtiff.so.6' '' ' because: File exists' \ + > "$T/sym.log" + _got="$(tr '\n' ' ' < "$T/sym.log" \ + | grep -aoE 'cannot duplicate symlink[[:space:]]+[^[:space:]]+[[:space:]]+at[[:space:]]+[^[:space:]]+' \ + | awk '{print $NF}')" + [ "$_got" = /usr/lib/libtiff.so.6 ] \ + || _p="$_p;the unquoted destination parses to '$_got'" +fi + # AN ENTRY IS ONLY DONE IF IT IS THERE AFTERWARDS -- EVERY KIND. The + # check covered git and tar only, so poppler (a `book` entry) came out + # `done (stack cache)` while pkg-config found no poppler at all, and + # every later run skipped it -- leaving zathura-pdf-poppler failing on a + # dependency the plan believed it had (1.14.246). + _cs="$(_slice_def "$pm_src" cmd_stack)" + printf '%s' "$_cs" | grep -q 'elif e\["kind"\] == "book"' \ + || _p="$_p;a book entry is recorded done without being checked" + printf '%s' "$_cs" | grep -q '_installed_now' \ + || _p="$_p;the done-check is still per-kind" + # and the reason zathura's backend needs poppler at all must be asked for + grep -A14 '^\[poppler\]' "$(dirname "$LFS_TOOL")/stacks/machine.conf" \ + 2>/dev/null | grep -q 'ENABLE_GLIB *= *ON' \ + || _p="$_p;poppler may build without the glib frontend the chain needs" + + # A PACKAGE PROMOTED OUT OF avoid.stack BRINGS ITS AVOIDED CHILDREN + # BACK INTO QUESTION. poppler became wanted for zathura's backend, and + # nss, boost and gpgmepp are all in avoid.stack AS POPPLER'S CHILDREN -- + # so each stops configure in turn, naming its own switch (1.14.243). + _mc7="$(dirname "$LFS_TOOL")/stacks/machine.conf" + _av7="$(dirname "$LFS_TOOL")/stacks/avoid.stack" + # FOUR soft-mandatory switches, found one configure at a time: NSS3, + # GPGME, BOOST, then QT6 (1.14.247). qt6 is avoided and poppler's page + # passes only ENABLE_QT5=OFF, because a BLFS system is expected to have + # qt6. The comment names the grep that lists them all at once. + for _sw in ENABLE_NSS3 ENABLE_GPGME ENABLE_BOOST ENABLE_QT6; do + grep -A14 '^\[poppler\]' "$_mc7" 2>/dev/null | grep -q "$_sw *= *OFF" \ + || _p="$_p;poppler will stop at configure for $_sw" + done + grep -B14 '^\[poppler\]' "$_mc7" 2>/dev/null | grep -q 'OBSERVED' \ + || _p="$_p;nothing separates the observed failure from the anticipated ones" + grep -A26 '^\[poppler\]' "$_mc7" 2>/dev/null \ + | grep -q 'find_soft_mandatory_package' \ + || _p="$_p;nothing says how to list poppler's soft-mandatory switches at once" + for _c in nss boost gpgmepp; do + grep -qE "^avoid +$_c" "$_av7" \ + || _p="$_p;$_c is no longer excluded, so poppler would build it" + done + +# AND THE SAME LIBRARY UNDER TWO NAMES. p_libtiff and p_tiff are the same +# upstream package: 12.4 titles the page libtiff, 13.1 titles it tiff. Two +# accounts own two copies of one library, and whichever installed last wins +# every file. Reported, not fixed: `packagemanager which-package +# /usr/lib/libtiff.so.6` says who holds it today. +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a symlink another package owns is removed, not handed over" +fi + +# ---- the version is what FINISHED (1.14.236) ------------------------------- # +# install_last is staged into the package home BEFORE the build runs, and +# the installed version was read out of it -- so webkitgtk 2.52.5 died at +# configure and `update webkitgtk` then answered "already up to date +# (WebKitGTK-2.52.5)" while /usr/lib/libwebkitgtk-6.0.so.4.13.8 on disk was +# still 2.50.5's. A failed build had advanced the version. +_p="" +_hlv="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$pm_src" ] && [ -f "$_hlv" ]; then + grep -q '.pkgusr-installed' "$pm_src" \ + || _p="$_p;the version still comes from the staged script" + grep -q '.pkgusr-installed' "$_hlv" \ + || _p="$_p;nothing records which version finished installing" + # ...AND THE VARIABLE MUST EXIST IN THAT SCOPE. `$home` is a local of + # the install function, not of the one that prints "done (all phases + # complete)", so with set -u the block died with + # lfs-helper: line 7345: home: unbound variable + # and the marker -- the entire point of it -- was never written + # (1.14.250). + grep -A14 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ + | grep -q '"\$home/install_last"' \ + && _p="$_p;the marker still reads a variable that is not in scope" + grep -A20 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ + | grep -q 'pkgusr_home_for "\$name"' \ + || _p="$_p;the marker does not resolve the home the way other callers do" + # the whole tool must survive set -u: an unbound variable anywhere in a + # success path is a silent no-op at best + bash -n "$_hlv" || _p="$_p;lfs-helper does not parse" + # written ONLY after every phase succeeded + # the marker line and the success line must be in the same few lines: + # a marker written anywhere else is a marker written on failure too + grep -A34 'WHAT IS INSTALLED IS WHAT FINISHED' "$_hlv" \ + | grep -q 'all phases complete' \ + || _p="$_p;the marker is not written at the success point" + _mk="$T/mk"; rm -rf "$_mk"; mkdir -p "$_mk/pkgusr/p_zzver" + printf "name='zzver'\nname_version='ZzVer-2.0'\n" \ + > "$_mk/pkgusr/p_zzver/install_last" + echo x > "$_mk/pkgusr/p_zzver/pkg.lst" + _vo="$(PKGUSR_BASE=$_mk PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYVER' +import sys, types, os +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +home = os.path.join(os.environ["PKGUSR_BASE"], "pkgusr", "p_zzver") +a = mod.gather_user("zzver").version # no marker: staged script +open(os.path.join(home, ".pkgusr-installed"), "w").write("ZzVer-1.0") +b = mod.gather_user("zzver").version # marker wins +print("OK" if a == "ZzVer-2.0" and b == "ZzVer-1.0" else "BAD %r %r" % (a, b)) +PYVER +)" + _vo="$(printf '%s\n' "$_vo" | tail -1)" + [ "$_vo" = OK ] || _p="$_p;the installed version is read wrongly: $_vo" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the recorded version is the one that finished, not the one staged" +fi + +# ---- one scalar, two paths (1.14.230) -------------------------------------- # +# `info opus` said "unvalidated: missing: dir /usr/include/opus +# /usr/share/doc/opus-1.6.1" with /usr/include/opus sitting right there. +# The book writes "Installed Directories: A and B", both landed in ONE +# `installed_directory=`, and the check looked for a directory literally +# called "A B". And the doc half was never going to exist: it needs Doxygen +# and texlive, which this system deliberately does not have. +_p="" +if [ -f "$pm_src" ]; then + _it="$(_slice_def "$pm_src" install_targets)" + printf '%s' "$_it" | grep -q 'def _paths' \ + || _p="$_p;a field with two paths in it is treated as one path" + _ito="$(python3 - "$pm_src" "$T" <<'PYIT' +import sys, types, os +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +bad = [] +_, _, dirs = mod.install_targets( + {"name": "opus", + "installed_directory": "/usr/include/opus /usr/share/doc/opus-1.6.1"}) +if dirs != ["/usr/include/opus", "/usr/share/doc/opus-1.6.1"]: + bad.append("two directories parse as %r" % (dirs,)) +progs, _, _ = mod.install_targets( + {"name": "x", "installed_programs": ["gst-launch-1.0 gst-inspect-1.0"]}) +if progs != ["gst-launch-1.0", "gst-inspect-1.0"]: + bad.append("two programs parse as %r" % (progs,)) +# a missing DOC dir is documentation, not a fault; a missing real one is +base = os.path.join(sys.argv[2], "op") +for name, extra, want in (("docsgone", "/nope-doc/share/doc/x-1.0", "installed"), + ("realgone", "/nope-real", "unvalidated")): + home = os.path.join(base, "pkgusr", "p_" + name) + os.makedirs(os.path.join(base, "inc"), exist_ok=True) + os.makedirs(home, exist_ok=True) + open(os.path.join(home, "install_last"), "w").write( + "name='%s'\nname_version='%s-1.0'\ninstalled_directory='%s %s'\n" + % (name, name, os.path.join(base, "inc"), extra)) + open(os.path.join(home, "pkg.lst"), "w").write(os.path.join(base, "inc") + "\n") +os.environ["PKGUSR_BASE"] = base +os.environ["PKGUSR_CONFIG"] = "/dev/null" +mod2 = types.ModuleType("pm2"); mod2.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod2.__dict__) +for name, want in (("docsgone", "installed"), ("realgone", "unvalidated")): + u = mod2.gather_user(name) + if not u.has_user: + continue # no account here; the parse tests stand + st, why = mod2.assess(u, None, run_cmd=False) + if st != want: + bad.append("%s -> %s (%s), wanted %s" % (name, st, why, want)) +print("BAD: " + "; ".join(bad) if bad else "OK") +PYIT +)" + [ "$_ito" = OK ] || _p="$_p;install target parsing: $_ito" + printf '%s' "$(_slice_def "$pm_src" assess)" | grep -q 'share/doc/' \ + || _p="$_p;a doc directory that was never built counts as missing" + # `why` EXPLAINS A VERDICT THAT NEEDS EXPLAINING. For an installed + # package it printed the routine sentence every time, above the very + # fields it describes, so the one line that ever matters was + # indistinguishable from noise (1.14.231). + grep -q '_routine = (' "$pm_src" \ + || _p="$_p;a routine verdict is still printed as a reason" + # _slice_def stops at the first line starting with a letter, and + # print_user_details has one before this point -- grep the file + grep -q '_label = "note" if state == "installed"' "$pm_src" \ + || _p="$_p;an installed package's note is still labelled why" + grep -q 'docs not built (no doxygen/texlive here)' "$pm_src" \ + || _p="$_p;it does not say WHY the doc directory is absent" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "two paths in one field parse as two, and absent docs are not a fault" +fi + +# ---- one quote character, three symptoms (1.14.220) ------------------------ # +# `_parse_bash_vars` read only key="value". The BOOK-generated scripts +# write name_version='Rustc-1.93.1' in single quotes, so name_version was +# never parsed for any package built from the book -- and that is where the +# installed version comes from. Hence: `info webkitgtk` showed uid, home, +# 1727 files and every dependency but not which webkitgtk; `update` filed it +# under "cannot judge"; late-deps reported 28 packages with "no recorded +# version". +_p="" +if [ -f "$pm_src" ]; then + _pv="$(python3 - "$pm_src" <<'PYPV' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +cases = [("name_version='Rustc-1.93.1'\n", "Rustc-1.93.1"), + ('name_version="openh264-2.6.0"\n', "openh264-2.6.0"), + ("name_version='WebKitGTK-2.50.2' # from the book\n", "WebKitGTK-2.50.2"), + ('name_version=""\n', "")] +bad = [t.strip() for t, w in cases + if mod._parse_bash_vars(t, ("name_version",), ()).get("name_version") != w] +print("BAD: " + "; ".join(bad) if bad else "OK") +PYPV +)" + [ "$_pv" = OK ] || _p="$_p;a scalar is misparsed: $_pv" + # THE BOOK PUTS cmake AFTER `&&`, and the detector anchored to the start + # of a line: webkitgtk's page writes + # mkdir -vp build && cd build && cmake -D CMAKE_BUILD_TYPE=Release \\ + # so all seven [webkitgtk] options were reported as "NOT applied" on the + # one package where they matter most (1.14.222). + _hc="$(_slice_def "$pm_src" _has_command)" + [ -n "$_hc" ] || _p="$_p;whether a build command exists is still a bare regex" + grep -q '_has_command(body, "cmake")' "$pm_src" \ + || _p="$_p;the cmake detector does not use it" + grep -q 'r"\^\\s\*cmake' "$pm_src" \ + && _p="$_p;the old line-anchored cmake test is still there" + _hco="$(python3 - "$pm_src" <<'PYHC' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +cases = [("mkdir -vp build && cd build && cmake -D X=1 ..\n", "cmake", True), + ("mkdir -vp build &&\ncd build &&\ncmake -D X=1 ..\n", "cmake", True), + ("## NOTE: add it to the cmake command\n:\n", "cmake", False), + ("mkdir build && cd build && meson setup --prefix=/usr ..\n", + "meson setup", True), + ("./configure --prefix=/usr\nmake\n", "cmake", False)] +bad = [c[0].splitlines()[0] for c in cases + if mod._has_command(c[0], c[1]) != c[2]] +b = "## NOTE: add it to the cmake command\nmkdir -p build && cd build && cmake -D P=G ..\n" +if "-DUSE_WOFF2=ON" not in mod._insert_on_command(b, "cmake", "-DUSE_WOFF2=ON").splitlines()[1]: + bad.append("insertion missed the real command line") +print("BAD: " + "; ".join(bad) if bad else "OK") +PYHC +)" + [ "$_hco" = OK ] || _p="$_p;command detection is wrong for: $_hco" + # AND THE VERSION MAP IS CACHED BETWEEN RUNS. `blfs versions` re-reads + # the whole book -- eighteen seconds -- and `info` paid it per command. + _bva="$(_slice_def "$pm_src" book_versions_all)" + printf '%s' "$_bva" | grep -q 'book-versions.json\|_versions_cache_path' \ + || _p="$_p;the book version map is rebuilt on every command" + printf '%s' "$_bva" | grep -q 'st_mtime' \ + || _p="$_p;the cache is not keyed to the book, so it can outlive it" + printf '%s' "$_bva" | grep -q 'did not read back' \ + || _p="$_p;a cache that fails to load would be silent" + + # THE PREFIX IS NOT PART OF THE BOOK'S NAME. `info webkitgtk` asked the + # version map for `p_webkitgtk`, missed, and fell through to a `blfs + # debug` SUBPROCESS that reads the book cache to answer one lookup and + # then fails -- the pause after "looking up p_webkitgtk in the book", + # and the reason no book version was shown (1.14.221). + _bvf="$(_slice_def "$pm_src" book_version)" + printf '%s' "$_bvf" | grep -q 'pkgusr_prefix()' \ + || _p="$_p;the account prefix is not stripped before the book lookup" + printf '%s' "$_bvf" | grep -q 'run_blfs(\["debug", _tries\[-1\]\])' \ + || _p="$_p;the fallback still asks with the prefixed name" + # no subprocess for a plain info, and the book version is shown + if [ -d "$T/iw/pkgusr/p_zzwk" ] || mkdir -p "$T/iw/pkgusr/p_zzwk"; then + printf "name='zzwk'\nname_version='ZzWk-1.2.3'\n" \ + > "$T/iw/pkgusr/p_zzwk/install_last" + echo x > "$T/iw/pkgusr/p_zzwk/pkg.lst" + _io2="$(PKGUSR_BASE=$T/iw PKGUSR_CONFIG=/dev/null \ + timeout 120 python3 "$pm_src" info zzwk 2>&1)" + printf '%s' "$_io2" | grep -q 'looking up p_' \ + && _p="$_p;info still spawns a book lookup for a prefixed name" + printf '%s' "$_io2" | grep -q 'version : 1.2.3' \ + || _p="$_p;info does not show the bare version number" + fi + # the label column: {colors.normal:<9} padded the ESCAPE SEQUENCE + printf '%s' "$_pd" | grep -q "colors.normal:<9" \ + && _p="$_p;the why label pads a colour code instead of the word" + # and info must PRINT the version, both the installed one and the book's + _pd="$(_slice_def "$pm_src" print_user_details)" + printf '%s' "$_pd" | grep -q '("version"' \ + || _p="$_p;info does not print the installed version as a field" + printf '%s' "$_pd" | grep -q '("book"' \ + || _p="$_p;info does not say what the book carries" + printf '%s' "$_pd" | grep -q 'name_version' \ + || _p="$_p;info has no answer when the version was not recorded" + # ...and update's "cannot judge" must name the book version beside each + grep -q 'no_record.append("%s (book: %s)"' "$pm_src" \ + || _p="$_p;'no recorded version' still does not say what the book has" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a version is parsed in either quote style, and both versions are shown" +fi + +# ---- application users from a file (1.14.217) ------------------------------ # +# The accounts a browser runs as were a command line per account, and the +# two files that list them all -- /etc/pam.d/su_u_user and the SHARED_* +# variables -- were hand-kept. Both are now derived from app-users.conf, +# and neither may clobber a hand-edited file. +_p="" +_au="$(dirname "$LFS_TOOL")/stacks/app-users.sh" +_ac2="$(dirname "$LFS_TOOL")/stacks/app-users.conf" +if [ -f "$_au" ] && [ -f "$_ac2" ]; then + bash -n "$_au" || _p="$_p;app-users.sh does not parse" + grep -qE '^cfg +app-users' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + || _p="$_p;nothing runs the app-users step" + # ONE NAMESPACE FOR THIS SYSTEM'S FACTS, AND IT MUST BE INERT. + # PKGUSR_BASE, PKGUSR_CONFIG and PKGUSR_LIB are READ by the tools as + # overrides, so exporting that namespace system-wide would reconfigure + # every tool in every login shell. PKGUSR_SYS_* is consumed by nothing + # (1.14.219). + _se3="$(_slice_def "$pm_src" _system_env_values)" + [ -n "$_se3" ] || _p="$_p;there is no place that states this system's facts" + printf '%s' "$_se3" | grep -q 'PKGUSR_SYS_MAIN_USER' \ + || _p="$_p;the main user is not among the exported facts" + for _bad in PKGUSR_BASE PKGUSR_CONFIG PKGUSR_LIB; do + printf '%s' "$_se3" | grep -q "\"$_bad\"" \ + && _p="$_p;$_bad is exported, which reconfigures the tools" + done + _ces="$(_slice_def "$pm_src" cmd_env_system)" + printf '%s' "$_ces" | grep -q 'SYSTEM_ENV_MARKER not in had' \ + || _p="$_p;a profile fragment that is not ours would be overwritten" + printf '%s' "$_ces" | grep -q 'getattr(args, "run", False)' \ + || _p="$_p;env --system writes without being asked to" + # the real command prints the facts and names what it does not know + _seo="$(timeout 60 python3 "$pm_src" env --system 2>&1)" + printf '%s' "$_seo" | grep -q 'export PKGUSR_SYS_USER_PREFIX=' \ + || _p="$_p;env --system does not print the prefixes it knows" + printf '%s' "$_seo" | grep -q 'dry run' \ + || _p="$_p;env --system is not a dry run by default" + # and the app-users step asks for them rather than guessing + grep -q 'PKGUSR_SYS_MAIN_USER' "$_au" \ + || _p="$_p;app-users.sh still guesses the main user first" + + # THE PREFIX IS CONFIGURABLE and must not be hard-coded + grep -q 'user-prefix' "$_au" \ + || _p="$_p;the account prefix is assumed rather than read" + # the quoted example in the comment is not code: skip comment lines + grep -vE '^\s*#' "$_au" | grep -qE '"u_[a-z]' \ + && _p="$_p;a u_ prefix is hard-coded in app-users.sh" + # IT IS A SCRIPT, NOT A CONFIG FILE. /etc/pam.d/su_u_user starts with + # `#!/bin/bash` and is RUN by pam_exec.so; written 644 it cannot be + # executed, and `su - u_claude` just asks for a password with nothing in + # the logs to say why (1.14.232). + [ "$(grep -c 'chmod 755 "$PAM_FILE"' "$_au")" -ge 2 ] \ + || _p="$_p;the PAM script is not executable after both write and merge" + grep -q 'pam_exec.so quiet' "$_au" \ + || _p="$_p;nothing says how pam is told to run the script" + # AND THE LINE IS ADDED, with a backup, above the other auth lines -- + # `sufficient` succeeds the stack immediately, so after them it would + # never run for the accounts it exists for. On a machine with no + # Linux-PAM (avoid.stack avoids it) the step says so instead of writing + # into a file pam never reads (1.14.233). + grep -q 'cp -p "$PAM_SU" "$bak"' "$_au" \ + || _p="$_p;/etc/pam.d/su is edited without a backup" + grep -q 'Linux-PAM is not installed here' "$_au" \ + || _p="$_p;a machine with no pam gets a line it will never read" + grep -q 'restoring the backup' "$_au" \ + || _p="$_p;a failed edit of /etc/pam.d/su is not undone" + _psu="$( + R="$T/au8"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" + { echo '#!/bin/sh'; echo 'case "$1 $2" in' + echo ' "config --show") case "$3" in main-user) echo t;; user-prefix) echo u_;; esac; exit 0;;' + echo ' "user setup"|"user create"|"user --help"|"env --system") exit 0;;' + echo 'esac'; echo 'exit 2'; } > "$R/pm"; chmod 755 "$R/pm" + printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\n' > "$R/conf" + printf '%s\n' '# Begin /etc/pam.d/su' 'auth sufficient pam_rootok.so' \ + 'auth required pam_unix.so' 'account required pam_unix.so' \ + > "$R/etc/pam.d/su" + APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ + APP_USERS_PAM_SU="$R/etc/pam.d/su" APP_USERS_PROFILE_D="$R/etc/profile.d" \ + APP_USERS_PM="$R/pm" bash "$_au" >/dev/null 2>&1 + head -2 "$R/etc/pam.d/su" | tail -1 | grep -c pam_exec )" + [ "${_psu:-0}" = 1 ] \ + || _p="$_p;the pam_exec line is not the first auth line in /etc/pam.d/su" + # (the step EDITS /etc/pam.d/su now, with a backup -- the old assertion + # asked for the refusal message it no longer prints) + grep -q 'if su is broken, put the backup back' "$_au" \ + || _p="$_p;it does not say how to undo the /etc/pam.d/su change" + _pmo="$( + R="$T/au7"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" + { echo '#!/bin/sh'; echo 'case "$1 $2" in' + echo ' "config --show") case "$3" in main-user) echo t;; user-prefix) echo u_;; esac; exit 0;;' + echo ' "user setup"|"user create"|"user --help"|"env --system") exit 0;;' + echo 'esac'; echo 'exit 2'; } > "$R/pm"; chmod 755 "$R/pm" + printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\n' > "$R/conf" + APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ + APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ + bash "$_au" >/dev/null 2>&1 + _m1="$(stat -c %a "$R/etc/pam.d/su_u_user" 2>/dev/null)" + chmod 644 "$R/etc/pam.d/su_u_user" 2>/dev/null + printf 'name|app|share|dir|desktop|launcher|groups\nzz1|/bin/true|no|no|no|no|-\nzz2|/bin/true|no|no|no|no|-\n' > "$R/conf" + APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ + APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ + bash "$_au" >/dev/null 2>&1 + echo "$_m1 $(stat -c %a "$R/etc/pam.d/su_u_user" 2>/dev/null)" )" + [ "$_pmo" = "755 755" ] \ + || _p="$_p;PAM script modes are '$_pmo', wanted '755 755'" + + # A HAND-EDITED FILE IS BACKED UP AND MERGED, NEVER OVERWRITTEN + grep -q 'cp -p "$PAM_FILE" "$bak"' "$_au" \ + || _p="$_p;the PAM file is changed without a backup" + grep -q 'pam_has' "$_au" \ + || _p="$_p;accounts already in the PAM file would be duplicated" + grep -q 'left alone' "$_au" \ + || _p="$_p;a profile fragment that is not ours would be overwritten" + # `user setup` NEEDS THE ACCOUNT TO EXIST. All three lines of + # app-users.conf answered "u_browser-mini: no such user. Create it + # first" -- which makes a file that DECLARES accounts useless for the + # one job it has. `user create` does both (1.14.231). + grep -q 'set -- user create' "$_au" \ + || _p="$_p;an account that does not exist yet can never be created" + grep -q 'id "$PREFIX$name" >/dev/null' "$_au" \ + || _p="$_p;it does not check whether the account exists first" + _cro="$( + R="$T/au4"; rm -rf "$R"; mkdir -p "$R/etc/pam.d" "$R/etc/profile.d" + { + echo '#!/bin/sh' + echo 'case "$1 $2" in' + echo ' "config --show") case "$3" in main-user) echo tester;; user-prefix) echo u_;; esac; exit 0;;' + echo ' "user setup") echo SETUP; exit 0;;' + echo ' "user create") echo CREATE; exit 0;;' + echo ' "user --help"|"env --system") exit 0;;' + echo 'esac' + echo 'exit 2' + } > "$R/pm"; chmod 755 "$R/pm" + printf 'name|app|share|dir|desktop|launcher|groups\nzz-no-such-acct|/bin/true|no|no|no|no|-\n' > "$R/conf" + APP_USERS_CONF="$R/conf" APP_USERS_PAM="$R/etc/pam.d/su_u_user" \ + APP_USERS_PROFILE_D="$R/etc/profile.d" APP_USERS_PM="$R/pm" \ + bash "$_au" 2>&1 | grep -cE '^CREATE$' )" + [ "${_cro:-0}" -ge 1 ] \ + || _p="$_p;a declared account that does not exist is not created" + + # THE COMMAND IS `packagemanager USER setup`. The first version called + # `packagemanager setup`, which is an ALIAS OF BOOTSTRAP -- so it did not + # fail as an unknown command, it reached bootstrap's parser and said + # "unrecognized arguments", three times. The stub in the first test said + # yes to anything, which is why the test passed (1.14.218). + grep -q 'set -- user setup' "$_au" \ + || _p="$_p;the setup command is missing its 'user' subcommand" + grep -q '$PM user setup --help' "$_au" \ + || _p="$_p;it does not check the command exists before the loop" + # ...and the stub below REFUSES anything that is not a real subcommand + # path, so a wrong command fails the test instead of passing it + # and the merge, for real: an existing file with one of the accounts + _pr="$T/au"; rm -rf "$_pr"; mkdir -p "$_pr/etc/pam.d" "$_pr/etc/profile.d" + { + echo '#!/bin/sh' + echo 'case "$1 $2" in' + echo ' "config --show") case "$3" in main-user) echo tester;; user-prefix) echo u_;; esac; exit 0;;' + echo ' "user setup") exit 0;;' + echo ' "user create") exit 0;;' + echo ' "user --help") exit 0;;' + echo ' "env --system") echo "export PKGUSR_SYS_SHARED=\"/zz/hub\""; exit 0;;' + echo 'esac' + echo 'echo "STUB refused: $*" >&2; exit 2' + } > "$_pr/pm"; chmod 755 "$_pr/pm" + printf '%s\n' '#!/bin/bash' 'case "$PAM_USER" in' \ + ' u_claude) USERS="u_claude tester"; ;;' \ + ' u_keepme) USERS="u_keepme tester"; ;;' \ + ' *) exit 1 ; ;;' 'esac' > "$_pr/etc/pam.d/su_u_user" + printf 'name|app|share|dir|desktop|launcher|groups\nclaude|/bin/true|slim|no|no|no|-\nzznew|/bin/true|full|no|no|no|-\n' \ + > "$_pr/conf.conf" + APP_USERS_CONF="$_pr/conf.conf" APP_USERS_PAM="$_pr/etc/pam.d/su_u_user" \ + APP_USERS_PROFILE_D="$_pr/etc/profile.d" APP_USERS_PM="$_pr/pm" \ + bash "$_au" >"$_pr/out" 2>&1 + [ "$(grep -c 'u_claude)' "$_pr/etc/pam.d/su_u_user")" = 1 ] \ + || _p="$_p;an account already in the PAM file was duplicated" + grep -q 'u_keepme)' "$_pr/etc/pam.d/su_u_user" \ + || _p="$_p;a hand-written PAM entry was lost in the merge" + grep -q 'u_zznew)' "$_pr/etc/pam.d/su_u_user" \ + || _p="$_p;a new account was not added to the PAM file" + grep -q 'STUB refused' "$_pr/out" \ + && _p="$_p;it called a subcommand that does not exist: $(grep -m1 'STUB refused' "$_pr/out")" + ls "$_pr/etc/pam.d/" | grep -q 'bak-' \ + || _p="$_p;the PAM file was edited with no backup beside it" + # the env fragment enumerates rather than lists + grep -q '"/zz/hub"/\*' "$_pr/etc/profile.d/shared-dirs.sh" 2>/dev/null \ + || _p="$_p;the SHARED_* variables are a fixed list, not what is there" + # ...and what it enumerates is THE HUB the tool names, not $HOME/Shared: + # in an app account that is its own data directory, and every FILE in it + # became a SHARED_* variable (1.14.312) + grep -v '^[[:space:]]*#' "$_pr/etc/profile.d/shared-dirs.sh" 2>/dev/null \ + | grep -q 'HOME/Shared\|HOME"/Shared' \ + && _p="$_p;the SHARED_* variables are still read from \$HOME/Shared" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "application users come from a file, and neither list-file is clobbered" +fi + +# ---- the handover selects by NUMBER (1.14.216) ----------------------------- # +# 7.2's `chown -R root:root` is refused on a chapter-8 tree (it would strip +# every package user), and the precise version selects files by the HOST's +# lfs uid instead -- the only thing that means the same on both sides of a +# chroot. Book 4.3 pins no uid for `lfs`, and package users here are +# created in the 10000-20000 range: if those numbers ever meet, a package +# user's files are handed to root and it loses all of them silently. +_p="" +if [ -f "$LFS_TOOL" ]; then + _hb="$(_slice_def "$LFS_TOOL" _handover_build_user_only)" + [ -n "$_hb" ] || _p="$_p;the precise handover cannot be read" + printf '%s' "$_hb" | grep -q 'etc", "passwd"' \ + || _p="$_p;it does not check the tree's passwd for that uid" + printf '%s' "$_hb" | grep -q 'REFUSING to hand files back' \ + || _p="$_p;a uid collision is not refused" + printf '%s' "$_hb" | grep -q 'chown -h root {}' \ + || _p="$_p;the handover follows symlinks or takes the group too" + printf '%s' "$_hb" | grep -q -- '-xdev' \ + || _p="$_p;the handover can cross into another filesystem" + # and it must refuse for real, with a tree whose passwd collides + rm -rf "$T/ho"; mkdir -p "$T/ho/etc" "$T/ho/usr" + printf 'root:x:0:0::/root:/bin/bash\np_light:x:10753:10753::/x:/bin/bash\n' \ + > "$T/ho/etc/passwd" + cat > "$T/ho/probe.py" <<'PYHO' +import sys, types, io, contextlib +mod = types.ModuleType("lfsmod"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +mod._build_user_uid = lambda: 10753 +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + n = mod._handover_build_user_only(sys.argv[2], run=True) +print("touched=%d" % n) +print("refused" if "REFUSING" in buf.getvalue() else "NOT-REFUSED") +PYHO + _ho="$(python3 "$T/ho/probe.py" "$LFS_TOOL" "$T/ho" 2>&1)" + printf '%s\n' "$_ho" | grep -qx 'refused' \ + || _p="$_p;a colliding uid is not refused in practice: $(printf '%s' "$_ho" | tr '\n' ' ')" + printf '%s\n' "$_ho" | grep -qx 'touched=0' \ + || _p="$_p;it changed files despite the collision" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the handover refuses when the build uid is also a package user's" +fi + +# ---- a missing python backend is a package (1.14.210) ---------------------- # +# numpy died on `BackendUnavailable: Cannot import 'mesonpy'` after 60 lines +# of pip internals, the gate did not recognise it, and the page that +# provides `mesonpy` is called **meson_python** -- not guessable from the +# module name. numpy's Required in the book is exactly +# `cython-3.2.9, meson_python-0.20.0`, so a --recursive install would have +# built it. +# (its own variable: borrowing $_hl9 from the block below made this block +# die with "unbound variable" when run alone -- the third time a borrowed +# variable has done that) +_p="" +_hl10="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl10" ]; then + grep -qF "BackendUnavailable: Cannot import" "$_hl10" \ + || _p="$_p;a missing python build backend is not recognised as a failure" + grep -q 'mesonpy) echo meson_python' "$_hl10" \ + || _p="$_p;the page that provides mesonpy is not named" + # AUTO-INSTALL IS OPT-IN, AND REFUSES TEST-ONLY MODULES. Installing + # what a failure names would have installed pytest for numpy, run a + # multi-minute test suite inside an install, and hidden the generator + # bug that put the test there (1.14.214). + _am="$(_slice_fn "$_hl10" _install_missing_module)" + [ -n "$_am" ] || _p="$_p;there is no way to install a missing build backend" + printf '%s' "$_am" | grep -q '_PY_TEST_ONLY' \ + || _p="$_p;a test-only module would be installed to satisfy an install" + printf '%s' "$_am" | grep -q 'NOT installing it' \ + || _p="$_p;it does not refuse a test dependency out loud" + grep -q 'auto_modules=1' "$_hl10" \ + || _p="$_p;there is no flag for it" + grep -q 'auto_modules="${PM_AUTO_MODULES:-0}"' "$_hl10" \ + || _p="$_p;auto-install is on by default, which changes the system unasked" + # reported on the SUCCESS path as well, which is when nobody looks + grep -q 'installed on the way (--auto-modules), not asked for' "$_hl10" \ + || _p="$_p;packages installed on the way are not reported" + [ "$(grep -c 'installed on the way (--auto-modules)' "$_hl10")" -ge 2 ] \ + || _p="$_p;it is reported only when the build fails" + # and the three outcomes, for real. The slices go into a FILE and the + # file is run: an eval nested inside a command substitution inside a + # double-quoted assignment is four levels of quoting, and getting one + # wrong made every case come out "refused" -- a test that fails for its + # own reasons teaches nothing. + { + echo 'say(){ :; }; note(){ :; }; warn(){ :; }' + echo 'unprefix_pkg_user(){ echo "${1#p_}"; }' + echo 'name=p_numpy' + echo 'packagemanager(){ return 0; }' + _slice_fn "$_hl10" _missing_py_module + awk '/^_PY_TEST_ONLY=/,/flaky"/' "$_hl10" + _slice_fn "$_hl10" _py_module_provider + _slice_fn "$_hl10" _install_missing_module + echo '_AUTO_MODULES_LOG="$1"; : > "$_AUTO_MODULES_LOG"' + echo 'printf "No module named %s\n" "'"'"'pytest'"'"'" > "$2/l1"' + echo '_install_missing_module "$2/l1" && echo pytest=INSTALLED || echo pytest=refused' + echo 'printf "Cannot import %s\n" "'"'"'mesonpy'"'"'" > "$2/l2"' + echo '_install_missing_module "$2/l2" && echo mesonpy=installed || echo mesonpy=REFUSED' + echo 'printf "No module named %s\n" "'"'"'zzunknown'"'"'" > "$2/l3"' + echo '_install_missing_module "$2/l3" && echo unknown=INSTALLED || echo unknown=refused' + echo 'grep -q meson_python "$1" && echo recorded=yes || echo recorded=NO' + } > "$T/am_probe.sh" + _amo="$(bash "$T/am_probe.sh" "$T/am.log" "$T" 2>&1)" + for _want in "pytest=refused" "mesonpy=installed" "unknown=refused" "recorded=yes"; do + printf '%s\n' "$_amo" | grep -qx "$_want" \ + || _p="$_p;auto-modules: expected $_want, got: $(printf '%s' "$_amo" | tr '\n' ' ')" + done + grep -q 'a python module is missing, not a permission' "$_hl10" \ + || _p="$_p;the failure box does not say what kind of failure it is" + # the extraction, on both wordings + for _pair in "Cannot import 'mesonpy'|mesonpy" \ + "ModuleNotFoundError: No module named 'numpy'|numpy"; do + printf '%s\n' "${_pair%|*}" > "$T/pylog" + _got="$(grep -aoE "Cannot import '[a-zA-Z0-9_.]+'|No module named '[a-zA-Z0-9_.]+'" \ + "$T/pylog" | head -n1 | cut -d"'" -f2)" + [ "$_got" = "${_pair#*|}" ] \ + || _p="$_p;'${_pair%|*}' parses to '$_got'" + done +fi +if [ -f "$pm_src" ]; then + # AND THE DEPENDENCY CHECK MUST NOT SWALLOW ITS OWN FAILURE: silence + # there is indistinguishable from "nothing missing" + # the SHAPE, not the phrase: the comment explaining this bug quotes the + # old code, and matching the quote matched the explanation (the fourth + # time prose has failed a check about code) + grep -vE '^\s*#' "$pm_src" | grep -q 'except (SystemExit, Exception):' \ + && _p="$_p;a failed dependency lookup is still silent" + grep -q 'could not work out what' "$pm_src" \ + || _p="$_p;a failed dependency lookup does not say so" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a missing python backend names its package, and a failed dep lookup is not silence" +fi + +# ---- the second retry must not re-clone (1.14.209) ------------------------- # +# opencv needed two directory grants (licenses, then cmake). Round 1 +# retried `install` only -- correct, the build was at 100%. Round 2 then +# ran `phase all` and re-cloned and rebuilt the whole tree, because +# LAST_FAILED_PHASE is cleared at the top of run_phase_as and was only SET +# in its all|update branch: a single-phase run left it empty, and cmd_build +# fell back to $phase. +_p="" +_hl9="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl9" ]; then + _rpa="$(_slice_fn "$_hl9" run_phase_as)" + _hb10="$(grep -A14 'IS A COMPOUND PHASE TOO' "$_hl9")" + printf '%s' "$_rpa" | grep -q '\*) LAST_FAILED_PHASE="$phase"' \ + || _p="$_p;a single-phase run does not record which phase failed" + # `update` IS A COMPOUND PHASE TOO. The mapping covered `all` only, so + # a failure inside `update` retried `update`: poppler's retry, after a + # grant for /usr/share/gir-1.0, re-unpacked the tarball and re-ran cmake + # with 145 files already installed (1.14.248). + printf '%s' "$_hb10" | grep -q 'all|update)' \ + || _p="$_p;a failure inside update re-runs the whole update" + _upd="$( phase=update; LAST_FAILED_PHASE=install + _r="$phase" + case "$phase" in all|update) + case "${LAST_FAILED_PHASE:-}" in + install|configure) _r="$LAST_FAILED_PHASE" ;; esac ;; esac + echo "$_r" )" + [ "$_upd" = install ] \ + || _p="$_p;update+failed install retries '$_upd', not install" + # and the decision it feeds: install stays install across rounds + _dec="$( phase=install; rc=1; LAST_FAILED_PHASE="" + case "$phase" in all|update) ;; *) LAST_FAILED_PHASE="$phase" ;; esac + _r="$phase" + [ "$phase" = all ] && case "${LAST_FAILED_PHASE:-}" in + install|configure) _r="$LAST_FAILED_PHASE" ;; esac + echo "$_r" )" + [ "$_dec" = install ] \ + || _p="$_p;a failed install retry resolves to '$_dec', not install" + # the comment that explains why must name the cost, not just the rule + grep -q 'reached 100%' "$_hl9" \ + || _p="$_p;nothing records what the full rebuild cost" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a second repair round retries the failed phase, it does not re-clone" +fi + +# ---- one list, two entry points (1.14.223) --------------------------------- # +# The media chain is the part that gets rebuilt -- a codec appears, a plugin +# set was configured without a feature, webkitgtk moves a version -- and it +# lived inside a 158-entry desktop stack. Copying it into a second file +# would give two lists of the same packages, and two lists drift: +# gst10-libav sat in one of them for three releases before anyone noticed. +# So `include <stack>` splices one stack into another. +_p="" +_ms="$(dirname "$LFS_TOOL")/stacks/media.stack" +_sw3="$(dirname "$LFS_TOOL")/stacks/sway.stack" +if [ -f "$_ms" ] && [ -f "$_sw3" ]; then + grep -qE '^include +media' "$_sw3" \ + || _p="$_p;sway.stack does not include the media stack" + # the ENTRIES must exist once, in media.stack, and not be copied back + for _e in webkitgtk gst10-plugins-bad gst10-libav opus; do + [ "$(grep -cE "^(book|git|tar) +$_e( |$)" "$_sw3")" = 0 ] \ + || _p="$_p;$_e is written in sway.stack as well as media.stack" + grep -qE "^(book|git|tar) +$_e( |$)" "$_ms" \ + || _p="$_p;$_e is in neither stack" + done + # A STATUS IS NOT A PLAN. On a system built from an older book, half + # the tree reads "(update available)" and the summary said "building + # next: ..." -- so `stack media --run` looked like it was about to + # rebuild glib2 and everything under it. A run builds what is MISSING + # and never touches an installed package (1.14.225). + grep -q 'to build (missing)' "$pm_src" \ + || _p="$_p;the status still presents missing packages as a build plan" + grep -q 'Installed packages are NOT' "$pm_src" \ + || _p="$_p;nothing says a stack run leaves installed packages alone" + # the SHAPE: the phrase in a printed string, not in the comment that + # explains why it was dropped (fifth time prose has failed a code check) + grep -vE '^\s*#' "$pm_src" | grep -q 'installed {u.version} (update available)' \ + && _p="$_p;'update available' still implies the stack will update it" + # ...and a single update names the requirements that differ, which is the + # real question behind "can I update webkitgtk without glib2" + _sr2="$(_slice_def "$pm_src" _report_stale_requires)" + [ -n "$_sr2" ] || _p="$_p;a single update says nothing about stale requirements" + printf '%s' "$_sr2" | grep -q 'not a refusal' \ + || _p="$_p;it refuses instead of reporting, which breaks the one workflow" + printf '%s' "$_sr2" | grep -q '_same_version' \ + || _p="$_p;it compares version strings without the book's spelling rules" + rm -rf "$T/sr"; mkdir -p "$T/sr/pkgusr/p_zzwk" "$T/sr/pkgusr/p_zzlib" + printf "name='zzwk'\nname_version='ZzWk-1.0'\nrequired=('ZzLib-9.9')\n" \ + > "$T/sr/pkgusr/p_zzwk/install_last" + echo x > "$T/sr/pkgusr/p_zzwk/pkg.lst" + printf "name='zzlib'\nname_version='ZzLib-1.0'\n" \ + > "$T/sr/pkgusr/p_zzlib/install_last" + echo x > "$T/sr/pkgusr/p_zzlib/pkg.lst" + _sro="$(PKGUSR_BASE=$T/sr PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYSR' +import sys, types, io, contextlib +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +buf = io.StringIO() +with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + mod._report_stale_requires(["zzwk"]) +out = buf.getvalue() +print("NAMED" if "zzlib" in out and "ZzLib-1.0" in out else "MISSED:%s" % out[:120]) +PYSR +)" + printf '%s\n' "$_sro" | grep -q NAMED \ + || _p="$_p;a requirement older than the book is not named: $_sro" + # THE MAJOR VERSION IS PART OF THE NAME. gtk3 and gtk4 both record + # name_version 'GTK-<v>', so the base 'GTK' matched whichever was + # indexed first and `GTK-4.20.3` resolved to **gtk3** -- webkitgtk + # listed gtk3 twice and gtk4 not at all. Same collision hid libsoup-3.x + # behind libsoup, sdl3 behind sdl, qt6 behind qt (1.14.226). + _rdo="$(python3 - "$pm_src" "$T/rd" 2>&1 <<'PYRD' +import sys, types, os +base = os.path.join(sys.argv[2], "pkgusr") +for name, nv in (("gtk3", "GTK-3.24.51"), ("gtk4", "GTK-4.20.3"), + ("libsoup3", "libsoup-3.6.6"), ("opus", "Opus-1.6.1"), + ("sdl3", "SDL3-3.4.0"), ("lcms2", "Little-CMS-2.18")): + d = os.path.join(base, "p_" + name) + os.makedirs(d, exist_ok=True) + open(os.path.join(d, "install_last"), "w").write( + "name='%s'\nname_version='%s'\n" % (name, nv)) +os.environ["PKGUSR_BASE"] = sys.argv[2] +os.environ["PKGUSR_CONFIG"] = "/dev/null" +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +inst = mod._pkg_dirs() +bad = [] +for dep, want in (("GTK-3.24.51", "gtk3"), ("GTK-4.20.3", "gtk4"), + ("libsoup-3.6.6", "libsoup3"), ("Opus-1.6.1", "opus"), + ("SDL3-3.4.0", "sdl3"), ("Little-CMS-2.18", "lcms2")): + got = mod._resolve_dep(dep, inst) + if got != want: + bad.append("%s -> %s (want %s)" % (dep, got, want)) +print("BAD: " + "; ".join(bad) if bad else "OK") +PYRD +)" + [ "$_rdo" = OK ] || _p="$_p;dependency resolution: $_rdo" + printf '%s' "$_sr2" | grep -q '_row not in rows' \ + || _p="$_p;the same package can be listed twice in the report" + # A TRUNCATED LIST MUST SAY HOW TO SEE THE REST, and there must be a way + # to see every package the book has moved past, not just the one asked + # about (1.14.227). + # (the literal is wrapped across two source lines, so match the part + # that is on one of them -- a pattern spanning a line break matches + # nothing and the assertion fails on correct code) + printf '%s' "$_sr2" | grep -q 'all of them: packagemanager' \ + || _p="$_p;'... and N more' is a dead end" + printf '%s' "$_sr2" | grep -q '_verbose' \ + || _p="$_p;-v does not show the full requirement list" + grep -q 'everything the book has moved past' "$pm_src" \ + || _p="$_p;nothing points at the whole-system update list" + # THE HINT MUST BE A COMMAND THAT WORKS. It printed + # `packagemanager update webkitgtk -v` and -v was a TOP-LEVEL flag only, + # so the tool answered "unrecognized arguments: -v" (1.14.228). + timeout 60 python3 "$pm_src" update --help 2>&1 | grep -q -- '\[-v\]' \ + || _p="$_p;-v is not accepted where the hint says to type it" + # A MIDDLE GROUND BETWEEN ONE PACKAGE AND EVERYTHING: a stack's own + # packages. `stack media --run` builds what is missing and updates + # nothing; `update webkitgtk` is one package; between them there was + # only a list of names to type. + timeout 60 python3 "$pm_src" update --help 2>&1 | grep -q -- '--stack NAME' \ + || _p="$_p;there is no way to update just one stack's packages" + _st2="$(_slice_def "$pm_src" _stack_targets)" + printf '%s' "$_st2" | grep -q '"book", "git", "tar"' \ + || _p="$_p;--stack would try to update check/cfg entries" + # 2>/dev/null and tail -1: _stack_targets PRINTS its refusal, so the + # message arrived before the verdict and the comparison read it as the + # answer + _sto="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")/stacks/media.stack" 2>/dev/null <<'PYST' +import sys, types, os +os.environ["PKGUSR_CONFIG"] = "/dev/null" +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +t = mod._stack_targets(sys.argv[2]) or [] +bad = [] +if "webkitgtk" not in t: bad.append("webkitgtk missing") +if "webrtc" in t: bad.append("the check entry is treated as a package") +if mod._stack_targets(sys.argv[2] + ".nope") is not None: + bad.append("a missing stack is not refused") +print("BAD: " + "; ".join(bad) if bad else "OK") +PYST +)" + _sto="$(printf '%s\n' "$_sto" | tail -1)" + [ "$_sto" = OK ] || _p="$_p;--stack targets: $_sto" + # and the plan must say what it will change, in its own line + grep -q 'this plan updates %d package' "$pm_src" \ + || _p="$_p;the plan does not state which packages it updates" + grep -q 'a comparison, not a list of work' "$pm_src" \ + || _p="$_p;the requirement report can still be read as a plan" + # A DRY RUN ASKS NOTHING. `update --stack media` prompted four times, + # once per already-current package, before printing a plan it was never + # going to carry out (1.14.229). + grep -q 'A DRY RUN ASKS NOTHING' "$pm_src" \ + || _p="$_p;a preview still prompts for already-current packages" + _cu="$(_slice_def "$pm_src" cmd_update)" + # -B8: the guard is the `continue` branch above the prompt, and three + # lines of it are the comment explaining why + printf '%s' "$_cu" | grep -B8 'Re-apply the update anyway' \ + | grep -q 'getattr(args, "run", False)' \ + || _p="$_p;the up-to-date prompt is not gated on --run" + # A GIT-TRACKED PACKAGE HAS NO VERSION TO COMPARE. gst-plugins-rs is a + # git entry following upstream; its name_version is + # `gst-plugins-rs-git`, which against the book's 1.28.6 read as "older" + # -- so the plan offered to replace a cargo-built checkout with the + # book's page, silently. + printf '%s' "$_cu" | grep -q 'endswith("-git")' \ + || _p="$_p;a git-tracked package is compared against a book version" + printf '%s' "$_cu" | grep -q 'git_tracked.append' \ + || _p="$_p;a git-tracked package is not reported as left alone" + # (1.14.307 replaced "left alone" with an answer: a branch is asked + # whether it moved, a tag whether a newer one exists. These two checks + # still looked for the old wording until 1.14.313.) + grep -q 'follow GIT, not the book' "$pm_src" \ + || _p="$_p;nothing says why those packages are not in the plan" + grep -q 'behind their branch -- fetch, rebuild, install' "$pm_src" \ + || _p="$_p;it does not say how to move a git-tracked package" + # ...and that command really does default to everything + timeout 60 python3 "$pm_src" update --help 2>&1 \ + | grep -q 'default: all installed' \ + || _p="$_p;bare \`update\` does not list every installed package" + # THE SKIP MARKER IS ONE CONSTANT. _enable_one_alternative matched + # "## OPTIONAL, skipped" and 1.14.212 renamed the marker, so webkitgtk's + # script came out with BOTH build blocks commented: a build phase of `:` + # and an install phase running ninja against no build.ninja. + _bl2="$(dirname "$LFS_TOOL")/blfs" + if [ -f "$_bl2" ]; then + grep -q '^SKIP_MARK = ' "$_bl2" \ + || _p="$_p;the skip marker is a literal in more than one place" + grep -q '_is_skip_line' "$_bl2" \ + || _p="$_p;the alternative picker matches the marker by hand" + _alt="$(python3 - "$_bl2" <<'PYALT' +import sys, types +mod = types.ModuleType("b"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +# both blocks skipped, under the CURRENT marker: one must come back +cmds = ("## SKIPPED -- the book says: If you want the GTK-3 version:\n" + "# mkdir -vp build &&\n# cd build &&\n# cmake -D PORT=GTK ..\n# ninja\n" + "## SKIPPED -- the book says: If you want the GTK-4 version:\n" + "# rm -rf * &&\n# cmake -D USE_GTK4=ON ..\n# ninja\n") +out = mod._enable_one_alternative(cmds) +live = [l for l in out.split("\n") + if l.strip() and not l.strip().startswith("#") and l.strip() != ":"] +print("OK" if any("cmake" in l for l in live) else "NONE-ENABLED") +PYALT +)" + [ "$_alt" = OK ] \ + || _p="$_p;a page of two alternatives still builds neither: $_alt" + fi + + # THE EXCLUSIONS ARE A PROPERTY OF THE MACHINE, not of one stack. When + # the media chain moved out, it lost all 87 `avoid` entries -- so + # `stack media --status` planned 274 packages including systemd, polkit + # and qt6, on a SysV machine that must never have systemd. avoid.stack + # is included by both (1.14.224). + _av="$(dirname "$LFS_TOOL")/stacks/avoid.stack" + [ -f "$_av" ] || _p="$_p;there is no shared avoid list" + # EVERY stack, not the two that were split. network, services and + # vision never got the include, so `stack services --status` planned + # systemd just as media had -- the same bug, three files over + # (1.14.235). + for _f in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do + case "$(basename "$_f")" in avoid.stack) continue ;; esac + grep -qE '^include +avoid' "$_f" \ + || _p="$_p;$(basename "$_f") does not include the avoid list" + [ "$(grep -c '^avoid ' "$_f")" = 0 ] \ + || _p="$_p;$(basename "$_f") still carries its own avoid lines" + done + # A PACKAGE MUST NOT BE BOTH BUILT AND AVOIDED. `avoid openssh` sat in + # avoid.stack while network.stack built it with the install-sshd + # bootscript -- two files contradicting each other, and whichever a + # plan saw first won. Checked for every stack now (1.14.241). + _clash="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")"/stacks/*.stack <<'PYCL' +import sys, types, os +os.environ["PKGUSR_CONFIG"] = "/dev/null" +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +bad = [] +for f in sys.argv[2:]: + if os.path.basename(f) == "avoid.stack": + continue + rows = mod._parse_stack_file(f) + build = {r["target"] for r in rows if r["kind"] in ("book", "git", "tar")} + avoid = {r["target"] for r in rows if r["kind"] == "avoid"} + both = sorted(build & avoid) + if both: + bad.append("%s builds and avoids %s" % (os.path.basename(f), ", ".join(both))) +print("BAD: " + "; ".join(bad) if bad else "OK") +PYCL +)" + _clash="$(printf '%s\n' "$_clash" | tail -1)" + [ "$_clash" = OK ] || _p="$_p;$_clash" + + # INSTALLING PAM IS NOT ENOUGH, AND WARNING ABOUT MODES IS NOT ENOUGH. + # A rebuild of shadow strips /usr/bin/su's setuid bit (the BLFS page + # never sets it; the LFS one does), so su was broken after the very + # rebuild that made PAM usable. `cfg shadow-pam` does both halves and + # FIXES the modes rather than reporting them (1.14.252). + _sp="$(dirname "$LFS_TOOL")/stacks/shadow-pam.sh" + _sc="$(dirname "$LFS_TOOL")/stacks/setid.conf" + # set HERE: only an earlier block set it, so a --only selection that + # did not pull that line in died on `set -u` + _hlv="$(dirname "$LFS_TOOL")/lfs-helper" + if [ -f "$_sp" ] && [ -f "$_sc" ]; then + bash -n "$_sp" || _p="$_p;shadow-pam.sh does not parse" + grep -qE '^cfg +shadow-pam' \ + "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + || _p="$_p;nothing rebuilds shadow after linux-pam" + grep -q 'update shadow --reinstall --regenerate' "$_sp" \ + || _p="$_p;it does not rebuild shadow against pam" + grep -q 'libpam\\.so' "$_sp" \ + || _p="$_p;it does not check whether su actually links pam" + # A PAM-LINKED su WITH NO /etc/pam.d/su IS WORSE THAN A NON-PAM ONE: + # su: pam_start: error 26 (PAM_ABORT -- no service file) + # `update` installs and does NOT configure, and those files come + # from the CONFIGURE phase of linux-pam and shadow (1.14.253). + grep -q 'script configure' "$_sp" \ + || _p="$_p;a pam-linked su is left with no /etc/pam.d/su to read" + # CONFIG BEFORE REBUILD. Between a pam-linked su and its config + # file, NOTHING can be built: every build runs as its package user + # through `su - p_<pkg>`, including the configure phases that would + # write the missing file. Root survives it only because pam_rootok + # is sufficient (1.14.254). + _i_cfg="$(grep -n 'script configure' "$_sp" | head -1 | cut -d: -f1)" + _i_reb="$(grep -n 'update shadow --reinstall' "$_sp" | head -1 | cut -d: -f1)" + [ -n "$_i_cfg" ] && [ -n "$_i_reb" ] && [ "$_i_cfg" -lt "$_i_reb" ] \ + || _p="$_p;shadow is rebuilt before its pam config is written" + grep -q 'NOTHING can be built' "$_sp" \ + || _p="$_p;nothing records why that order matters" + # the helpers must be defined before the first step that calls them + _i_def="$(grep -n '^_pam_installed()' "$_sp" | head -1 | cut -d: -f1)" + _i_use="$(grep -n 'if _pam_installed; then' "$_sp" | head -1 | cut -d: -f1)" + [ -n "$_i_def" ] && [ -n "$_i_use" ] && [ "$_i_def" -lt "$_i_use" ] \ + || _p="$_p;_pam_installed is called before it is defined" + # THE SET, NOT A SAMPLE: su and system-auth existed while fourteen + # other files did not, and it said "in place" (1.14.262). + grep -q '^_book_pam_set() {' "$_sp" \ + || _p="$_p;shadow-pam checks a sample of the pam.d files, not the set" + grep -q 'script fetch "$1"' "$_sp" \ + || _p="$_p;shadow-pam does not read the package's own script" + grep -q '_fetch_script "$_cfgpkg" --regenerate' "$_sp" \ + || _p="$_p;a stale script that names no pam file is run again as it is" + # power.sh read packagemanager.conf (key=value) with a JSON pattern + # and stopped the stack on the booted machine (1.14.265). The + # prefix comes from the resolver, exported to cfg scripts. + _pws="$(dirname "$LFS_TOOL")/stacks/power.sh" + grep -q '"collector_prefix"' "$_pws" \ + && _p="$_p;power.sh parses the conf with the lfs config's JSON syntax" + grep -q 'LFS_COLLECTOR_PREFIX' "$_pws" \ + || _p="$_p;power.sh does not read the exported collector prefix" + grep -q '_cenv\["LFS_COLLECTOR_PREFIX"\]\|("collector_prefix", "LFS_COLLECTOR_PREFIX")' \ + "$(dirname "$LFS_TOOL")/packagemanager" \ + || _p="$_p;cfg scripts are not given the resolved collector prefix" + # ID RANGES (1.14.267): collector groups at 90000+ with no account, + # application users at 30000+, power's group not a system gid, and + # a symlinked binary judged by its target. + _pmf="$(dirname "$LFS_TOOL")/packagemanager" + sed -n '/^def cmd_nimgnu_create/,/^def /p' "$_pmf" | grep -q 'add_package_user\|find_add_package_user' \ + && _p="$_p;sysgroup create makes an account in the package-user range" + sed -n '/^def cmd_user_create/,/^def /p' "$_pmf" | grep -q 'APP_UID_MIN' \ + || _p="$_p;application users are created in the package-user range" + grep -q 'groupadd -r' "$_pws" \ + && _p="$_p;power.sh makes its collector group with a system gid" + grep -q 'readlink -f' "$_pws" \ + || _p="$_p;power.sh judges a symlink instead of its target" + grep -q '/etc/rc.d/rcS.d' "$_pws" \ + || _p="$_p;power.sh does not know LFS's rcS.d" + # GLVND (1.14.270): machine.conf enables it, so libglvnd must come + # before mesa, and --clean must never delete a file that moved to + # another package (libGL.so.1 goes from mesa to libglvnd). + _swy="$_DESKTOP_STACK" + _mcf="$(dirname "$LFS_TOOL")/stacks/machine.conf" + _l_g="$(grep -nE '^git +libglvnd' "$_swy" | head -1 | cut -d: -f1)" + _l_m="$(grep -nE '^book +mesa' "$_swy" | head -1 | cut -d: -f1)" + if grep -qE '^glvnd *= *enabled' "$_mcf"; then + [ -n "$_l_g" ] && [ -n "$_l_m" ] && [ "$_l_g" -lt "$_l_m" ] \ + || _p="$_p;mesa is built with glvnd but libglvnd does not come before it" + fi + sed -n '/^def _orphan_sweep/,/^def /p' "$_pmf" | grep -q 'st.st_uid != own_uid' \ + || _p="$_p;--clean can delete a file another package now owns" + # A MOVE OF MANY FILES IS ONE ROUND (1.14.271): libglvnd taking + # Mesa's headers went one file per round and was called a bundled + # subproject. meson's install plan hands the rest over at once, and + # only a subprojects/ source counts as a bundled copy. + _lhf="$(dirname "$LFS_TOOL")/lfs-helper" + sed -n '/^meson_takeover_from_log() {/,/^}/p' "$_lhf" | grep -q 'intro-installed.json' \ + || _p="$_p;a file move between packages still takes one meson round per file" + sed -n '/^meson_takeover_from_log() {/,/^}/p' "$_lhf" | grep -q '/subprojects/' \ + || _p="$_p;any one-file handover streak is called a bundled subproject" + # AN EXTRA DOWNLOAD UNDER ITS OWN NAME (1.14.272): the patch was + # fetched with -O $pkg, wget said 416, and patch found no file. + grep -qF '_pp_fetch "$li" "$f"' "$(dirname "$LFS_TOOL")/lfs-phases" \ + || _p="$_p;additional downloads are saved under the tarball's name" + # A FAILED BUILD MUST NOT ADVANCE THE VERSION (1.14.273): a package + # with no .pkgusr-installed had its version read from install_last, + # which is replaced before the build. + grep -q 'RECORD WHAT IS INSTALLED BEFORE OVERWRITING THE RECORD' "$_lhf" \ + || _p="$_p;a legacy package's failed build still advances its version" + # SCANS PRUNE; CMAKE'S WRAPPED ERROR IS READ (1.14.274) + sed -n '/^snapshot() {/,/^}/p' "$_lhf" | grep -q -- '-prune' \ + || _p="$_p;the tree scan walks /usr/src and filters afterwards" + grep -q '_cmakedirs=' "$_lhf" \ + || _p="$_p;a permission error CMake wraps over two lines is not read" + # su_u_user CHECKS THE CALLER, root OWNS IT, AND A CACHED cfg IS + # RE-CHECKED (1.14.275) + _aus="$(dirname "$LFS_TOOL")/stacks/app-users.sh" + grep -qF -- "echo '[[ \" \$USERS \" == *\" \$PAM_RUSER \"* ]]'" "$_aus" \ + || _p="$_p;su_u_user compares the target, not the caller (anyone gets in)" + grep -q 'chown root:root "$PAM_FILE"' "$_aus" \ + || _p="$_p;su_u_user, which root runs, is not forced to root ownership" + grep -q 'cached as done, but its check fails' "$_pmf" \ + || _p="$_p;a cached cfg entry is never re-checked" + grep -qE '^cfg +app-users +\\$' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + || _p="$_p;cfg app-users has no expect= to re-check" + # REGROUP A DIRECTORY (1.14.276): a command, dispatched, that asks + # and shows a plan before it changes anything + grep -q '^ regroup-dir) shift; cmd_regroup_dir' "$_lhf" \ + || _p="$_p;lfs-helper has no regroup-dir" + sed -n '/^cmd_regroup_dir() {/,/^}/p' "$_lhf" | grep -q "apply? \[y/N\]" \ + || _p="$_p;regroup-dir changes a directory without confirmation" + # KERNEL OPTIONS THIS MACHINE WANTS SURVIVE A CONFIG REFRESH (1.14.277) + grep -q '^CONFIG_SET=.*LEGACY_TIOCSTI=n' "$(dirname "$LFS_TOOL")/kernel.conf" \ + || _p="$_p;kernel.conf does not disable legacy TIOCSTI (su loses job control)" + grep -q 'CONFIG_SET did not take' "$(dirname "$LFS_TOOL")/lfs-kernel" \ + || _p="$_p;lfs-kernel applies CONFIG_SET without reading it back" + # FIRMWARE PATTERNS AT ANY DEPTH, FLAT NAMES FROM WHENCE (1.14.279) + grep -q '^_fw_whence_links() {' "$(dirname "$LFS_TOOL")/lfs-kernel" \ + || _p="$_p;flat firmware names (WHENCE Link:) are never created" + grep -q -- '-name "$d" -print' "$(dirname "$LFS_TOOL")/lfs-kernel" \ + || _p="$_p;a FIRMWARE_ONLY pattern only matches the top level" + # AFTER ITS FIRST INSTALL A PACKAGE CLAIMS ONLY -tmp/-pass STAGES + # (1.14.280: "gave 'p_llvm' 13 path(s)" was su_u_user and the tools) + sed -n '/^_claim_earlier_stages() {/,/^}/p' "$_lhf" | grep -q '_installed_before' \ + || _p="$_p;an installed package still claims root files from its own manifest" + # THE INSTALLED VERSION IS RECORDED ON EVERY PATH THAT FINISHES + # (1.14.281): `--phase all` never wrote .pkgusr-installed, and + # VERSION could not parse name_version='...'. + [ "$(_slice_fn "$_lhf" cmd_build | grep -c '_record_installed "$name" "$owner" "$s"')" -ge 2 ] \ + || _p="$_p;a completed --phase all install does not record its version" + _ri_out="$( _rt="$(mktemp -d)"; mkdir -p "$_rt/home" + printf "name_version='Zz-2.0'\n" > "$_rt/script" + eval "$(_slice_fn "$_lhf" _record_installed)" + pkgusr_home_for() { echo "$_rt/home"; }; real_chown() { :; } + _record_installed zz p_zz "$_rt/script" + cat "$_rt/home/.pkgusr-installed"; head -1 "$_rt/home/VERSION"; rm -rf "$_rt" )" + [ "$_ri_out" = "Zz-2.0 +Zz-2.0" ] || _p="$_p;_record_installed does not write both records from name_version='...' (got: $_ri_out)" + # EXISTING APPLICATION ACCOUNTS MOVE ON `user setup` TOO (1.14.282) + sed -n '/^def cmd_user_setup/,/^def /p' "$_pmf" | grep -q '_relocate_app_home(u)' \ + || _p="$_p;an existing application account is never moved into its subdirectory" + # `packagemanager stack` WITH NO NAME LISTS THE STACKS (1.14.284) + _sl="$(cd "$(dirname "$LFS_TOOL")" && PKGUSR_CONFIG=/dev/null python3 packagemanager stack 2>/dev/null)" + printf '%s\n' "$_sl" | grep -qE '^ sway +[0-9]+/[0-9]+ done' \ + || _p="$_p;packagemanager stack without a name does not list the stacks" + # NO UNDEFINED NAMES IN cmd_install (1.14.287: `ignore` was + # build_install_plan's local, used in cmd_install) + _pyf="$(python3 -m pyflakes --version >/dev/null 2>&1 && echo y)" + if [ -n "$_pyf" ]; then + # every undefined name in the Python tools (STACK_DIR is read + # through a globals() guard on purpose) + _un="$(cd "$(dirname "$LFS_TOOL")" && python3 -m pyflakes packagemanager lfs blfs 2>/dev/null \ + | grep "undefined name" | grep -v "'STACK_DIR'")" + [ -z "$_un" ] || _p="$_p;undefined names in the Python tools: $(printf '%s' "$_un" | tr '\n' ' ')" + else + sed -n '/^def cmd_install(/,/^def /p' "$_pmf" | grep -q 'ignore=ignore)' \ + && _p="$_p;cmd_install passes a name it never defines (ignore)" + fi + # THE SCRIPT LAST INSTALLED, NOT THE FIRST IN THE ALPHABET (1.14.288) + _vs="$( _vd="$(mktemp -d)"; touch "$_vd/install_Mesa-26.1.7"; sleep 0.01; touch "$_vd/install_Mesa-25.3.5" + printf "name_version='Mesa-26.1.7'\n" > "$_vd/install_last" + cd "$(dirname "$LFS_TOOL")" && PKGUSR_CONFIG=/dev/null python3 -c " +import types,sys +m=types.ModuleType('pm'); exec(compile(open('packagemanager').read(),'pm','exec'), m.__dict__) +print(m._versioned_script_in(sys.argv[1]))" "$_vd" 2>/dev/null | tail -1; rm -rf "$_vd" )" + case "$_vs" in */install_Mesa-26.1.7) ;; *) _p="$_p;script runs the alphabetically first version, not the installed one ($_vs)" ;; esac + # THE su TRAP: shadow's configure writes su WITHOUT the pam_exec + # line, so app-users must run after shadow-pam to put it back. + _svs="$(dirname "$LFS_TOOL")/stacks/services.stack" + _i_sp="$(grep -nE '^cfg +shadow-pam' "$_svs" | head -1 | cut -d: -f1)" + _i_au="$(grep -nE '^cfg +app-users' "$_svs" | head -1 | cut -d: -f1)" + [ -n "$_i_sp" ] && [ -n "$_i_au" ] && [ "$_i_sp" -lt "$_i_au" ] \ + || _p="$_p;app-users does not run after shadow-pam, so shadow's su loses the pam_exec line" + grep -q 'grep -qF "$(basename "$PAM_FILE")" "$PAM_SU"' \ + "$(dirname "$LFS_TOOL")/stacks/app-users.sh" \ + || _p="$_p;app-users would not notice su lost its pam_exec line" + grep -q 'pam.d/system-auth' "$_sp" \ + || _p="$_p;system-auth is not checked, and su needs it too" + grep -q 'pam_start' "$_sp" \ + || _p="$_p;nothing records the error that missing file produces" + # EVERY BUILD IS A `su - p_<pkg>`, so a pam_unix lookup failure is + # not cosmetic: add_package_user writes /etc/passwd and leaves no + # /etc/shadow entry, and pam_unix says "cannot retrieve + # authentication info" on every package build for ever (1.14.255). + grep -q 'pwconv' "$_sp" \ + || _p="$_p;package users have no shadow entry and pam says so on every build" + grep -q 'cp -p /etc/shadow' "$_sp" \ + || _p="$_p;/etc/shadow is rewritten without a backup" + # NO /etc/shadow AT ALL was the case the guard missed -- and it is + # the system that needs this most: pam_unix has nowhere to look for + # ANY account (1.14.257). pwconv CREATES the file from + # /etc/passwd, which is a change to how the machine authenticates, + # so passwd is what gets backed up and root's line is checked. + grep -q 'elif \[ ! -e /etc/shadow \]' "$_sp" \ + || _p="$_p;a system with no /etc/shadow is skipped in silence" + grep -q 'cp -p /etc/passwd "$_pwbak"' "$_sp" \ + || _p="$_p;/etc/passwd is changed by pwconv without a backup" + grep -q "grep -q '\^root:' /etc/shadow" "$_sp" \ + || _p="$_p;it does not check root survived the conversion" + grep -q 'cp -p "$_pwbak" /etc/passwd' "$_sp" \ + || _p="$_p;a failed conversion is not rolled back" + grep -q 'if login is broken' "$_sp" \ + || _p="$_p;it does not say how to undo it by hand" + grep -q 'still without a shadow entry' "$_sp" \ + || _p="$_p;it does not read back whether the entries appeared" + grep -q 'UNVERIFIED FROM WHERE THIS WAS WRITTEN' "$_sp" \ + || _p="$_p;the unverified part is not marked unverified" + # ORDER: chown clears the setuid bit, so chmod must come last + _ci="$(grep -n 'chown' "$_sp" | head -1 | cut -d: -f1)" + _cm="$(grep -n 'chmod "$mode"' "$_sp" | head -1 | cut -d: -f1)" + [ -n "$_ci" ] && [ -n "$_cm" ] && [ "$_ci" -lt "$_cm" ] \ + || _p="$_p;chmod runs before chown, which clears the setuid bit" + grep -q 'su.*| *4711 * | *root' "$_sc" \ + || _p="$_p;setid.conf does not declare su as setuid root" + # shadow's install asks for NINE bits and the first list declared + # five. These four came off that request list (1.14.258). + for _f in chfn chsh newgidmap newuidmap; do + grep -q "/$_f *|" "$_sc" \ + || _p="$_p;$_f is missing from setid.conf, so its bit stays lost" + done + # APPLY, DO NOT RECITE: report_setid_requests runs as root right + # after the install, which is the one moment the missing thing can + # simply be done. It printed nine chmod lines instead. + _rsr="$(_slice_fn "$_hlv" report_setid_requests)" + printf '%s' "$_rsr" | grep -q 'cmd_setid --run' \ + || _p="$_p;the setid requests are printed and not applied" + printf '%s' "$_rsr" | grep -q 'id -u.*= 0' \ + || _p="$_p;it would try to apply them as a non-root caller" + printf '%s' "$_rsr" | grep -q 'SOME COULD NOT BE APPLIED' \ + || _p="$_p;a failed apply is not reported" + # A BACKTICK INSIDE DOUBLE QUOTES RUNS THE COMMAND. The success + # line said `lfs-helper setid` and therefore EXECUTED it, printing a + # second full report in the middle of a sentence (1.14.259). + printf '%s' "$_rsr" | grep -q '`lfs-helper setid`' \ + && _p="$_p;a backtick in the success line runs lfs-helper again" + # KEEP THE GROUP: only the USER must be root for a setuid bit to + # mean root, and the group is how this system records which package + # owns a file. It forced root:root. + _cs2="$(_slice_fn "$_hlv" cmd_setid)" + printf '%s' "$_cs2" | grep -q '_setid_apply "$mode" root "$f"' \ + && _p="$_p;the setid apply throws away the package group" + printf '%s' "$_cs2" | grep -q '_grp_now' \ + || _p="$_p;it does not preserve the group the file already has" + # TWO TABLES, ONE ANSWER. The "what upstream installs" table + # hard-coded `root`, so it ran right after the request table and + # undid it: `su is now 4755 root:p_shadow` followed by `su is now + # 4755 root:root`, 17 changes to reach the wrong one (1.14.260). + _kn="$(_setid_table "$_hlv")" + printf '%s' "$_kn" | grep -q '^/usr/bin/su|[0-7]*|root|' \ + && _p="$_p;the upstream table still forces root:root on su" + printf '%s' "$_kn" | grep -q '^/usr/bin/su|[0-7]*|-|' \ + || _p="$_p;su's group is not marked as keep-what-is-there" + # ONE TABLE (1.14.263): lfs-helper said 4755 and setid.conf 4711, so + # shadow-pam and `setid --run` corrected each other. The function + # reads the file; no path may be written into it again. + _slice_fn "$_hlv" _setid_known | grep -q '^/usr/bin/' \ + && _p="$_p;_setid_known carries its own list again, beside setid.conf" + _slice_fn "$_hlv" cmd_setid | grep -q 'setid.conf decides this one' \ + || _p="$_p;a requested bit and setid.conf are both applied to one file" + # a setGID entry must keep its NAMED group -- there the group is the point + printf '%s' "$_kn" | grep -q 'dbus-daemon-launch-helper|4750|messagebus' \ + || _p="$_p;a setgid entry lost the group that is its purpose" + printf '%s' "$_cs2" | grep -q 'if \[ "$grp" = "-" \]' \ + || _p="$_p;the keep marker is not honoured when applying" + # the fixture file must carry a NON-root group, or "keep the group" + # and "force root" are indistinguishable (a root-created file is + # root:root either way -- the first version of this check compared + # nothing) + _sgo="$( + R="$T/sg"; rm -rf "$R"; mkdir -p "$R"; : > "$R/su"; chmod 755 "$R/su" + _othergrp="$(awk -F: '$1!="root" && $3>0 {print $1; exit}' /etc/group)" + chgrp "$_othergrp" "$R/su" 2>/dev/null || _othergrp=root + warn() { :; }; ok() { :; } + _setid_state() { stat -c '%a %U:%G' "$1"; } + eval "$(_slice_fn "$_hlv" _setid_apply)" + _setid_apply 4755 "$(stat -c '%G' "$R/su")" "$R/su" >/dev/null 2>&1 + stat -c '%a %U:%G' "$R/su" )" + case "$_sgo" in + "4755 root:root") _p="$_p;the group became root instead of staying put" ;; + "4755 root:"*) ;; # kept whatever it was + *) _p="$_p;setid apply produced '$_sgo'" ;; + esac + for _f in passwd chage gpasswd newgrp unix_chkpwd; do + grep -q "/$_f *|" "$_sc" \ + || _p="$_p;$_f is not in setid.conf, so its bit stays lost" + done + # and it must FIX, in a run, not just report + _spo="$( + R="$T/sp"; rm -rf "$R"; mkdir -p "$R/bin" + : > "$R/bin/su"; chmod 755 "$R/bin/su" + printf '%s | 4711 | root | -\n' "$R/bin/su" > "$R/setid.conf" + printf '#!/bin/sh\nexit 0\n' > "$R/pm"; chmod 755 "$R/pm" + SETID_CONF="$R/setid.conf" SHADOW_PAM_PM="$R/pm" \ + SHADOW_PAM_SU="$R/bin/su" bash "$_sp" >/dev/null 2>&1 + stat -c '%a' "$R/bin/su" 2>/dev/null )" + [ "$_spo" = 4711 ] \ + || _p="$_p;the setuid bit is reported but not applied (mode $_spo)" + else + _p="$_p;there is no step that makes su use pam and keeps its bits" + fi + + # ...and linux-pam is now a WANTED package, not an avoided one: su + # without a password needs pam_exec, which needs pam + grep -qE '^book +linux-pam' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + || _p="$_p;nothing installs linux-pam, so passwordless su cannot work" + grep -qE '^avoid +linux-pam' "$_av" \ + && _p="$_p;linux-pam is both built and avoided" + _lpo="$(python3 - "$pm_src" "$(dirname "$LFS_TOOL")/stacks/services.stack" <<'PYLP' +import sys, types, os +os.environ["PKGUSR_CONFIG"] = "/dev/null" +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +print("OK" if o.index("linux-pam") < o.index("app-users") + else "linux-pam is built after the accounts that need it") +PYLP +)" + [ "$_lpo" = OK ] || _p="$_p;$_lpo" + # the shadow rebuild is the dangerous half and must stay manual + grep -B14 -E '^book +linux-pam' "$(dirname "$LFS_TOOL")/stacks/services.stack" \ + | grep -q 'update shadow --reinstall' \ + || _p="$_p;nothing says shadow must be rebuilt for pam to be read" + grep -qE '^(book|git|tar) +shadow' "$(dirname "$LFS_TOOL")"/stacks/*.stack \ + && _p="$_p;shadow is rebuilt by a stack, which can lock you out of su" + _ac3="$(python3 - "$pm_src" "$_ms" "$_sw3" <<'PYAV' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +bad = [] +for f in sys.argv[2:]: + rows = mod._parse_stack_file(f) + av = [r["target"] for r in rows if r["kind"] == "avoid"] + for must in ("systemd", "qt6", "cups", "texlive", "openjdk"): + if must not in av: + bad.append("%s does not avoid %s" % (f.split("/")[-1], must)) + # INCLUDE ONCE: owning the "already spliced" set per include LINE meant + # avoid was spliced twice over, 332 entries where there are 158 + tg = [r["target"] for r in rows] + for t in ("qt6", "systemd"): + if tg.count(t) > 1: + bad.append("%s lists %s %d times" % (f.split("/")[-1], t, tg.count(t))) +print("BAD: " + "; ".join(bad) if bad else "OK") +PYAV +)" + [ "$_ac3" = OK ] || _p="$_p;$_ac3" + _io3="$(python3 - "$pm_src" "$_sw3" "$_ms" <<'PYINC' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +sw = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +me = [r["target"] for r in mod._parse_stack_file(sys.argv[3])] +problems = [] +# every media entry appears in the composed sway plan, exactly once +for t in me: + if sw.count(t) != 1: + problems.append("%s appears %d times in sway" % (t, sw.count(t))) +# and in the same relative order +if [t for t in sw if t in me] != me: + problems.append("the included entries are reordered") +# the include lands after what webkitgtk needs and before the compositor +for need in ("gtk4", "libsoup3", "mesa"): + if need in sw and sw.index(need) > sw.index("webkitgtk"): + problems.append("%s comes after webkitgtk" % need) +if "sway" in sw and sw.index("sway") < sw.index("webkitgtk"): + problems.append("the compositor is built before the media chain") +print("BAD: " + "; ".join(problems) if problems else "OK") +PYINC +)" + [ "$_io3" = OK ] || _p="$_p;include composition is wrong: $_io3" + # a cycle must be refused, not recursed + printf '# t\n# stack-version: 0\ninclude zzb\n' > "$T/zza.stack" + printf '# t\n# stack-version: 0\ninclude zza\n' > "$T/zzb.stack" + _cyc2="$(python3 - "$pm_src" "$T/zza.stack" 2>&1 <<'PYCY' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +try: + mod._parse_stack_file(sys.argv[2]) + print("ACCEPTED") +except ValueError as e: + print("circular" if "circular" in str(e) else "refused: %s" % e) +except RecursionError: + print("RECURSED") +PYCY +)" + case "$_cyc2" in + circular) ;; + *) _p="$_p;a circular include is not refused cleanly: $_cyc2" ;; + esac +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the media chain is one list, included by sway, and a cycle is refused" +fi + +# ---- a new stack keeps the conventions (1.14.207) -------------------------- # +# vision.stack is the first stack added since the rules grew, and a stack +# file that skips them fails silently: no stack-version means the +# one-release-one-number check has nothing to compare, and an out-of-book +# entry with no probe installs and says nothing about whether it works. +# +# Checked, not assumed: `git ls-remote` lists opencv branch 5.x AND tag +# 5.0.0, so the tag is pinned rather than the branch. +_p="" +for _sf in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do + [ -f "$_sf" ] || continue + _n="$(basename "$_sf")" + grep -q '^# stack-version:' "$_sf" \ + || _p="$_p;$_n has no stack-version line" + # every git/tar entry (an out-of-book one) needs a have= or the stack + # cannot tell installed from not, and an expect= or nothing proves it + _miss="$(python3 - "$pm_src" "$_sf" <<'PYCONV' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +for r in mod._parse_stack_file(sys.argv[2]): + if r["kind"] not in ("git", "tar"): + continue + o = r["opts"] + if not (o.get("have") or o.get("prog")): + print("NOSTATE %s" % r["target"]) + if not o.get("expect"): + print("NOPROBE %s" % r["target"]) +PYCONV +)" + case "$_miss" in + *NOSTATE*) _p="$_p;$_n: an out-of-book entry has no have= or prog=: ${_miss}" ;; + esac +done +# and the opencv entry specifically: pinned, and building where the package +# user owns the tree +_vs="$(dirname "$LFS_TOOL")/stacks/vision.stack" +if [ -f "$_vs" ]; then + grep -qE '^git +opencv' "$_vs" || _p="$_p;opencv is not in vision.stack" + grep -A4 -E '^git +opencv' "$_vs" | grep -q 'ref=5\.' \ + || _p="$_p;opencv follows a branch instead of a pinned tag" + grep -A8 -E '^git +opencv' "$_vs" | grep -q 'cmake -S . -B build' \ + || _p="$_p;opencv builds outside the tree the package user owns" + grep -A8 -E '^git +opencv' "$_vs" | grep -q 'OPENCV_GENERATE_PKGCONFIG=ON' \ + || _p="$_p;the .pc file upstream stopped generating is not asked for" + grep -A8 -E '^git +opencv' "$_vs" | grep -q 'cmake --build build' \ + || _p="$_p;the build ignores CMAKE_BUILD_PARALLEL_LEVEL" + grep -B32 -E '^git +opencv' "$_vs" | grep -q 'ls-remote' \ + || _p="$_p;nothing records how the ref was checked" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every stack file keeps the conventions, and opencv is pinned and probed" +fi + +# ---- info printed everything, then crashed (1.14.206) ---------------------- # +# `packagemanager info x264 openh264 gst10-plugins-ugly` printed all three +# packages and then: +# NameError: name '_states' is not defined +# Three bugs in two lines: `_states` was never initialised in cmd_info (it +# belongs to verify's loop, where the lines were copied from), `name` does +# not exist in that scope -- `anchor` does -- and the line was there twice. +# py_compile cannot see any of it: a NameError needs the line to RUN. +_p="" +if [ -f "$pm_src" ]; then + _ci="$(_slice_def "$pm_src" cmd_info)" + [ -n "$_ci" ] || _p="$_p;cmd_info cannot be read" + printf '%s' "$_ci" | grep -q '_states\[name\]' \ + && _p="$_p;cmd_info still assigns _states[name], which does not exist" + printf '%s' "$_ci" | grep -q 'counts, _states = {}, {}' \ + || _p="$_p;_states is used in cmd_info without being initialised" + # and it RUNS: a name error only shows up when the line executes + rm -rf "$T/ib"; mkdir -p "$T/ib/pkgusr/p_zzdemo" + printf "name='zzdemo'\nname_version='zzdemo-1'\n" \ + > "$T/ib/pkgusr/p_zzdemo/install_last" + _io="$(PKGUSR_BASE=$T/ib PKGUSR_CONFIG=/dev/null \ + python3 "$pm_src" info zzdemo 2>&1)" + case "$_io" in + *Traceback*|*NameError*) _p="$_p;info still crashes: $(printf '%s' "$_io" | tail -1)" ;; + esac + printf '%s' "$_io" | grep -q 'Total:' \ + || _p="$_p;info does not reach its own summary line" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "info runs to the end, and names the packages that need attention" +fi + +# ---- one release, one number (1.14.194) ------------------------------------ # +# The stack files carry `# stack-version:` and it DRIFTED: network and +# services said 1.14.167 and sway said 1.14.188 while the tools were at +# 1.14.193, because each release bumped them with a sed matching the exact +# previous number -- so any release that did not touch a stack broke the +# chain for every release after it. +# +# They ship as one artifact, so they carry one number, and this asserts it +# rather than trusting a sed. +_p="" +_ver="$(python3 "$LFS_TOOL" --version 2>/dev/null \ + | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)" +if [ -n "$_ver" ]; then + for _sf in "$(dirname "$LFS_TOOL")"/stacks/*.stack; do + [ -f "$_sf" ] || continue + _sv="$(sed -n 's/^# stack-version: *//p' "$_sf" | head -1)" + if [ -z "$_sv" ]; then + _p="$_p;$(basename "$_sf") carries no stack-version" + elif [ "$_sv" != "$_ver" ]; then + _p="$_p;$(basename "$_sf") says $_sv, the tools say $_ver" + fi + done +else + _p="$_p;the tool version cannot be read, so nothing can be compared to it" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every stack file carries the same version as the tools" +fi + # ---- every tool reports its version ----------------------------------------- # # Without one there is no way to tell whether the copy inside the chroot is the # one that was just fixed -- and several rounds of diagnosis were spent on @@ -6782,22 +8979,25 @@ if [ -f "$pmt" ]; then && ok "a shared user joins the session group" \ || bad "a shared user cannot reach the display or audio" jx="$(sed -n '/^def _join_xdg_runtime_group/,/^def _setup_application_user/p' "$pmt")" - # the group must be READ from the directory, never assumed by name + # THE GROUP IS NAMED (1.14.286, the user's decision): <collector + # prefix>_xdg-runtime, created in the collector range, joined by the main + # user and every shared account. Reading it off the directory found the + # person's own group, refused, and nothing ever set one up. + grep -q '^def xdg_runtime_group' "$pmt" \ + && ok "the session group has one name, from the collector prefix" \ + || bad "the session group is not named" case "$jx" in - *"getgrgid(os.stat(rt).st_gid)"*) - ok "the group is read from the runtime directory" ;; - *) bad "the session group name is hardcoded" ;; + *"_create_collector_group(g)"*) ok "the session group is created in the collector range" ;; + *) bad "the session group is never created" ;; esac case "$jx" in - *'u_xdg_runtime"'*) bad "a specific group name is hardcoded" ;; - *) ok "no group name is baked in" ;; - esac - # a personal group is not a shared group - case "$jx" in - *"which is not a shared "*) - ok "it refuses to add the user to a personal group" ;; - *) bad "an application user could join the person's own group" ;; + *'for who in (main, u)'*) ok "the main user joins it too (chgrp needs membership)" ;; + *) bad "the main user is not in the session group" ;; esac + _aus2="$(dirname "$LFS_TOOL")/stacks/app-users.sh" + grep -q 'cat > /usr/bin/xdg-runtime-share' "$_aus2" \ + && ok "the session applies the group (xdg-runtime-share)" \ + || bad "nothing applies the group to a runtime directory made at boot" # both /run/user/<uid> and /tmp/xdg-<user> layouts xd="$(sed -n '/^def _xdg_runtime_dir_of/,/^def _join_xdg/p' "$pmt")" for p in "/run/user" "/tmp/xdg-"; do @@ -6855,13 +9055,83 @@ if [ -f "$pmt" ]; then ok "nothing is written when the program is not installed" ;; *) bad "a launcher could point at a program that does not exist" ;; esac - # and neither file is clobbered if it is already there + # a hand-written file is kept; one of ours that went stale is rewritten + # (1.14.289: ~/bin/browser-mini kept running a renamed-away u_browser) case "$de" in - *"already exists -- left alone"*) ok "an existing menu entry is kept" ;; - *) bad "the desktop entry would be overwritten" ;; + *'"Comment=Runs as the application user " not in cur'*) ok "a hand-written menu entry is kept" ;; + *) bad "the desktop entry would overwrite a hand-written one" ;; esac -fi - + wl="$(sed -n '/^def _write_launcher/,/^def cmd_user_launcher/p' "$pmt")" + # (1.14.292) the launcher builds the argument list itself: {} is what + # you type, a search term inside a URL, nothing without {} + _lt="$( _lh="$(mktemp -d)"; PKGUSR_CONFIG=/dev/null python3 - "$pmt" "$_lh" <<'PY' 2>/dev/null +import types, sys, os +m = types.ModuleType("pm"); exec(compile(open(sys.argv[1]).read(), "pm", "exec"), m.__dict__) +h = sys.argv[2]; m._run = lambda c: True; m._home_of = lambda u: h; m._app_home = lambda u: "/home/x/" + u +m._write_launcher("me", "u_a", "browser-mini https://s.example/r?q={}", "srch") +m._write_launcher("me", "u_a", "browser-mini https://fixed.example/", "fixed") +print(h) +PY +)" + _lt="$(printf '%s\n' "$_lt" | tail -1)" + if [ -f "$_lt/bin/srch" ]; then + # NEUTRALISE su BY PATH, NOT BY SED. + # + # This rewrote the launcher's last line with `sed '/^exec su - /'`, + # which assumed the launcher ends in exactly one way. Since 1.14.295 + # there are TWO ways out of it -- `exec su` with a terminal, script(1) + # with none -- and the `exec su` is indented inside the if, so the sed + # matched neither: the REAL su ran and the block failed with "user u_a + # does not exist". A stub first in PATH tests whatever shape the + # launcher has, including both of these. + # + # su's own arguments are `- <user> -c <command> --`, five of them, + # and what follows is the argument list the launcher built. + mkdir -p "$_lt/fake" + printf '#!/bin/bash\nshift 5\nprintf "<%%s>" "$@"\n' > "$_lt/fake/su" + chmod 755 "$_lt/fake/su" + # a pty turns \n into \r\n; the stub prints none, but strip it anyway + _runl() { PATH="$_lt/fake:$PATH" bash "$_lt/bin/$1" "${@:2}" 2>/dev/null | tr -d '\r'; } + [ "$(_runl srch a b)" = "<browser-mini><--app-id><srch><https://s.example/r?q=a+b>" ] \ + && [ "$(_runl srch)" = "<browser-mini><--app-id><srch><https://s.example/r>" ] \ + && [ "$(_runl fixed extra)" = "<browser-mini><--app-id><fixed><https://fixed.example/>" ] \ + && ok "the launcher passes {} as arguments or a search term, and nothing without {}" \ + || bad "the launcher's argument handling is wrong: $(_runl srch a b) / $(_runl srch) / $(_runl fixed extra)" + # A MENU ENTRY HAS NO TERMINAL (1.14.295). su fails without one and + # its message goes nowhere, so the entry "does nothing". The launcher + # must notice and give it a pty. + if grep -q 'if \[ -t 0 \] && \[ -t 1 \]' "$pmt" && grep -q 'script -qec' "$pmt"; then + ok "with no terminal the launcher gives su a pty" + else + bad "the launcher runs su with no pty: every .desktop entry does nothing" + fi + # ...and a default page for a search launcher that was given nothing + # to search for (1.14.293): {https://site/} rather than the URL cut + # at its ? + _lt2="$( _lh2="$(mktemp -d)"; PKGUSR_CONFIG=/dev/null python3 - "$pmt" "$_lh2" <<'PY' 2>/dev/null +import types, sys +m = types.ModuleType("pm"); exec(compile(open(sys.argv[1]).read(), "pm", "exec"), m.__dict__) +h = sys.argv[2]; m._run = lambda c: True; m._home_of = lambda u: h; m._app_home = lambda u: "/home/x/" + u +m._write_launcher("me", "u_a", "browser-mini https://s.example/r?q={https://s.example/}", "dflt") +print(h) +PY +)" + _lt2="$(printf '%s\n' "$_lt2" | tail -1)" + _rund() { PATH="$_lt/fake:$PATH" bash "$_lt2/bin/dflt" "$@" 2>/dev/null | tr -d '\r'; } + [ "$(_rund)" = "<browser-mini><--app-id><dflt><https://s.example/>" ] \ + && [ "$(_rund x)" = "<browser-mini><--app-id><dflt><https://s.example/r?q=x>" ] \ + && ok "with nothing typed, a {url} default page wins over cutting at the ?" \ + || bad "the {url} default page is wrong: $(_rund) / $(_rund x)" + rm -rf "$_lt" "$_lt2" + else + bad "the launcher could not be generated" + fi + case "$wl" in + *"was not written by these tools -- left alone"*) ok "a hand-written launcher is kept, a stale one of ours rewritten" ;; + *) bad "a stale launcher is never rewritten" ;; + esac +fi + # ---- changing LFS_TGT invalidates the toolchain ----------------------------- # # gcc-pass2 installs its C++ headers under $LFS_TGT. Change LFS_TGT afterwards # and the compiler looks under the new name while the headers sit under the @@ -8275,7 +10545,7 @@ if [ -f "$pmt" ]; then import importlib.machinery as m, os pm = m.SourceFileLoader('pm', '$pmt').load_module() print(pm.BASE_DIR, pm.appuser_home())" 2>/dev/null)" - [ "$got" = "/usr/src /usr/src" ] \ + [ "$got" = "/usr/src /home/shared_user" ] \ && ok "an unconfigured system keeps the existing layout" \ || bad "the layout changed without being asked (got '$got')" printf 'collector_prefix=n\npkgusr_home=/usr/src/package_users\nappuser_home=/home/shared_users\n' \ @@ -8857,7 +11127,19 @@ if not hasattr(lfs, "_auto_file_block"): print(" FAIL a section that ships a file instead of commands is ignored") sys.exit(1) -raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() +try: + raw = open(sys.argv[2], encoding='utf-8', errors='replace').read() +except OSError as e: + print(" SKIP rc.site needs a book on disk (%s)" % e.strerror) + sys.exit(77) +# A CHECK'S PRECONDITION IS NOT A FAILURE. rc.site is a SysV section: the +# systemd book has no ch-config-site at all, so this reported FAIL on a +# machine where nothing was wrong, and every clean run had to be read with +# "except that one" in mind. Say what is missing instead (1.14.147). +if "ch-config-site" not in raw: + print(" SKIP this book has no rc.site section (SysV only) -- " + "build one with lfs-sysvbook, or point the suite at a SysV book") + sys.exit(77) blk = lfs._auto_file_block(raw, "ch-config-site", "/etc/sysconfig/rc.site") if not blk: problems.append("rc.site is not extracted from the book") @@ -9906,8 +12188,9 @@ helper_src="$(dirname "$LFS_TOOL")/lfs-helper" if [ -f "$helper_src" ]; then _cb="$(_slice_fn "$helper_src" cmd_build)" _p="" + # (1.14.274: pruned rather than filtered -- either form excludes it) case "$_cb" in - *'-not -path "$_bld/*"'*) ;; + *'-not -path "$_bld/*"'*|*'-path "$_bld" \) -prune'*) ;; *) _p="$_p;the build scratch is still scanned into every manifest" ;; esac case "$_cb" in @@ -11919,7 +14202,7 @@ grep -q '"/etc/pkgusr/stacks"' "$LFS_TOOL" \ || _p="$_p;an installed lfs cannot find the stack files to refresh" grep -q 'no stack files found to refresh' "$LFS_TOOL" \ || _p="$_p;finding no stacks to refresh is silent" -_stk="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_stk="$_DESKTOP_STACK" if [ -f "$_stk" ]; then # The recommendations are FOLLOWED again: a blanket --ignore-recommended # dropped things the builds hard-require (curl for cmake, icu for @@ -16627,7 +18910,7 @@ if command -v git >/dev/null 2>&1; then || _p="$_p;a valid ref no longer checks out" fi # the stack's own pin was the wrong one and is corrected -_ss="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ss="$_DESKTOP_STACK" [ -f "$_ss" ] && { grep -q 'usrsctp.git ref=1.0.0' "$_ss" \ && _p="$_p;the stack still pins a usrsctp tag that does not exist"; } if [ -n "$_p" ]; then @@ -17960,7 +20243,7 @@ fi # get their dependencies from the book, git entries got nothing. _p="" _pmU="$(dirname "$LFS_TOOL")/packagemanager" -_ssU="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ssU="$_DESKTOP_STACK" # the stack parser keeps needs=, and the sway entry declares its own if [ -f "$_ssU" ]; then python3 - "$_pmU" "$_ssU" <<'PYNEEDS' || _p="$_p;needs= is not parsed from a stack entry" @@ -18058,7 +20341,7 @@ printf '%s' "$_cr" | grep -q 'y pins" if bad == 1 else "ies pin' \ sed -n '/if getattr(args, "check_refs", False)/,+2p' "$_pmV" | grep -q 'sys.exit' \ || _p="$_p;--check-refs falls through into building the stack" # and the shipped stack no longer pins a ref that never existed -_ssV="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ssV="$_DESKTOP_STACK" [ -f "$_ssV" ] && { grep -q 'ref=1.24.1' "$_ssV" \ && _p="$_p;the stack still pins foot at a tag that does not exist"; } if [ -n "$_p" ]; then @@ -18090,7 +20373,7 @@ printf '%s' "$_nb" | grep -q 'unreachable += 1' \ printf '%s' "$_nb" | grep -q 'timeout=' \ || _p="$_p;the no-ref probe can hang" # the four local entries point at real repositories, not a placeholder -_ssW="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ssW="$_DESKTOP_STACK" if [ -f "$_ssW" ]; then grep -qE '^git +(minibrowser|swbr|swov|swas).*github\.com/nimbin2' "$_ssW" \ && _p="$_p;the local sway tools still point at the old placeholder urls" @@ -18791,8 +21074,11 @@ printf '%s' "$_rs" | grep -q 'setuid/setgid' \ || _p="$_p;the report does not say what kind of request it was" _slice_fn "$_hlpU" cmd_build | grep -q 'report_setid_requests "$owner"' \ || _p="$_p;the build never reports them" -# it prints each line and then clears the file, so a later build does not -# repeat requests that have been dealt with +# it prints each line -- and KEEPS the ledger. Clearing it here made this a +# notice you had to be watching for: `su` lost its setuid bit at install time +# and six hours later nothing on the system could say so (1.14.147). The +# file is now cleared by `setid --run`, once the bit has been applied and +# read back. mkdir -p "$T/sidrep" printf 'chmod 2755 /usr/bin/fcrontab\n' > "$T/sidrep/.pkgusr-setid.requests" _o="$( warn() { echo "$*" >&2; } @@ -18803,8 +21089,12 @@ case "$_o" in *"chmod 2755 /usr/bin/fcrontab"*) ;; *) _p="$_p;the refused command is not shown" ;; esac +case "$_o" in + *"lfs-helper setid"*) ;; + *) _p="$_p;the report does not say how to apply them later" ;; +esac [ -s "$T/sidrep/.pkgusr-setid.requests" ] \ - && _p="$_p;the requests are repeated on every later build" + || _p="$_p;the ledger is emptied by the report, so nothing can ask later" if [ -n "$_p" ]; then printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do [ -n "$m" ] && bad "$m" @@ -18813,6 +21103,2133 @@ else ok "a refused setuid/setgid bit is written down and reported to root" fi +# ---- the setid ledger, and su (1.14.147) ----------------------------------- # +# `su -` said +# setgid: Operation not permitted +# on a system where everything else worked. shadow installs su with +# install -m 4755 ... /usr/bin/su +# and the install WRAPPER downgraded that to 755 -- correctly, a package user +# may not set it -- but, unlike the chmod wrapper beside it, wrote nothing +# down. Same refusal, one of two places implementing the rule. (And a +# package user CAN setuid its own file: the result runs as the package user, +# which is worse than no bit at all.) +_p="" +_hlpS="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hlpS" ]; then + # the wrapper is written by make-wrappers as a heredoc; extract and run it + mkdir -p "$T/instw/home" "$T/instw/dest" + python3 - "$_hlpS" "$T/instw/install" <<'PYIN' +import sys, re +src = open(sys.argv[1], encoding="utf-8", errors="replace").read() +m = re.search(r'cat > "\$WRAPPERS/install" <<\'EOF\'\n(.*?)\nEOF\n', src, re.S) +open(sys.argv[2], "w").write(m.group(1) if m else "") +PYIN + if [ ! -s "$T/instw/install" ]; then + _p="$_p;the install wrapper could not be extracted" + else + chmod 755 "$T/instw/install" + bash -n "$T/instw/install" || _p="$_p;the install wrapper does not parse" + grep -q 'pkgusr-setid.requests' "$T/instw/install" \ + || _p="$_p;install -m 4755 is downgraded without recording it" + # a setid install records the file that ACTUALLY landed + echo x > "$T/instw/su" + ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 su "$T/instw/dest/su" ) >/dev/null 2>&1 + [ -f "$T/instw/dest/su" ] || _p="$_p;the file was not installed at all" + grep -q "chmod 4755 $T/instw/dest/su" "$T/instw/home/.pkgusr-setid.requests" 2>/dev/null \ + || _p="$_p;the refused setuid bit was not written down" + # into a DIRECTORY: the request must name the installed path, not the source + ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 su "$T/instw/dest/" ) >/dev/null 2>&1 + grep -q "chmod 4755 $T/instw/dest/su" "$T/instw/home/.pkgusr-setid.requests" 2>/dev/null \ + || _p="$_p;installing into a directory records the wrong path" + # an ordinary mode is not recorded + : > "$T/instw/home/.pkgusr-setid.requests" + ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 644 su "$T/instw/dest/su2" ) >/dev/null 2>&1 + [ -s "$T/instw/home/.pkgusr-setid.requests" ] \ + && _p="$_p;an ordinary install is recorded as a set-id request" + # a FAILED install must not leave a request behind: root would chmod + # a file that is not there + : > "$T/instw/home/.pkgusr-setid.requests" + ( cd "$T/instw" && HOME="$T/instw/home" bash ./install -m 4755 nosuchfile "$T/instw/dest/nope" ) >/dev/null 2>&1 + [ -s "$T/instw/home/.pkgusr-setid.requests" ] \ + && _p="$_p;a failed install still records a request" + fi + # A RECORDER MUST NOT BE THE REASON A BUILD FAILS. + # + # The ledger was left root:root by a root-side truncation, the install + # wrapper (running as the package user) could not append, and the SHELL + # printed "Permission denied" for the failed redirection -- which the + # swallowed-failure check then treated as a failed install. openssh had + # finished installing (1.14.192). + _sr="$(_slice_fn "$_hlpS" _sid_record)" + [ -n "$_sr" ] || _p="$_p;the recorder still writes without checking it can" + printf '%s' "$_sr" | grep -q '\[ -w "$f" \]' \ + || _p="$_p;it redirects first and lets the shell report the failure" + printf '%s' "$_sr" | grep -q 'could not record' \ + || _p="$_p;a refusal it cannot record is silent" + # `printf >> f 2>/dev/null || echo` does NOT work: prove the shape + _rt="$T/ro"; rm -rf "$_rt"; mkdir -p "$_rt"; : > "$_rt/led"; chmod 400 "$_rt/led" + if [ "$(id -u)" = 0 ]; then + : # root can write a 400 file; the check below would prove nothing + else + _bad="$( { printf 'x\n' >> "$_rt/led" 2>/dev/null || echo NOTE; } 2>&1 )" + case "$_bad" in + *"Permission denied"*) ;; # exactly the trap, still true + *) _p="$_p;the redirection trap this guards against no longer exists" ;; + esac + fi + # and every root write hands the ledger back to its user + grep -q 'chown "$owner:$owner" "$_sidf"' "$_hlpS" \ + || _p="$_p;a root truncation leaves a ledger the package user cannot write" + # (the whole file: $_sid is a _slice_fn slice and cmd_setid contains a + # line starting with }, so the slice ends before this) + grep -q 'gave $u back' "$_hlpS" \ + || _p="$_p;setid does not repair a ledger an older release left root-owned" + + # the command that applies them + _sid="$(_slice_fn "$_hlpS" cmd_setid)" + [ -n "$_sid" ] || _p="$_p;there is no way to apply the refused bits" + printf '%s' "$_sid" | grep -q '_setid_apply' \ + || _p="$_p;setid does not use the one applier" + _ap="$(_slice_fn "$_hlpS" _setid_apply)" + # A SETUID FILE OWNED BY A PACKAGE USER RUNS AS THAT PACKAGE USER, which + # is how `su` came to be setuid something that is not root. The mode + # without the chown is not a fix. + printf '%s' "$_ap" | grep -q 'chown "root:' \ + || _p="$_p;the mode is set without giving the file to root" + printf '%s' "$_ap" | grep -q '_setid_state' \ + || _p="$_p;setid does not read back what it changed" + printf '%s' "$_ap" | grep -q 'nosuid' \ + || _p="$_p;a nosuid mount is not mentioned when the bit does not stick" + # su is checked even when no request was recorded (an older build) + _kn="$(_setid_table "$_hlpS")" + for _f in /usr/bin/su /usr/bin/passwd /usr/bin/newgrp; do + printf '%s' "$_kn" | grep -q "^$_f|" \ + || _p="$_p;$_f is not among the programs checked on disk" + done + # every flag the report suggests must be one the command accepts + _rs2="$(_slice_fn "$_hlpS" report_setid_requests)" + case "$_rs2" in + *"setid --run"*) + printf '%s' "$_sid" | grep -q -- '--run' \ + || _p="$_p;the suggested --run is not a flag setid accepts" ;; + esac + # it is dispatched, and it is in the help + grep -q ' setid)' "$_hlpS" || _p="$_p;setid is not dispatched" + grep -q 'lfs-helper setid' "$_hlpS" || _p="$_p;setid is not in the help" + # and verify says so instead of waiting to be asked + _slice_fn "$_hlpS" cmd_verify | grep -q 'setid' \ + || _p="$_p;verify is silent about refused setid bits" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a refused setuid bit is recorded by BOTH wrappers and applied by setid" +fi + +# ---- lfs-backup: back up, restore without replacing (1.14.291) ------------ # +# lfs-config (a git-repo manager) became lfs-backup: one config of paths, +# an archive with modes and owner NAMES, symlinks stored as their files, and a +# restore that creates what is missing, leaves identical files alone and puts +# everything else beside the live file as .new -- account databases always. +_p="" +_bk="$(dirname "$LFS_TOOL")/lfs-backup" +if [ -f "$_bk" ] && [ "$(id -u)" = 0 ]; then + bash -n "$_bk" || _p="$_p;lfs-backup does not parse" + grep -qE '^TOOLS[[:space:]]*:?=.*lfs-backup' "$(dirname "$LFS_TOOL")/Makefile" \ + || _p="$_p;lfs-backup is not installed with the other tools" + grep -qE '^TOOLS[[:space:]]*:?=.*lfs-config' "$(dirname "$LFS_TOOL")/Makefile" \ + && _p="$_p;the retired lfs-config is still installed" + _B="$T/bk"; rm -rf "$_B"; mkdir -p "$_B/old/etc/profile.d" "$_B/new/etc/profile.d" \ + "$_B/old/home/u1/.config/a" "$_B/old/home/u1/d" "$_B/new/home/u2" "$_B/old/root" "$_B/new/root" + printf 'root:x:0:0::/root:/bin/bash\nu1:x:1500:1500::/home/u1:/bin/bash\n' > "$_B/old/etc/passwd" + printf 'root:x:0:\nu1:x:1500:\n' > "$_B/old/etc/group" + printf "export PKGUSR_SYS_MAIN_USER='u1'\n" > "$_B/old/etc/profile.d/pkgusr-system.sh" + printf 'root:x:0:0::/root:/bin/bash\nu2:x:1600:1600::/home/u2:/bin/bash\n' > "$_B/new/etc/passwd" + printf 'root:x:0:\nu2:x:1600:\n' > "$_B/new/etc/group" + printf "export PKGUSR_SYS_MAIN_USER='u2'\n" > "$_B/new/etc/profile.d/pkgusr-system.sh" + echo "real" > "$_B/old/home/u1/d/a.conf"; ln -s ../../d/a.conf "$_B/old/home/u1/.config/a/conf" + echo "old fstab" > "$_B/old/etc/fstab"; echo "new fstab" > "$_B/new/etc/fstab" + printf 'root:secret::\n' > "$_B/old/etc/shadow"; chmod 600 "$_B/old/etc/shadow" + chown -R 1500:1500 "$_B/old/home/u1"; chown 1600:1600 "$_B/new/home/u2" + printf '[user]\n~/.config/a/conf\n[system]\n/etc/fstab\n/etc/shadow\n' > "$_B/conf" + export LFS_BACKUP_CONF="$_B/conf" + LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/a.tgz" >/dev/null 2>&1 \ + || _p="$_p;create failed" + [ "$(stat -c %a "$_B/a.tgz" 2>/dev/null)" = 600 ] || _p="$_p;the archive (it holds secrets) is not mode 600" + tar -xzOf "$_B/a.tgz" files/user/.config/a/conf 2>/dev/null | grep -qx real \ + || _p="$_p;a symlink was not stored as the file it points to" + _out="$(LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/a.tgz" --run 2>&1)" + grep -qx real "$_B/new/home/u2/.config/a/conf" 2>/dev/null \ + || _p="$_p;a missing file was not created in the new main user's home" + [ "$(stat -c %u "$_B/new/home/u2/.config/a/conf" 2>/dev/null)" = 1600 ] \ + || _p="$_p;the old main user's file does not belong to the new main user" + grep -qx "new fstab" "$_B/new/etc/fstab" || _p="$_p;an existing file was REPLACED" + grep -qx "old fstab" "$_B/new/etc/fstab.new" 2>/dev/null || _p="$_p;a differing file got no .new" + [ ! -e "$_B/new/etc/shadow" ] && [ -f "$_B/new/etc/shadow.new" ] \ + || _p="$_p;a missing /etc/shadow was dropped in instead of written as .new" + printf '%s' "$_out" | grep -q 'TO MERGE BY HAND' || _p="$_p;the .new files are not listed at the end" + # SECTIONS (1.14.292): --skip leaves a named section out of a restore; + # clean removes this archive's untouched .new files and keeps an edited one + printf '[system:acct]\n/etc/shadow\n[system:other]\n/etc/fstab\n' > "$_B/conf2" + export LFS_BACKUP_CONF="$_B/conf2" + rm -f "$_B/new/etc/shadow.new" "$_B/new/etc/fstab.new" + LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/b.tgz" >/dev/null 2>&1 + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --skip acct --run >/dev/null 2>&1 + [ ! -e "$_B/new/etc/shadow.new" ] || _p="$_p;--skip did not leave the named section out" + [ -f "$_B/new/etc/fstab.new" ] || _p="$_p;--skip left out a section it was not given" + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --only acct --run >/dev/null 2>&1 + echo "merged" >> "$_B/new/etc/shadow.new" + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" clean "$_B/b.tgz" --run >/dev/null 2>&1 + [ ! -e "$_B/new/etc/fstab.new" ] || _p="$_p;clean left an untouched .new" + [ -f "$_B/new/etc/shadow.new" ] || _p="$_p;clean removed a .new that had been edited" + # PLAN (lfs-backup 1.2.0): a plan line set to skip is not restored, a + # changed target is used, and a [root] file is root's whatever it recorded + rm -f "$_B/new/etc/shadow.new" "$_B/new/etc/fstab.new" "$_B/plan" + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" plan "$_B/b.tgz" "$_B/plan" >/dev/null 2>&1 + sed -i -e 's/^restore | [a-z]* *| system:acct /skip | x | system:acct /' \ + -e 's#| /etc/fstab | /etc/fstab$#| /etc/fstab | /etc/fstab.moved#' "$_B/plan" + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/b.tgz" --plan "$_B/plan" --run >/dev/null 2>&1 + [ ! -e "$_B/new/etc/shadow.new" ] || _p="$_p;a plan line set to skip was restored" + grep -qx "old fstab" "$_B/new/etc/fstab.moved" 2>/dev/null || _p="$_p;a changed plan target was not used" + # SECRETS OWNER-ONLY (1.2.1): a world-readable recorded mode is not copied + [ "$(stat -c %a "$_B/new/etc/shadow.new" 2>/dev/null)" != "" ] || true + grep -q 'SECRET_RE=' "$_bk" || _p="$_p;a secret file is restored with whatever mode it had" + # A NAME IN BOTH [user] AND [root] (1.2.2): the plan key includes the kind + grep -q -- '-v n="$scope:$name"' "$_bk" || _p="$_p;a plan line for ~/.bashrc applies to both the user's and root's" + # MERGE (1.3.0): k removes the .new, n takes its content and keeps the + # live file's mode; the account databases cannot be taken whole + printf '[system:m]\n/etc/fstab\n' > "$_B/conf3"; export LFS_BACKUP_CONF="$_B/conf3" + echo "old fstab" > "$_B/old/etc/fstab"; echo "live fstab" > "$_B/new/etc/fstab"; chmod 640 "$_B/new/etc/fstab" + rm -f "$_B/new/etc/fstab.new" + LFS_BACKUP_ROOT="$_B/old" bash "$_bk" create --run "$_B/c.tgz" >/dev/null 2>&1 + LFS_BACKUP_ROOT="$_B/new" bash "$_bk" restore "$_B/c.tgz" --run >/dev/null 2>&1 + printf 'n\n' | LFS_BACKUP_ROOT="$_B/new" setsid bash "$_bk" merge "$_B/c.tgz" >/dev/null 2>&1 + grep -qx "old fstab" "$_B/new/etc/fstab" || _p="$_p;merge n did not take the backup's content" + [ "$(stat -c %a "$_B/new/etc/fstab")" = 640 ] || _p="$_p;merge n changed the live file's mode" + [ ! -e "$_B/new/etc/fstab.new" ] || _p="$_p;merge n left the .new behind" + grep -q 'account database: no whole take' "$_bk" || _p="$_p;merge would take an old passwd whole" + unset LFS_BACKUP_CONF +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "lfs-backup stores files by name and mode, and restores without replacing anything" +fi + +# ---- a zero exit is not proof, in FOUR dialects (1.14.180) ----------------- # +# llvm's configure died -- +# CMake Error at CMakeLists.txt:6 (set): +# -- Configuring incomplete, errors occurred! +# -- and the run printed "llvm: done (all phases complete)". The swallowed- +# failure check knew make's wording only; cmake fails before make runs. And +# its `ninja` alternative required `***`, which ninja never prints, so the +# one build system it was supposed to cover was not covered either. +_p="" +_hl8="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl8" ]; then + _pat="$(grep -m1 -o '"\^(make.*could not compile"' "$_hl8" | sed 's/^"//; s/"$//')" + [ -n "$_pat" ] || _p="$_p;the swallowed-failure pattern cannot be read" + # every dialect this system builds in must be caught, and nothing else + _chk() { # _chk <name> <line> <want: 1 catch / 0 ignore> + if printf '%s\n' "$2" | grep -m1 -E "$_pat" >/dev/null 2>&1; then + [ "$3" = 1 ] || _p="$_p;'$1' is treated as a failure and is not one" + else + [ "$3" = 0 ] || _p="$_p;$1 is not recognised as a failure" + fi + } + _chk "cmake configure" "CMake Error at CMakeLists.txt:6 (set):" 1 + _chk "cmake summary" "-- Configuring incomplete, errors occurred!" 1 + _chk "autotools" "configure: error: cannot find libfoo" 1 + _chk "meson" "meson.build:12:0: ERROR: Dependency not found" 1 + _chk "meson install" "ERROR: Unhandled python OSError." 1 + _chk "cargo" "error: could not compile \`gst-plugin-rtp\`" 1 + _chk "make" "make: *** [Makefile:408: all] Error 2" 1 + _chk "ninja" "ninja: build stopped: subcommand failed." 1 + _chk "configuring done" "-- Configuring done" 0 + _chk "cmake warning" "CMake Warning at CMakeLists.txt:6" 0 +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a failure is recognised in make, ninja, cmake, meson, autotools and cargo" +fi + +# ---- how many jobs, once (1.14.179) ---------------------------------------- # +# make reads MAKEFLAGS, cmake reads CMAKE_BUILD_PARALLEL_LEVEL, cargo reads +# CARGO_BUILD_JOBS, and ninja reads NOTHING (nproc+2). Setting MAKEFLAGS +# limited one of the four, so "use at most 13" was not sayable: a cmake or +# rust package still ran wide. +_p="" +_hl7="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl7" ]; then + for _v in CMAKE_BUILD_PARALLEL_LEVEL CARGO_BUILD_JOBS; do + grep -q "$_v" "$_hl7" || _p="$_p;--jobs does not limit $_v" + done + # READ THE VALUE THAT WILL BE PASSED, not the ambient one. build.env's + # MAKEFLAGS=-j13 is in $envpass by then; ${MAKEFLAGS:-} in a chroot root + # shell is empty, so CARGO_BUILD_JOBS was never set and rust built on + # every core (1.14.181). + grep -qF 'grep -oE "MAKEFLAGS=' "$_hl7" \ + || _p="$_p;the job count in build.env is not seen at all" + _jn="$( envpass="KF6_PREFIX=/usr MAKEFLAGS=-j13 NINJAJOBS=13" + printf '%s' "$envpass" | grep -oE "MAKEFLAGS='?-j ?[0-9]+" \ + | grep -oE '[0-9]+' | head -n1 )" + [ "$_jn" = 13 ] || _p="$_p;a build.env-style MAKEFLAGS parses to '$_jn'" + # derived from MAKEFLAGS when --jobs was not given, or build.env limits + # only make + grep -qF 'grep -oE '"'"'\-j ?[0-9]+'"'"'' "$_hl7" \ + || _p="$_p;a job count in MAKEFLAGS is not carried to cmake and cargo" +fi +if [ -f "$pm_src" ]; then + # NINJA IS NOT THE ONLY ONE THAT IGNORES THE ENVIRONMENT. rust's own + # page says the build is limited by "a COMBINATION of --jobs <N> on each + # invocation of ./x.py and CARGO_BUILD_JOBS"; with only the variable set + # it used every core. + _jp="$(grep -A22 '_jobs = opts.pop' "$pm_src")" + printf '%s' "$_jp" | grep -q 'x\.py' \ + || _p="$_p;./x.py is never given a job limit" + printf '%s' "$_jp" | grep -q 'cargo' \ + || _p="$_p;cargo build lines are never given a job limit" + # an explicit -j in the script or stack entry must win + printf '%s' "$_jp" | grep -q -- '\\s--jobs' \ + || _p="$_p;a line that already names --jobs would be given a second one" + _out="$(python3 - <<'PYJ' +import re +body = ("cmake .. && ninja\n./x.py build\n./x.py install\n" + "cargo cbuild --release\ncargo cinstall -j 4 --release\nninja -j2 check\n") +n = "13" +for pat, rep in ( + (r"(?m)(^|&&[ \t]*|;[ \t]*)(ninja\b(?![^\n]*(\s-j|\s--jobs)))", r"\1\2 -j" + n), + (r"(?m)^(\s*\.?/?x\.py\s+\w+(?![^\n]*(\s-j|\s--jobs)))", r"\1 --jobs " + n), + (r"(?m)^(\s*cargo\s+c?(?:build|install)\b(?![^\n]*(\s-j|\s--jobs)))", r"\1 -j " + n)): + body = re.sub(pat, rep, body) +print("OK" if ("x.py build --jobs 13" in body + and "x.py install --jobs 13" in body + and "cargo cbuild -j 13" in body + and "cargo cinstall -j 4 --release" in body + and "&& ninja -j13" in body + and "ninja -j2 check" in body) else "WRONG:\n" + body) +PYJ +)" + case "$_out" in + OK) ;; + *) _p="$_p;the job patch does not land where it must ($_out)" ;; + esac + # A MESSAGE THAT NAMES A THING MUST BE CHECKED AGAINST THE THING. + # With only [all] jobs = 13 set, the run said + # machine.conf [rust]: x.py --jobs 13 + # machine.conf has [rust] but the script has no meson/cmake line + # -- options NOT applied + # for a file with no [rust] section, about options that HAD been applied + # and with nothing left to apply (1.14.183). + _ms2="$(_slice_def "$pm_src" apply_machine_opts)" + _os2="$(_slice_def "$pm_src" _opt_sections)" + [ -n "$_os2" ] || _p="$_p;the section a value came from is never worked out" + _out2="$(PM_MACHINE_CONF=$T/mo.conf PM_PACKAGE_FIXES=$T/nof.conf python3 - "$pm_src" <<'PYMO' +import sys, types, os, io, contextlib +open(os.environ["PM_MACHINE_CONF"], "w").write( + "[all]\njobs = 13\n\n[webkitgtk]\njobs = 8\n\n[odd]\nweird = yes\n") +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +d = os.path.dirname(os.environ["PM_MACHINE_CONF"]) +for name, body in (("rust", "install_pkg() {\n./x.py build\n}\n"), + ("webkitgtk", "build_pkg() {\ncmake ..\nninja\n}\n"), + ("odd", "install_pkg() {\nmake install\n}\n")): + sp = os.path.join(d, "install_" + name) + open(sp, "w").write(body) + buf = io.StringIO() + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + mod.apply_machine_opts(name, sp) + print("%s|%s" % (name, (buf.getvalue().strip().splitlines() or [""])[0])) +PYMO +)" + case "$_out2" in + *"rust|machine.conf [all]: x.py --jobs 13"*) ;; + *) _p="$_p;a jobs value from [all] is credited to a section that does not exist" ;; + esac + case "$_out2" in + *"webkitgtk|machine.conf [webkitgtk]: ninja -j8"*) ;; + *) _p="$_p;the overridden [all] value is still credited" ;; + esac + # per LINE: a glob over the whole capture matched "rust|" here and + # "NOT applied" three lines later, and called that a bug + printf '%s\n' "$_out2" | grep '^rust|' | grep -q 'NOT applied' \ + && _p="$_p;options that were applied are reported as not applied" + case "$_out2" in + *"odd|machine.conf [odd] set weird"*) ;; + *) _p="$_p;an option that really cannot be applied is not reported" ;; + esac + # [all] in machine.conf, with a named section still winning + _mo="$(_slice_def "$pm_src" _machine_opts_for)" + printf '%s' "$_mo" | grep -q '"all"' \ + || _p="$_p;there is no way to set an option for every package" + _got="$(PM_MACHINE_CONF=$T/mc.conf PM_PACKAGE_FIXES=$T/none.conf python3 - "$pm_src" <<'PYALL' +import sys, types, os +open(os.environ["PM_MACHINE_CONF"], "w").write( + "[all]\njobs = 13\n\n[webkitgtk]\njobs = 8\n") +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +print(mod._machine_opts_for("mesa").get("jobs"), + mod._machine_opts_for("webkitgtk").get("jobs")) +PYALL +)" + [ "$_got" = "13 8" ] \ + || _p="$_p;[all] and a named section do not resolve to '13 8' but '$_got'" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "one job count reaches make, cmake, cargo and ninja" +fi + +# ---- a book command that works once (1.14.176) ----------------------------- # +# libtiff's page moves its html docs into a versioned directory. The first +# install creates /usr/share/doc/libtiff-4.7.1/tiff; the second cannot: +# mv: cannot overwrite '.../tiff': Directory not empty +# Nothing was wrong with the system -- the destination was the package's own +# output from last time, and a reinstall means replacing it -- but the repair +# gate did not recognise it, so a 38-package rebuild pass stopped on a +# directory the package itself had put there. +_p="" +_hl6="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl6" ]; then + _cs="$(_slice_fn "$_hl6" _clear_stale_dest)" + [ -n "$_cs" ] || _p="$_p;a destination that already exists is still unrecognised" + printf '%s' "$_cs" | grep -q 'cannot overwrite' \ + || _p="$_p;the message mv actually prints is not matched" + # ONLY THE PACKAGE'S OWN OUTPUT. Another package's directory in the way + # is a conflict, and deleting it would be the worst possible repair. + printf '%s' "$_cs" | grep -q 'stat -c %U' \ + || _p="$_p;it would delete a directory without checking who owns it" + printf '%s' "$_cs" | grep -q 'that is a conflict' \ + || _p="$_p;another package's directory is not distinguished from stale output" + # THE GATE DECIDES; THE LOOP ACTS. Called from the gate, the repair + # removed the directory -- correctly, and said so -- and then the retry + # loop ran ZERO rounds, because the loop only counts what it fixed + # itself. The obstacle was gone and the package failed anyway (1.14.178). + # (whole file, not the cmd_build slice: _slice_fn stops at the first line + # starting with }, and cmd_build is 1700 lines with several of those -- + # the slice ended long before these lines, so both assertions were + # testing an empty string) + # A RULE ON DISK IS NOT A RULE IN FORCE. light installs + # 90-backlight.rules, and until udevd is told, the brightness files keep + # their old group and light still cannot write them -- on a system where + # every file is exactly where it should be. A reboot hides it + # (1.14.194). + _ur="$(_slice_fn "$_hl6" _udev_reload_if_rules_changed)" + [ -n "$_ur" ] || _p="$_p;a new udev rule is never loaded" + grep -q '_udev_reload_if_rules_changed "$man"' "$_hl6" \ + || _p="$_p;the reload is defined and never called" + printf '%s' "$_ur" | grep -q 'control --reload' \ + || _p="$_p;udevd is not told the rules changed" + # ONLY the subsystems the rules mention: a bare trigger re-adds every + # device on the machine, which on a desktop can drop a session + printf '%s' "$_ur" | grep -q -- '--subsystem-match=' \ + || _p="$_p;it re-triggers every device instead of the ones concerned" + # THE ACTION MATTERS AS MUCH AS THE SUBSYSTEM. `udevadm trigger` sends + # **change** by default and light's rule is ACTION=="add", so the + # trigger fired, the rule did not match, and brightness stayed + # root:root -- while the output said it had re-triggered the subsystem + # (1.14.197). + printf '%s' "$_ur" | grep -q -- '--action=' \ + || _p="$_p;the trigger sends change, so an ACTION==add rule never runs" + printf '%s' "$_ur" | grep -q 'ACTION==' \ + || _p="$_p;the action is not read out of the rule" + _acts="$(printf '%s\n' \ + 'ACTION=="add", SUBSYSTEM=="backlight", RUN+="/bin/chgrp video x"' \ + | sed -n 's/.*ACTION=="\([a-z]*\)".*/\1/p' | sort -u | tr '\n' ' ')" + [ "$_acts" = "add " ] \ + || _p="$_p;the action parses to '$_acts', not the one in the rule" + # a bare `udevadm trigger` with no match option (the comment mentioning + # one does not count) + # a line that triggers WITHOUT a match option. (`trigger([^-]|$)` + # matched the space in `trigger --subsystem-match`, so the first version + # of this failed on correct code.) + printf '%s' "$_ur" | grep -v '^ *#' | grep 'udevadm trigger' \ + | grep -qv -- '--subsystem-match' \ + && _p="$_p;there is a bare udevadm trigger in there" + printf '%s' "$_ur" | grep -q 'this is a chroot' \ + || _p="$_p;in a chroot it tries to talk to a udevd that is not there" + # prove the subsystem is read out of the rule + mkdir -p "$T/udv/usr/lib/udev/rules.d" + printf 'ACTION=="add", SUBSYSTEM=="backlight", RUN+="/bin/true"\n' \ + > "$T/udv/usr/lib/udev/rules.d/90-backlight.rules" + _sub="$(sed -n 's/.*SUBSYSTEM=="\([a-z0-9_-]*\)".*/\1/p' \ + "$T/udv/usr/lib/udev/rules.d/90-backlight.rules")" + [ "$_sub" = backlight ] \ + || _p="$_p;the subsystem parses to '$_sub', not the one in the rule" + + # AN ABSOLUTE PATH IN THE MESSAGE IS THE ANSWER. + # + # install: cannot create regular file + # '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied + # + # The file does not exist (nothing to hand over) and the directory is + # another package's (a grant is what is needed), but the extractor only + # fired on a `cd` plus a RELATIVE path -- so nothing was extracted and + # light's udev rule stopped the install (1.14.193). + grep -q '_instdirs' "$_hl6" \ + || _p="$_p;an absolute path in a permission message is not used" + _got2="$(printf '%s\n' \ + "install: cannot create regular file '/usr/lib/udev/rules.d/90-backlight.rules': Permission denied" \ + "mkdir: cannot create directory '/usr/share/vim/vimfiles': Permission denied" \ + | grep -aoE "cannot create (regular file|directory) '/[^']+'" \ + | sed "s/.*'\(\/[^']*\)'.*/\1/" | sed 's|/[^/]*$||' | sort -u | tr '\n' ' ')" + case "$_got2" in + "/usr/lib/udev/rules.d /usr/share/vim "*) ;; + *) _p="$_p;the parent directory parses to '$_got2'" ;; + esac + grep -q '_clear_stale_dest "$log" "${owner:-}" && _fixed_something=1' "$_hl6" \ + || _p="$_p;clearing a stale destination does not trigger a retry" + grep -q '_clear_stale_dest .* && _perm_ish=1' "$_hl6" \ + && _p="$_p;the repair acts in the gate, so nothing re-runs the phase" + # -F: the thing being looked for IS a regex, so matching it as one + # matches something else entirely + grep -qF "cannot overwrite '[^']+'" "$_hl6" \ + || _p="$_p;the gate cannot recognise the failure it now repairs" + # CMAKE NAMES THE DESTINATION ACROSS THREE LINES, and /usr/bin is sticky + # (drwxrwxr-t root:install), so a group grant cannot help: only the + # file's owner may replace it, and /usr/bin/llvm-ar belonged to p_rust. + # The meson handover beside it needs "Unhandled python OSError" and + # "Installing SRC to DIR" lines, so it saw nothing -- zero rounds again + # (1.14.182). + _tc="$(_slice_fn "$_hl6" _take_cmake_dest)" + [ -n "$_tc" ] || _p="$_p;a cmake destination owned by another package is not handled" + printf '%s' "$_tc" | grep -qF 'file INSTALL cannot copy file' \ + || _p="$_p;cmake's own wording is not matched" + printf '%s' "$_tc" | grep -q "joined into one line" \ + || _p="$_p;a message spanning three lines cannot be read line by line" + printf '%s' "$_tc" | grep -q 'root|""' \ + || _p="$_p;it would take a file owned by root" + grep -q '_take_cmake_dest "$log" "${owner:-}" && _fixed_something=1' "$_hl6" \ + || _p="$_p;the handover does not trigger a retry" + # prove it on the real three-line text + mkdir -p "$T/cm/bin" + printf 'x\n' > "$T/cm/bin/llvm-ar" + printf 'CMake Error at tools/llvm-ar/cmake_install.cmake:52 (file):\n file INSTALL cannot copy file\n "/src/build/bin/llvm-ar" to\n "%s": Permission denied.\n' \ + "$T/cm/bin/llvm-ar" > "$T/cm/log" + _dest="$(tr '\n' ' ' < "$T/cm/log" \ + | grep -aoE 'file INSTALL cannot copy file[[:space:]]+"[^"]+"[[:space:]]+to[[:space:]]+"[^"]+"' \ + | sed 's/.*[[:space:]]to[[:space:]]*"//; s/"$//')" + [ "$_dest" = "$T/cm/bin/llvm-ar" ] \ + || _p="$_p;the destination parses to '$_dest', not the file cmake named" + + # and a second round must find nothing, or the loop spins + printf '%s' "$_cs" | grep -q '\[ -e "$path" \] || continue' \ + || _p="$_p;a path already removed is reported as fixed again" + # A REPAIR MUST NOT BE ABLE TO FAIL THE THING IT REPAIRS. The gate + # called it with "$user"; cmd_build calls the package user `owner`, so + # `set -u` aborted a git build whose install had just finished -- with a + # line number and nothing else (1.14.177). + _cb="$(_slice_fn "$_hl6" cmd_build)" + printf '%s' "$_cb" | grep -qE '"\$user"' \ + && _p="$_p;cmd_build reads \$user, which it never sets" + printf '%s' "$_cb" | grep -q '_clear_stale_dest "$log" "${owner:-}"' \ + || _p="$_p;the repair is called with a variable that may be unset" + printf '%s' "$_cs" | grep -q 'user="${2:-}"' \ + || _p="$_p;the repair itself dies when called with nothing" + # every variable the gate passes must be one the function assigns first + printf '%s' "$_cb" | grep -q 'owner=' \ + || _p="$_p;owner is read in cmd_build and never assigned there" + # prove all three outcomes on real text + mkdir -p "$T/stale/mine/tiff" "$T/stale/theirs/x" + echo x > "$T/stale/mine/tiff/i.html" + printf "mv: cannot overwrite '%s': Directory not empty\n" "$T/stale/mine/tiff" > "$T/stale/log1" + printf "make: *** Error 1\n" > "$T/stale/log2" + _out="$( warn() { echo "!! $*"; }; note() { echo "$*"; } + eval "$_cs" + _clear_stale_dest "$T/stale/log1" "$(id -un)"; echo "rc1=$?" + _clear_stale_dest "$T/stale/log2" "$(id -un)"; echo "rc2=$?" )" + case "$_out" in + *rc1=0*) ;; + *) _p="$_p;a stale destination the package owns is not cleared" ;; + esac + case "$_out" in + *rc2=0*) _p="$_p;an unrelated failure is treated as a stale destination" ;; + esac + [ -e "$T/stale/mine/tiff" ] && _p="$_p;it reported clearing the directory and did not" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a destination the package itself left behind is cleared, anyone else's is not" +fi + +# ---- the help is a list, and lists go stale (1.14.175) --------------------- # +# The grouped `packagemanager --help` -- the one a reader actually uses -- +# had no `stack`, `probe`, `late-deps`, `reclaim`, `bootstrap` or `env`, and +# stack is the primary command. The module docstring was worse: a "Roadmap" +# marking check, search, dependencies, install, remove and reload-pkg-list as +# [TODO] years after they were implemented. +# +# argparse knows the real list, and says it when given a bad command. Ask it. +_p="" +if [ -f "$pm_src" ]; then + _choices="$(python3 "$pm_src" __no_such_command__ 2>&1 \ + | sed -n "s/.*choose from //p" | tr -d "')(" | tr ',' '\n' \ + | sed "s/^ *//;s/'//g" | grep -v '^$')" + [ -n "$_choices" ] || _p="$_p;argparse does not list its own commands" + _grouped="$(python3 "$pm_src" --help 2>&1 \ + | sed -n '/commands, by what you are doing/,$p')" + # aliases do not need their own line + for _c in $_choices; do + case "$_c" in + deps|setup|which-pkg|regen-script|add-dir-to-nimgnu|nimgnu) continue ;; + esac + printf '%s' "$_grouped" | grep -q -- "$_c" \ + || _p="$_p;$_c is a command and is not in the grouped help" + done + # and no second list anywhere in the file + # the SHAPE of a roadmap entry, not the word: the comment explaining this + # bug quotes [TODO], and matching that matched the explanation + grep -qE '^ +[a-z_-]+ +\[(TODO|IMPLEMENTED)\]' "$pm_src" \ + && _p="$_p;the module docstring still keeps a roadmap of commands" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "every command appears in the grouped help, and there is one list" +fi + +# ---- a package owns what it installed, not where (1.14.158) --------------- # +# /usr/include/gstreamer-1.0/gst/mse belonged to p_linux-headers, so +# gst-plugins-bad could not install into its own directory. The repair handed +# it one file per round; after three rounds the bundled-copy heuristic +# concluded meson was vendoring linux-headers. Every step reasoned correctly +# from a wrong fact, and the wrong fact was one line: +# chown -R "$_hacct:$_hacct" /usr/include +# which gave the headers package every header on the system. +_p="" +_k5="$(dirname "$LFS_TOOL")/lfs-kernel" +if [ -f "$_k5" ]; then + # PROSE IS NOT CODE: the comment that explains this bug quotes the old + # line, and the first version of this assertion matched the comment. + grep -vE '^[[:space:]]*#' "$_k5" \ + | grep -qE 'chown -R [^ ]*_hacct[^ ]* /usr/include' \ + && _p="$_p;the headers package still takes ownership of all of /usr/include" + # it must chown the paths it copied, which means knowing what they were + grep -q "find . -mindepth 1 -printf" "$_k5" \ + || _p="$_p;nothing records which headers were installed" + grep -q 'xargs -0 -r chown -h' "$_k5" \ + || _p="$_p;the chown is not limited to the recorded paths" +fi +# AND THE REPAIR FOR A MACHINE THAT ALREADY HAS IT. One tree-wide chown is +# not a per-package problem: /usr/include/glib-2.0 belonged to +# p_linux-headers, so glib2 could not install its own headers either +# (1.14.166). +if [ -f "$pm_src" ]; then + python3 "$pm_src" reclaim --help 2>/dev/null | grep -q -- '--all' \ + || _p="$_p;the ownership repair has to be run one package at a time" + _rcl="$(_slice_def "$pm_src" cmd_reclaim)" + printf '%s' "$_rcl" | grep -q '_pkg_dirs()' \ + || _p="$_p;--all does not mean every installed package" + printf '%s' "$_rcl" | grep -q 'getattr(args, "all", False)' \ + || _p="$_p;a package with no manifest is an error under --all" +fi + +# and the audit that finds this class after the fact +if [ -f "$pm_src" ]; then + python3 "$pm_src" late-deps --help >/dev/null 2>&1 \ + || _p="$_p;there is no way to ask which packages were built too early" + _ld="$(_slice_def "$pm_src" cmd_late_deps)" + [ -n "$_ld" ] || _p="$_p;cmd_late_deps could not be read out of the tool" + printf '%s' "$_ld" | grep -q '3600' \ + || _p="$_p;two packages built in one run would be reported as a problem" + # SLACK THAT HIDES THE CASE IS TOO MUCH SLACK. A day of it filed + # `ffmpeg ... alsa-lib +0 day(s)` as fine, and alsa-lib a few HOURS after + # ffmpeg is the original bug (1.14.168). + printf '%s' "$_ld" | grep -q '> 86400' \ + && _p="$_p;a gap of hours is still treated as no gap" + printf '%s' "$_ld" | grep -q 'hour(s)" % (gap // 3600)' \ + || _p="$_p;a sub-day gap is reported as zero days" + printf '%s' "$_ld" | grep -q 'only a probe can settle it' \ + || _p="$_p;a same-session gap is presented as if timestamps settled it" + # ...and the probe exists, reads the stacks, and names the rebuild + _pr="$(_slice_def "$pm_src" cmd_probe)" + [ -n "$_pr" ] || _p="$_p;there is no way to run the expect= probes after the fact" + printf '%s' "$_pr" | grep -q 'expect' \ + || _p="$_p;the probe runner does not read the stacks' probes" + printf '%s' "$_pr" | grep -q -- '--only' \ + || _p="$_p;a failed probe does not say how to rebuild" + python3 "$pm_src" probe --help >/dev/null 2>&1 \ + || _p="$_p;probe is not a command" + # AN EDITED STACK IS NEVER OVERWRITTEN, so the tool can carry probes that + # no installed stack has -- and "no expect= probes in the stacks" then + # reads as "nothing to check" (1.14.169). + printf '%s' "$_pr" | grep -q 'stack.new' \ + || _p="$_p;no probes does not say the newer stacks are waiting" + printf '%s' "$_pr" | grep -q -- '--take-new' \ + || _p="$_p;it does not say how to take the newer stack" + grep -q -- '--take-new' "$pm_src" \ + || _p="$_p;the suggested --take-new is not a flag that exists" + # A STACK ENTRY DOES NOT REBUILD AN INSTALLED PACKAGE. The expect= + # failure suggested `stack sway.stack --only sdl3 --run` -- with the + # .stack suffix, which is not a stack name, and running `install`, which + # answers "everything is installed and up to date". Re-running it could + # never fix the feature (1.14.171). + _ex2="$(grep -n 'feature check failed' -A25 "$pm_src")" + printf '%s' "$_ex2" | grep -q '_rebuild_cmd' \ + || _p="$_p;a failed feature check still points at a command that no-ops" + # AND THE COMMAND DEPENDS ON THE KIND OF ENTRY. `update --reinstall + # --regenerate` regenerates from the BOOK, and gst-plugins-rs is a git + # entry with no book page -- that advice could not work at all + # (1.14.202). + printf '%s' "$_ex2" | grep -q 'e\["kind"\] == "book"' \ + || _p="$_p;a git entry is told to regenerate from a book page it does not have" + printf '%s' "$_ex2" | grep -q -- '--run --force' \ + || _p="$_p;nothing says --force is what rebuilds a git entry" + # ...and have= proving less than it looks is worth saying out loud: a + # part-installed package satisfies it + printf '%s' "$_ex2" | grep -q 'one file is not the whole package' \ + || _p="$_p;a part-installed package that satisfies have= is not explained" + printf '%s' "$_ex2" | grep -q 'will NOT rebuild it' \ + || _p="$_p;it does not say that re-running the entry cannot help" + printf '%s' "$_ex2" | grep -q 'stack %s --only %s --run' \ + && _p="$_p;the suggested stack name still carries the .stack suffix" + # the drift notice must show WHAT differs: two equal entry counts argued + # there was nothing to take, for files differing in every entry + grep -q 'the entry count is the same' "$pm_src" \ + || _p="$_p;the newer-stack notice offers a count as its only evidence" + grep -q 'line(s) in the file; the plan below' "$pm_src" \ + || _p="$_p;take-new and the header report two different entry counts" + + # A PROBE THAT FAILS BECAUSE THE PACKAGE IS NOT THERE IS A DIFFERENT + # FINDING. gst10-libav had never been built -- it was missing from the + # stack -- and the report said "the feature is NOT in the installed + # package", sending the reader to look for a build option (1.14.170). + printf '%s' "$_pr" | grep -q 'NOT INSTALLED' \ + || _p="$_p;a probe cannot tell an absent package from a missing feature" + printf '%s' "$_pr" | grep -q 'pkg.lst' \ + || _p="$_p;installed-ness is assumed rather than checked" + # ...and a probe can be wrong about a path as easily as a package can be + # wrong about a feature, so show what it said + printf '%s' "$_pr" | grep -q 'the check said' \ + || _p="$_p;a failing probe's own output is thrown away" + # the sdl3 probe must not name one library file: SDL3 installs + # libSDL3.so.0.<n> and the bare symlink is not always there + # $_sw belongs to the next block; name the file, do not borrow a variable + grep -q 'libSDL3.so\*' "$_DESKTOP_STACK" \ + || _p="$_p;the SDL3 probe names a path that may not exist" + # AN UNMATCHED DEPENDENCY IS NOT A CLEAN RESULT. A silent miss would + # under-report exactly the thing being looked for. + printf '%s' "$_ld" | grep -q 'match no INSTALLED package' \ + || _p="$_p;dependencies it could not resolve are not reported" + # ...and say that an unmatched name is usually just a package that is not + # installed, which is nothing to fix: 103 of them read as 103 problems + printf '%s' "$_ld" | grep -q 'cannot have been built after anything' \ + || _p="$_p;unmatched names read as unfinished work" + printf '%s' "$(_slice_def "$pm_src" _built_when)" | grep -q 'pkg.lst' \ + || _p="$_p;the install time does not come from the manifest" + # THREE DIFFERENT ANSWERS. On a real system the report was 40 packages + # in one list: some fixable by order, some CIRCULAR (cairo requires + # glib2, mesa requires libxcb -- no order exists, only a second pass), + # and a third of them documentation tools (graphviz, docutils). One + # list means the real findings are read as noise (1.14.159). + printf '%s' "$_ld" | grep -q 'no order exists' \ + || _p="$_p;a circular dependency is reported as an ordering mistake" + printf '%s' "$_ld" | grep -q 'in the stack: put' \ + || _p="$_p;an orderable case does not say what to change" + printf '%s' "$(_slice_def "$pm_src" _depends_on)" | grep -q 'seen' \ + || _p="$_p;the dependency walk can loop for ever" + grep -q '_DOC_DEPS' "$pm_src" \ + || _p="$_p;documentation tools are mixed in with real features" + # the rebuild order is not the report order + printf '%s' "$_ld" | grep -q 'rebuild, in this order' \ + || _p="$_p;there is no rebuild plan" + printf '%s' "$_ld" | grep -q 'g_order.get(n, set()) & nodes' \ + || _p="$_p;the rebuild plan is not ordered by dependency" + printf '%s' "$_ld" | grep -q 'add --run to do it' \ + || _p="$_p;--rebuild writes without being asked" + # INSTALL DOES NOT REBUILD. The first plan emitted `install --run + # --regenerate`, which answered "Nothing to do -- everything is installed + # and up to date" 28 times and looked like a successful pass. + _rc="$(_slice_def "$pm_src" _rebuild_cmd)" + printf '%s' "$_rc" | grep -q '"update"' \ + || _p="$_p;the rebuild plan uses a verb that does not rebuild" + printf '%s' "$_rc" | grep -q -- '--reinstall' \ + || _p="$_p;the rebuild plan does not force a reinstall" + printf '%s' "$_ld" | grep -q 'install %s --run --regenerate' \ + && _p="$_p;a rebuild line still says install" + # RECOMMENDED IS A DEPENDENCY TOO when asking whether an order exists: + # BLFS lists mesa->libxcb and cairo->glib2 as recommended, and with + # `required` alone the report advised putting mesa before libxcb. + printf '%s' "$_ld" | grep -q 'g_order' \ + || _p="$_p;only required edges are used to detect a cycle" + printf '%s' "$_ld" | grep -q '"required", "recommended"' \ + || _p="$_p;a recommended cycle is reported as an ordering mistake" + # A CYCLE BROKEN ALPHABETICALLY IS NOT AN ORDER. base needs mesa, mesa + # needs libxcb, bad needs base -- one cluster with no entry point, and + # `sorted(left)[0]` then emitted gst10-plugins-bad before + # gst10-plugins-base, cairo and fontconfig before freetype2. That is the + # exact mistake the plan exists to prevent (1.14.163). + printf '%s' "$_ld" | grep -v '^[[:space:]]*#' | grep -q 'ready = \[sorted(left)' \ + && _p="$_p;a dependency cycle is still broken alphabetically" + printf '%s' "$_ld" | grep -q 'sum(1 for m in cs' \ + || _p="$_p;the cycle pick ignores how many packages wait for it" + # ORDER THE CLUSTERS, NOT THE NODES. Picking one node out of a cycle and + # returning to the ready scan stranded the rest at the end: freetype2 and + # harfbuzz came out LAST, after fontconfig and cairo, which require + # freetype2. Tarjan's components form a DAG, so they sort (1.14.164). + printf '%s' "$_ld" | grep -q '_strong' \ + || _p="$_p;a dependency cycle is still broken one node at a time" + printf '%s' "$_ld" | grep -q 'iterative' \ + || _p="$_p;the component walk recurses, and this graph has 400 nodes" + # prove it on the shape that failed + _scc="$(PKGUSR_BASE=$T/scc PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYSCC' +import sys, types, os, time, io, contextlib +base = os.environ["PKGUSR_BASE"] + "/pkgusr" +def mk(name, nv, req, rec, opt, old=True): + d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) + f = os.path.join(d, "install_last") + open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" + "recommended=(%s)\noptional=(%s)\n" + % (name, nv, req, rec, opt)) + w = time.time() - (400 * 86400 if old else 86400) + os.utime(f, (w, w)) + if not old: + p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n"); os.utime(p, (w, w)) +mk("freetype2", "freetype-2.14.1", "", "'harfbuzz-12.1.0'", "'brotli-1.1.0'") +mk("harfbuzz", "harfbuzz-12.1.0", "'freetype-2.14.1'", "", "'brotli-1.1.0'") +mk("fontconfig", "fontconfig-2.17.1", "'freetype-2.14.1'", "", "'brotli-1.1.0'") +mk("cairo", "cairo-1.18.6", "'freetype-2.14.1' 'fontconfig-2.17.1'", "", "'brotli-1.1.0'") +mk("brotli", "brotli-1.1.0", "", "", "", old=False) +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +class A: + kind = "recommended,optional"; package = []; verbose = False; why = False + include_docs = False; rebuild = True; run = False +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + mod.cmd_late_deps(A()) +order = [ln.split()[2] for ln in buf.getvalue().splitlines() + if ln.strip().startswith("packagemanager update")] +ok = (order.index("freetype2") < order.index("fontconfig") + and order.index("harfbuzz") < order.index("cairo")) +print("CLUSTER_FIRST" if ok else "WRONG %s" % order) +PYSCC +)" + case "$_scc" in + *CLUSTER_FIRST*) ;; + *) _p="$_p;a cycle is still rebuilt after the packages that need it ($_scc)" ;; + esac + # AND THE OPTIONAL EDGE DECIDES THE ORDER TOO. The graph was + # required+recommended, so mpg123 was rebuilt before pulseaudio -- and + # mpg123's missing feature IS pulseaudio (1.14.165). + printf '%s' "$_ld" | grep -q '"required", "recommended", "optional"' \ + || _p="$_p;the rebuild order ignores the optional edges it exists for" + _opt="$(PKGUSR_BASE=$T/optord PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYOPT' +import sys, types, os, time, io, contextlib +base = os.environ["PKGUSR_BASE"] + "/pkgusr" +def mk(name, nv, req, rec, opt, old=True): + d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) + f = os.path.join(d, "install_last") + open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" + "recommended=(%s)\noptional=(%s)\n" + % (name, nv, req, rec, opt)) + w = time.time() - (400 * 86400 if old else 86400) + os.utime(f, (w, w)) + if not old: + p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n"); os.utime(p, (w, w)) +mk("mpg123", "mpg123-1.33.3", "", "'alsa-lib-1.2.14'", "'PulseAudio-17.0'") +mk("pulseaudio", "pulseaudio-17.0", "", "'alsa-lib-1.2.14'", "") +mk("alsa-lib", "alsa-lib-1.2.14", "", "", "", old=False) +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +class A: + kind = "recommended,optional"; package = []; verbose = False; why = False + include_docs = False; rebuild = True; run = False +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + mod.cmd_late_deps(A()) +o = [ln.split()[2] for ln in buf.getvalue().splitlines() + if ln.strip().startswith("packagemanager update")] +print("PULSE_FIRST" if o.index("pulseaudio") < o.index("mpg123") else "WRONG %s" % o) +PYOPT +)" + case "$_opt" in + *PULSE_FIRST*) ;; + *) _p="$_p;an optional dependency is still rebuilt after its user ($_opt)" ;; + esac + + # and a suspicious position must be explainable + python3 "$pm_src" late-deps --why --help >/dev/null 2>&1 \ + || _p="$_p;there is no way to see which edges were used" + printf '%s' "$_ld" | grep -q 'in dependency cycles' \ + || _p="$_p;a cycle is broken silently, so a second pass is a surprise" + # ...and prove it on the shape that failed: bad needs base, base needs + # mesa, mesa needs base (a cycle) -- base must come out before bad + _cyc="$(PKGUSR_BASE=$T/cyc PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYCYC' +import sys, types, os, time +base = os.environ["PKGUSR_BASE"] + "/pkgusr" +def mk(name, nv, req, rec, opt, old=True): + d = os.path.join(base, "p_" + name); os.makedirs(d, exist_ok=True) + f = os.path.join(d, "install_last") + open(f, "w").write("name='%s'\nname_version='%s'\nrequired=(%s)\n" + "recommended=(%s)\noptional=(%s)\n" + % (name, nv, req, rec, opt)) + when = time.time() - (400 * 86400 if old else 1 * 86400) + os.utime(f, (when, when)) + if not old: + p = os.path.join(d, "pkg.lst"); open(p, "w").write("x\n") + os.utime(p, (when, when)) +mk("gst10-plugins-base", "gst-plugins-base-1.28.1", + "'mesa-25.2.3'", "", "'alsa-lib-1.2.14'") +mk("gst10-plugins-bad", "gst-plugins-bad-1.28.1", + "'gst-plugins-base-1.28.1'", "", "'libnice-0.1.22'") +mk("mesa", "mesa-25.2.3", "'gst-plugins-base-1.28.1'", "", "'libnice-0.1.22'") +mk("alsa-lib", "alsa-lib-1.2.14", "", "", "", old=False) +mk("libnice", "libnice-0.1.22", "", "", "", old=False) +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +class A: + kind = "recommended,optional"; package = []; verbose = False + include_docs = False; rebuild = True; run = False +import io, contextlib +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + mod.cmd_late_deps(A()) +out = buf.getvalue() +order = [ln.split()[2] for ln in out.splitlines() + if ln.strip().startswith("packagemanager update")] +print("BASE_BEFORE_BAD" if order.index("gst10-plugins-base") + < order.index("gst10-plugins-bad") else "WRONG %s" % order) +PYCYC +)" + case "$_cyc" in + *BASE_BEFORE_BAD*) ;; + *) _p="$_p;in a cycle the package others wait for is still rebuilt last ($_cyc)" ;; + esac + + # AN EDGE THAT DOES NOT RESOLVE IS AN EDGE THAT DOES NOT EXIST. + # 119 dependency names did not match a package, and the rebuild order was + # computed from what was left: gst10-plugins-bad sorted BEFORE + # gst10-plugins-base (which it requires), fontconfig before freetype2. + # The mapping is in the data -- each script records name_version, the + # page title the deps are written with (1.14.162). + _di="$(_slice_def "$pm_src" _dep_index)" + [ -n "$_di" ] || _p="$_p;dependency titles are matched by guesswork" + printf '%s' "$_di" | grep -q "name_version='" \ + || _p="$_p;the title-to-package mapping ignores what each script records" + # resolve the two cases that broke the order, for real + _res="$(PKGUSR_BASE=$T/depidx PKGUSR_CONFIG=/dev/null python3 - "$pm_src" <<'PYRES' +import sys, types, os +base = os.environ["PKGUSR_BASE"] + "/pkgusr" +for name, nv in (("gst10-plugins-base", "gst-plugins-base-1.28.1"), + ("gst10-plugins-bad", "gst-plugins-bad-1.28.1"), + ("freetype2", "freetype-2.14.1"), + ("fontconfig", "fontconfig-2.17.1")): + d = os.path.join(base, "p_" + name) + os.makedirs(d, exist_ok=True) + open(os.path.join(d, "install_last"), "w").write( + "name='%s'\nname_version='%s'\n" % (name, nv)) +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +inst = mod._pkg_dirs() +print(mod._resolve_dep("gst-plugins-base-1.28.1", inst), + mod._resolve_dep("FreeType-2.14.1", inst)) +PYRES +)" + case "$_res" in + *"gst10-plugins-base freetype2"*) ;; + *) _p="$_p;a dependency written as its page title still does not resolve ($_res)" ;; + esac + # --reinstall IS NOT A QUESTION ABOUT VERSIONS. With no recorded version + # `update curl --reinstall` said "nothing to update, 1 it cannot judge" + # and did nothing -- 28 times. + python3 - "$pm_src" <<'PYRI' || _p="$_p;--reinstall does nothing for a package with no recorded version" +import sys +src = open(sys.argv[1]).read() +i = src.index("no_record.append(name)") +blk = src[max(0, i - 1200):i] +sys.exit(0 if 'reinstall' in blk and '"(reinstall)", bv' in blk else 1) +PYRI + + # NO HOST, NO STALENESS. On a booted system the stamp records what was + # copied in during the build, so every later `make install` made all + # seven tools "differ" and packagemanager printed, before every command, + # an instruction to run sync-tools on a host that no longer exists + # (1.14.161). + _st="$(_slice_def "$pm_src" warn_if_tools_are_stale)" + printf '%s' "$_st" | grep -q '_in_chroot' \ + || _p="$_p;the host-staleness warning fires on a booted system" + printf '%s' "$(_slice_def "$pm_src" _in_chroot)" | grep -q '/proc/1/root' \ + || _p="$_p;the chroot test does not ask pid 1 where its root is" + # ...and whoever installs the tools records what they are + grep -q 'tool-stamps.json' "$(dirname "$LFS_TOOL")/Makefile" \ + || _p="$_p;make install leaves the stamp describing older copies" + # every flag the message suggests must exist + printf '%s' "$_st" | grep -q 'sync-tools --run' \ + && { python3 "$(dirname "$LFS_TOOL")/lfs" build-system sync-tools --help \ + >/dev/null 2>&1 || _p="$_p;the suggested sync-tools command does not exist"; } + + # and a stack run must raise the question itself + grep -q 'packagemanager late-deps --rebuild --run' "$pm_src" \ + || _p="$_p;a finished stack never mentions the second pass" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the headers package owns only its headers, and late builds are findable" +fi + +# ---- the desktop's non-book pieces (1.14.184) ------------------------------ # +# light (backlight), wl-clipboard, and the icon/patched/Noto fonts are in no +# book, so the stack carries them -- and everything about them is ours to +# keep true: refs, URLs, and the archive layouts. +_p="" +_sw2="$_DESKTOP_STACK" +_fs="$(dirname "$LFS_TOOL")/stacks/fonts.sh" +if [ -f "$_sw2" ]; then + # jq is not in the book either, and its probe has to test the half of jq + # that can be missing: `jq --version` passes on a jq built without + # oniguruma, and `test`/`match`/`capture` are what most scripts use + # (1.14.198). + grep -qE '^tar +jq ' "$_sw2" || _p="$_p;jq is in no stack" + grep -A6 -E '^tar +jq ' "$_sw2" | grep -q 'test(' \ + || _p="$_p;the jq probe would pass on a jq with no regex" + grep -B10 -E '^tar +jq ' "$_sw2" | grep -q 'packaged oniguruma' \ + || _p="$_p;nothing records whether jq needs a system oniguruma" + # the probe string must survive the parser and still be runnable + _jqp="$(python3 - "$pm_src" "$_sw2" <<'PYJQ' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +for r in mod._parse_stack_file(sys.argv[2]): + if r["target"] == "jq": + print(r["opts"].get("expect", "")) + break +PYJQ +)" + case "$_jqp" in + *'test('*) ;; + *) _p="$_p;the jq probe does not survive the stack parser: '$_jqp'" ;; + esac + # it must be valid shell, whatever jq itself says + ( eval "jq() { :; }; $_jqp" ) >/dev/null 2>&1 \ + || _p="$_p;the jq probe is not runnable shell" + # wf-recorder's AUDIO IS CONDITIONAL and its --help says `-a, --audio` + # either way -- that text is outside the #ifdef. So the probe has to + # check what the binary LINKS, not what it prints, and the entry has to + # come after pipewire so the library is there at configure time + # (1.14.208). + grep -qE '^git +wf-recorder' "$_sw2" || _p="$_p;wf-recorder is in no stack" + grep -A6 -E '^git +wf-recorder' "$_sw2" | grep -q 'ldd /usr/bin/wf-recorder' \ + || _p="$_p;the wf-recorder probe does not check what it linked" + grep -A6 -E '^git +wf-recorder' "$_sw2" | grep -q -- '--help' \ + && _p="$_p;the probe reads help text, which says --audio either way" + _wo="$(python3 - "$pm_src" "$_sw2" <<'PYWF' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +bad = [n for n in ("ffmpeg", "pipewire") + if n in o and o.index(n) > o.index("wf-recorder")] +print("LATE " + " ".join(bad) if bad else "OK") +PYWF +)" + [ "$_wo" = OK ] \ + || _p="$_p;wf-recorder configures before what it needs: $_wo" + # ZATHURA IS THREE PACKAGES and the core opens no formats at all: girara + # is its widget library, the backend is a separate plugin, and a zathura + # with no plugin starts, shows a grey window and says nothing. So the + # probe looks for the PLUGIN. All three share one release tag -- mixing + # tags is the usual way to get a zathura that cannot open a PDF + # (1.14.241). + for _e in girara zathura zathura-pdf-poppler imv; do + grep -qE "^git +$_e " "$_sw2" || _p="$_p;$_e is in no stack" + done + # NO INVENTED OPTIONS, NO INVENTED FILENAMES. `-Dsqlite=enabled` + # stopped meson with `Unknown option: "sqlite"` (zathura takes sqlite3 + # unconditionally and has no such option), and have=girara-gtk3.pc made + # the stack refuse to record a girara that had installed perfectly -- + # its meson.build says `pkg.generate(name: 'girara')`, so the file is + # girara.pc (1.14.244). + # zathura 2026.07.18 ALSO needs libxxhash, which the book mentions eight + # times and gives no page (1.14.245). Its checkout ships + # libxxhash.pc.in, so `libxxhash` really is the module meson asks for. + grep -qE '^git +xxhash ' "$_sw2" \ + || _p="$_p;zathura will stop at configure for libxxhash" + _xo="$(python3 - "$pm_src" "$_sw2" <<'PYXX' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +bad = [n for n in ("xxhash", "girara") if n in o and o.index(n) > o.index("zathura")] +print("BAD: " + ", ".join(bad) if bad else "OK") +PYXX +)" + [ "$_xo" = OK ] || _p="$_p;built after zathura, which needs it: $_xo" + grep -A6 -E '^git +xxhash ' "$_sw2" | grep -q 'libxxhash.pc' \ + || _p="$_p;nothing proves xxhash installed the module meson wants" + grep -A6 -E '^git +zathura ' "$_sw2" | grep -q 'Dsqlite' \ + && _p="$_p;zathura is built with an option it does not have" + grep -A6 -E '^git +girara ' "$_sw2" | grep -q 'girara-gtk3.pc' \ + && _p="$_p;girara's have= names a file it does not install" + grep -A6 -E '^git +girara ' "$_sw2" | grep -q 'pkgconfig/girara.pc' \ + || _p="$_p;girara has no have= that its build actually produces" + _zt="$(grep -oE 'ref=2026\.[0-9.]+' "$_sw2" | sort -u | wc -l)" + [ "$_zt" = 1 ] \ + || _p="$_p;the zathura family does not share one release tag ($_zt seen)" + grep -A6 -E '^git +zathura-pdf-poppler' "$_sw2" | grep -q 'libpdf-poppler.so' \ + || _p="$_p;nothing proves zathura got a PDF backend" + grep -qE '^book +poppler' "$_sw2" \ + || _p="$_p;the pdf backend has no poppler to build against" + grep -qE '^avoid +poppler' "$_av" \ + && _p="$_p;poppler is both built and avoided" + grep -qE '^avoid +mupdf' "$_av" \ + || _p="$_p;the second pdf backend is no longer excluded" + # imv links libicuio and CALLS libicuuc, with -Wl,--no-undefined on the + # link line, so the indirect dependency is an ERROR: "undefined + # reference to symbol 'ubrk_close_78' ... DSO missing from command + # line" (1.14.251). + grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'c_link_args=-licuuc' \ + || _p="$_p;imv will not link: libicuuc is not on its link line" + grep -B10 -E '^git +imv ' "$_sw2" | grep -q 'DSO missing' \ + || _p="$_p;nothing records why that flag is there" + # ...AND NOR IS ITS SHARED DIRECTORY. `appuser_home()` returned + # BASE_DIR, so `cfg app-users` ran `mkdir -p /usr/src/u_claude/Shared` + # and pointed ~/Shared/Claude into the package tree -- even after the + # homes had been moved to /home with usermod (1.14.256). + _ah="$(_slice_def "$pm_src" appuser_home)" + # (1.14.282: the default is ONE subdirectory, APPUSER_HOME_DEFAULT) + printf '%s' "$_ah" | grep -q 'or APPUSER_HOME_DEFAULT' \ + || _p="$_p;application users default to the package base, not their subdirectory" + grep -q '^APPUSER_HOME_DEFAULT = "/home/shared_user"$' "$pm_src" \ + || _p="$_p;application users do not default to /home/shared_user" + sed -n '/^def _relocate_app_home/,/^def /p' "$pm_src" | grep -q 'want = os.path.join(root, u)' \ + || _p="$_p;an application home is still moved to a hard-coded /home/<user>" + _aph="$(_slice_def "$pm_src" _app_home)" + [ -n "$_aph" ] || _p="$_p;nothing resolves an application user's real home" + printf '%s' "$_aph" | grep -q '_home_of(u) or' \ + || _p="$_p;a home moved with usermod is ignored" + # (1.14.315: the directory lives in the hub, not in any home -- the + # account's ~/Shared is a link to it; see the block for 1.14.315) + grep -q 'share = os.path.join(_hub, args.name' "$pm_src" \ + || _p="$_p;the shared directory is not built in the hub" + _apo="$(PKGUSR_CONFIG=/dev/null python3 - "$pm_src" 2>&1 <<'PYAH' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +mod._home_of = lambda u: "/home/u_moved" if u == "u_moved" else None +bad = [] +if mod.appuser_home() != "/home/shared_user": + bad.append("appuser_home() = %s" % mod.appuser_home()) +if mod._app_home("u_moved") != "/home/u_moved": + bad.append("a moved home is not used") +if not mod._app_home("u_fresh").startswith("/home/shared_user/"): + bad.append("a new account does not land in /home/shared_user") +print("BAD: " + "; ".join(bad) if bad else "OK") +PYAH +)" + _apo="$(printf '%s\n' "$_apo" | tail -1)" + [ "$_apo" = OK ] || _p="$_p;application home resolution: $_apo" + + # AN APPLICATION USER'S HOME IS NOT A BUILD TREE. add_package_user + # creates the account and puts the home where PACKAGE homes go, so + # u_claude landed in /usr/src. It holds a browser profile, not a build. + _rah="$(_slice_def "$pm_src" _relocate_app_home)" + [ -n "$_rah" ] || _p="$_p;an application user's home is left in the package tree" + printf '%s' "$_rah" | grep -q 'usermod", "-d", want, "-m"' \ + || _p="$_p;the home is moved without taking its contents" + printf '%s' "$_rah" | grep -q 'os.path.exists(want)' \ + || _p="$_p;it would move a home onto an existing directory" + printf '%s' "$_rah" | grep -q 'startswith(base' \ + || _p="$_p;it would move a home that is already somewhere deliberate" + grep -q '_relocate_app_home(u)' "$pm_src" \ + || _p="$_p;the relocation is defined and never called" + # a probe with an unexpanded \$( in it tests nothing + grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'expect=' \ + || _p="$_p;imv is installed without proving it can use wayland" + grep -A6 -E '^git +imv ' "$_sw2" | grep -q 'ldd /usr/bin/imv' \ + || _p="$_p;the imv probe relies on a substitution the parser eats" + for _e in light wl-clipboard; do + grep -qE "^git +$_e " "$_sw2" || _p="$_p;$_e is not in the sway stack" + done + # light writes to /sys/class/backlight, which root owns: the udev rule is + # what makes it usable without setuid + grep -A6 '^git light ' "$_sw2" | grep -q -- '--with-udev' \ + || _p="$_p;light is built without the udev rule that grants access" + grep -B8 '^git light ' "$_sw2" | grep -q 'video' \ + || _p="$_p;nothing says which group the backlight rule grants to" + grep -qE '^cfg +fonts' "$_sw2" || _p="$_p;the fonts step is not in the stack" + # -fno-common HAS BEEN GCC'S DEFAULT SINCE GCC 10, and light's helpers.h + # declares light_loglevel without extern: every object defines it and + # the link fails. -fcommon is not a workaround for a broken package, it + # is the behaviour the code was written against -- and the entry has to + # say which, or the next reader deletes it (1.14.189). + grep -A8 '^git light ' "$_sw2" | grep -q -- '-fcommon' \ + || _p="$_p;light will not link on any compiler from GCC 10 onwards" + # ...AND IN THE ENVIRONMENT, NOT ON THE CONFIGURE LINE. light's + # autogen.sh ends by running ./config.status --recheck and then `make`, + # so the compile happens BEFORE the ./configure in the build line is + # reached: CFLAGS passed there fixed nothing and the link failed exactly + # as before (1.14.191). + grep -A8 '^git light ' "$_sw2" | grep -q "export CFLAGS=" \ + || _p="$_p;CFLAGS reaches only the configure that runs too late" + grep -A8 '^git light ' "$_sw2" | grep -q 'rm -f config.status' \ + || _p="$_p;autogen.sh will re-run a stale config.status with the old flags" + # and the flag does what the comment says it does + if command -v gcc >/dev/null 2>&1; then + printf 'int lvl;\n' > "$T/h.h" + printf '#include "h.h"\nint main(void){return lvl;}\n' > "$T/a.c" + printf '#include "h.h"\nint f(void){return lvl;}\n' > "$T/b.c" + ( cd "$T" && gcc -c a.c -o a.o && gcc -c b.c -o b.o \ + && gcc a.o b.o -o demo ) >/dev/null 2>&1 \ + && _p="$_p;this compiler links a tentative definition anyway, so the -fcommon claim is wrong here" + ( cd "$T" && gcc -fcommon -c a.c -o a.o && gcc -fcommon -c b.c -o b.o \ + && gcc a.o b.o -o demo ) >/dev/null 2>&1 \ + || _p="$_p;-fcommon does not fix the multiple definition it is there for" + fi + grep -B64 '^git light ' "$_sw2" | grep -q 'multiple definition' \ + || _p="$_p;the reason for -fcommon is not written down" + # OPENSSH IS NETWORKING. It sat in services.stack when the stack that + # installs wpa_supplicant and dhcpcd is where it belongs (1.14.190). + _ns2="$(dirname "$LFS_TOOL")/stacks/network.stack" + _sv2="$(dirname "$LFS_TOOL")/stacks/services.stack" + grep -qE '^book +openssh' "$_ns2" 2>/dev/null \ + || _p="$_p;openssh is not in the network stack" + grep -qE '^book +openssh' "$_sv2" 2>/dev/null \ + && _p="$_p;openssh is in two stacks" + grep -q 'install-sshd' "$_sv2" 2>/dev/null \ + || _p="$_p;openssh is installed with no bootscript to start it" + # A LEGACY iptables ON AN NFT-ONLY KERNEL FAILS EVERY RULE. BLFS + # configures --disable-nftables, the kernel had no ip_tables, and the + # symptom was "Module ip_tables not found" once per rule at boot + # (1.14.195). + # BUILDING NFTABLES SUPPORT DOES NOT SWITCH TO IT. The build log said + # "nftables support: yes", xtables-nft-multi was installed, and + # `iptables --version` still said (legacy) because the plain names stay + # on xtables-legacy-multi. So the package entry proves the BACKEND + # exists and a separate step makes it the one in use (1.14.196). + grep -A2 -E '^book +iptables' "$_ns2" 2>/dev/null | grep -q 'xtables-nft-multi' \ + || _p="$_p;nothing checks that the nft backend was built" + grep -qE '^cfg +iptables-nft' "$_ns2" 2>/dev/null \ + || _p="$_p;the nft backend is built and never selected" + _in="$(dirname "$LFS_TOOL")/stacks/iptables-nft.sh" + if [ -f "$_in" ]; then + bash -n "$_in" || _p="$_p;iptables-nft.sh does not parse" + grep -q 'iptables --version' "$_in" \ + || _p="$_p;it does not check what the switch achieved" + grep -q 'real file, not a symlink' "$_in" \ + || _p="$_p;it would replace a real file it did not install" + grep -q 'xtables-legacy-multi /usr/sbin/iptables' "$_in" \ + || _p="$_p;there is no way back to the legacy backend" + grep -q 'already uses the nft backend' "$_in" \ + || _p="$_p;it relinks on every replay instead of noticing" + # every generic name, not just iptables: a legacy iptables-restore + # reading an nft ruleset is the same failure one command later + for _n in iptables-save iptables-restore ip6tables ip6tables-save; do + grep -q "$_n" "$_in" || _p="$_p;$_n is left on the legacy backend" + done + else + _p="$_p;there is no step that selects the nft backend" + fi + grep -B14 -E '^book +iptables' "$_ns2" 2>/dev/null | grep -q 'disable-nftables' \ + || _p="$_p;the book's --disable-nftables is not mentioned where it bites" + _nw2="$(dirname "$LFS_TOOL")/stacks/net-wifi.sh" + if [ -f "$_nw2" ]; then + grep -q 'ip_tables(legacy)' "$_nw2" \ + || _p="$_p;the report does not say what the kernel offers" + grep -q 'every rule will fail' "$_nw2" \ + || _p="$_p;a userspace/kernel mismatch is not called out" + # both directions, not just the one that happened + [ "$(grep -c 'every rule will fail' "$_nw2")" -ge 2 ] \ + || _p="$_p;only one of the two mismatch directions is reported" + # and the variant parses out of the real strings + for _pair in "iptables v1.8.12 (legacy)|legacy" "iptables v1.8.12 (nf_tables)|nft"; do + _in="${_pair%|*}"; _want="${_pair#*|}" + case "$_in" in + *nf_tables*) _got=nft ;; + *legacy*) _got=legacy ;; + *) _got=unknown ;; + esac + [ "$_got" = "$_want" ] \ + || _p="$_p;'$_in' reads as $_got, not $_want" + done + fi + + # AND ITS ACCOUNT IS DECLARED, not typed out of a failure box hours later + _ac="$(dirname "$LFS_TOOL")/stacks/accounts.conf" + _as="$(dirname "$LFS_TOOL")/stacks/accounts.sh" + grep -qE '^cfg +accounts' "$_ns2" 2>/dev/null \ + || _p="$_p;nothing creates the declared system accounts" + grep -q '^sshd:' "$_ac" 2>/dev/null \ + || _p="$_p;the sshd account is not declared anywhere" + if [ -f "$_as" ]; then + bash -n "$_as" || _p="$_p;accounts.sh does not parse" + _ord="$(python3 - "$pm_src" "$_ns2" <<'PYORD2' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +# BUILD entries only. Since every stack includes avoid.stack, an `avoid +# openssh` line sat at index 50 while the real `book openssh` was at 73 -- +# so this compared an exclusion against a build step and called the order +# wrong (1.14.241). +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2]) + if r["kind"] != "avoid"] +print("OK" if "accounts" in o and "openssh" in o + and o.index("accounts") < o.index("openssh") else "LATE") +PYORD2 +)" + [ "$_ord" = OK ] \ + || _p="$_p;the accounts step runs after openssh, so the build still refuses" + grep -q -- '"$@" -r' "$_as" \ + || _p="$_p;an account with no declared number gets an ordinary uid" + grep -q 'left alone: it may own files' "$_as" \ + || _p="$_p;it would renumber an account that already owns files" + grep -q 'is already group' "$_as" \ + || _p="$_p;a gid collision does not say what holds the number" + printf 'zz-test-acct:-:-:-:-:suite\n' > "$T/acc.conf" + ACCOUNTS_CONF="$T/acc.conf" bash "$_as" >/dev/null 2>&1 + if id zz-test-acct >/dev/null 2>&1; then + [ "$(id -u zz-test-acct)" -lt 1000 ] \ + || _p="$_p;a declared system account got uid $(id -u zz-test-acct)" + ACCOUNTS_CONF="$T/acc.conf" bash "$_as" 2>&1 \ + | grep -q '0 created, 1 already there' \ + || _p="$_p;a second run does not leave the account alone" + userdel zz-test-acct 2>/dev/null; groupdel zz-test-acct 2>/dev/null + else + _p="$_p;accounts.sh did not create the account it was given" + fi + fi + # A URL THAT WAS CHECKED SAYS SO. haikarainen/light does not resolve + # while three other github URLs in this stack do, and the clone failed + # with "could not read Username" -- GitHub's answer for a repository it + # does not have. An unverified URL that looks verified costs someone a + # build (1.14.186). + # -B32: the note is as long as the finding needed, and -B14 cut it off + # the finding moved from "this URL is dead" to "upstream moved, and the + # new host could not be reached from here": both are provenance, and the + # assertion is that the entry carries it + grep -B64 '^git light ' "$_sw2" \ + | grep -qiE 'did not resolve|could NOT be verified|upstream moved' \ + || _p="$_p;the light URL carries no note about how it was checked" + grep -B64 '^git light ' "$_sw2" | grep -q 'git ls-remote' \ + || _p="$_p;it does not say how to check the URL" +fi +if [ -f "$_fs" ]; then + bash -n "$_fs" || _p="$_p;fonts.sh does not parse" + # THREE UNPACKERS: unzip is a separate BLFS page and nothing requires it, + # so depending on one is how this fails on a system that could unpack + grep -q 'command -v bsdtar' "$_fs" || _p="$_p;fonts.sh cannot use libarchive" + grep -q 'zipfile' "$_fs" || _p="$_p;fonts.sh has no unpacker that is always present" + # a font present but uncached is a font nothing can find + grep -q 'fc-cache -f' "$_fs" || _p="$_p;the font cache is never rebuilt" + grep -q 'fc-list' "$_fs" || _p="$_p;it does not check that fontconfig sees them" + # already installed is done: this runs on every replay + grep -q 'already installed' "$_fs" || _p="$_p;it re-downloads fonts on every run" + # one bad URL must not stop the rest + grep -q 'continue' "$_fs" || _p="$_p;one moved URL stops every other font" + # A GLOB DOES NOT CROSS A SLASH. `*otfs/*.otf` could never match + # fontawesome-free-6.7.2-desktop/otfs/fa-solid-900.otf -- the layout + # every release actually has (1.14.187). + grep -q 'find . -type f -path' "$_fs" \ + || _p="$_p;the archive is searched with a glob that cannot cross a directory" + _fx="$T/fx"; rm -rf "$_fx"; mkdir -p "$_fx/pkg-6.7.2-desktop/otfs" + : > "$_fx/pkg-6.7.2-desktop/otfs/fa-solid-900.otf" + _hit="$(cd "$_fx" && find . -type f -path "*/*.otf" -o -type f -path "./*.otf" | head -1)" + [ -n "$_hit" ] || _p="$_p;find cannot see a font one directory down either" + # and the Noto lines had to be checked, not guessed + grep -q 'WHAT WAS CHECKED, AND HOW' "$_fs" \ + || _p="$_p;the URLs do not say whether they were verified" + grep -q 'notofonts.github.io' "$_fs" \ + || _p="$_p;the release-asset URLs that did not exist are not recorded as such" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "backlight, clipboard and fonts are in the stack, and the fonts step survives a moved URL" +fi + +# ---- a feature is not proved by a file existing (1.14.156) ----------------- # +# ffmpeg and SDL3 came out with no ALSA support on a machine where alsa-lib +# was built and verified. Both autodetect ALSA; both were built BEFORE it, +# because `book alsa-lib` sat at entry 7 while ffmpeg is built inside mesa's +# plan (entry 2) and SDL3 inside gstreamer's. Nothing failed: they +# configured cleanly, installed cleanly, passed have=, and had no sound. +# +# Two things were missing. An order (the stack already states this rule for +# harfbuzz/libass: what an earlier entry installed is simply there), and a +# way to check what was BUILT rather than whether a build happened. +_p="" +# THE STACK IS NOW TWO FILES. The media chain moved into media.stack +# and sway.stack pulls it in with `include media`, so a grep for an +# ENTRY has to read both -- eight assertions here failed the moment +# the split landed, which is the split being noticed rather than a +# fault in it (1.14.223). +_sw="$_DESKTOP_STACK" +if [ -f "$_sw" ] && [ -f "$pm_src" ]; then + # the order, read out of the stack itself + _ord="$(python3 - "$pm_src" "$_sw" <<'PYORD' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +rows = [r for r in mod._parse_stack_file(sys.argv[2])] +pos = {} +for i, r in enumerate(rows): + pos.setdefault(r["target"], i) +# a library that something autodetects must come before every consumer of it +for lib, users in (("alsa-lib", ("ffmpeg", "sdl3", "mpv", "gstreamer10", + "pipewire", "mesa", "wireplumber")),): + if lib not in pos: + print("MISSING %s" % lib); continue + for u in users: + if u in pos and pos[u] < pos[lib]: + print("LATE %s after %s" % (lib, u)) +print("OK") +PYORD +)" + case "$_ord" in + *"LATE alsa-lib"*) _p="$_p;alsa-lib is built after something that autodetects it: ${_ord%%$'\n'*}" ;; + *"MISSING alsa-lib"*) _p="$_p;alsa-lib is not in the sway stack at all" ;; + esac + # the probe: expect= must be parsed, kept for the runner, and enforced + _ex="$(python3 - "$pm_src" "$_sw" <<'PYEX' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +n = 0 +for r in mod._parse_stack_file(sys.argv[2]): + if r["opts"].get("expect"): + n += 1 + # a probe must run something and test its result -- a bare command + # name with no pipe, grep or redirect is not a check. (First version + # only allowed pipes and grep, so `gst-inspect-1.0 webrtcbin + # >/dev/null 2>&1` -- a perfectly good probe -- was called thin.) + _e = r["opts"]["expect"] + # (`pkg-config --exists` IS a test -- its exit status is the answer; + # librice and xxhash were called thin once the check read every + # desktop stack, 1.14.320.) + if any(t in _e for t in ("|", "grep", ">/dev/null", "test ", "[ ", + "--exists")): + continue + print("THIN %s" % r["target"]) +print("COUNT %d" % n) +PYEX +)" + case "$_ex" in + *"COUNT 0"*) _p="$_p;nothing in the stack checks what it was built with" ;; + esac + case "$_ex" in *THIN*) _p="$_p;an expect= probe does not actually test anything" ;; esac + grep -q 'expect' "$pm_src" || _p="$_p;the runner does not know about expect=" + # A FAILED PROBE MUST NOT BE RECORDED AS DONE, or the entry disappears + # from every later run and the missing feature becomes permanent. + python3 - "$pm_src" <<'PYRC' || _p="$_p;a failed feature check is still recorded as done" +import sys, re +src = open(sys.argv[1]).read() +i = src.index('_expect = e["opts"].get("expect")') +j = src.index("_stack_mark_done(path, e[\"_key\"])", i) +blk = src[i:j] +sys.exit(0 if "_expect_ok" in blk and "not _expect_ok" in blk else 1) +PYRC + # and the key must not reach a build as an option + grep -q 'opts.pop("expect"' "$pm_src" \ + || _p="$_p;expect= can be passed to meson as -Dexpect=" +fi +# the probes themselves have to work on the output they claim to read +_o="$(printf 'Devices:\n D alsa ALSA audio output\n')" +printf '%s' "$_o" | grep -qw alsa || _p="$_p;the ffmpeg probe does not match ffmpeg -devices output" +printf 'Devices:\n D oss\n' | grep -qw alsa && _p="$_p;the ffmpeg probe matches output with no alsa" +printf 'libasound.so.2\0' > "$T/fakelib.so" +strings "$T/fakelib.so" 2>/dev/null | grep -q libasound \ + || _p="$_p;the SDL3 probe cannot see a dlopen name in a library" +# A LIBRARY IS NOT A PLUGIN. gst-plugins-bad always builds +# libgstwebrtc-1.0.so, so the file list and `verify` are clean while the +# webrtcbin ELEMENT -- the thing a browser loads -- is absent, because +# libnice was not installed when it configured. The browser said so; the +# system did not (1.14.157). +_gst="$(python3 - "$pm_src" "$_sw" <<'PYGST' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +pos, exp = {}, {} +for i, r in enumerate(mod._parse_stack_file(sys.argv[2])): + pos.setdefault(r["target"], i) + if r["opts"].get("expect"): + exp[r["target"]] = r["opts"]["expect"] +# each codec and transport library before the plugin package that looks for it +# gst10-libav needs ffmpeg AND gst-plugins-base, and it was not in the stack +# at all -- so H.264 decoded by nothing and the symptom was a black frame +# (1.14.167). +# absent entirely is the case that started this: name it as missing rather +# than as an unproven probe +if "gst10-libav" not in pos: + print("MISSING gst10-libav") +for lib, pkg in (("ffmpeg", "gst10-libav"), ("gst10-plugins-base", "gst10-libav"), + ("opus", "gst10-plugins-base"), ("libvpx", "gst10-plugins-good"), + ("libnice", "gst10-plugins-bad"), ("libsrtp", "gst10-plugins-bad"), + ("usrsctp", "gst10-plugins-bad")): + if lib not in pos: + print("MISSING %s" % lib) + elif pkg in pos and pos[pkg] < pos[lib]: + print("LATE %s after %s" % (lib, pkg)) +# and the element probe, not a file probe +b = exp.get("gst10-plugins-bad", "") +if "webrtcbin" not in b: + print("NOPROBE gst10-plugins-bad") +if "gst-inspect" not in exp.get("gst10-plugins-good", ""): + print("NOPROBE gst10-plugins-good") +if "avdec" not in exp.get("gst10-libav", ""): + print("NOPROBE gst10-libav") +# DECODING IS NOT ENCODING. avdec_h264 was present while x264enc, +# openh264enc and openh264dec were all missing, so the browser could play +# H.264 and not produce it -- which is what a camera site needs to publish. +# Checked against BLFS 13.1: gst10-plugins-ugly IS a book page, openh264 is +# not (1.14.199). +if "gst10-plugins-ugly" not in pos: + print("MISSING gst10-plugins-ugly") +elif "x264enc" not in exp.get("gst10-plugins-ugly", ""): + print("NOPROBE gst10-plugins-ugly") +if "openh264" not in pos: + print("MISSING openh264") +elif "gst10-plugins-bad" in pos and pos["openh264"] > pos["gst10-plugins-bad"]: + print("LATE openh264 after gst10-plugins-bad") +print("OK") +PYGST +)" +case "$_gst" in + *"LATE "*) _p="$_p;a WebRTC library is built after the plugins that look for it: ${_gst%%$'\n'*}" ;; + *"MISSING "*) _p="$_p;the WebRTC chain is incomplete: ${_gst%%$'\n'*}" ;; +esac +case "$_gst" in + *"NOPROBE "*) _p="$_p;a gst plugin package does not prove its elements registered" ;; +esac +# AN OUT-OF-BOOK ENTRY CARRIES ITS OWN TRUTH. gst-plugins-rs has no BLFS +# page, so the ref is not a pin unless we say so, the crates come from the +# network, and the plugin list is a choice -- all three belong next to the +# entry (1.14.172). +if [ -f "$_sw" ]; then + grep -A26 '^git gst-plugins-rs' "$_sw" | grep -q 'cargo cinstall' \ + || _p="$_p;the rust plugins are built without cargo-c" + # A PLUGIN IS A SHARED OBJECT. Without --library-type cdylib cargo-c + # also installs a static libgstrs*.a into /usr/lib/gstreamer-1.0, which + # nothing loads and which is what the first real run died on. + grep -A26 '^git gst-plugins-rs' "$_sw" | grep -q -- '--library-type cdylib' \ + || _p="$_p;cargo-c installs a static library nothing loads" + # audiornnoise IS NOT ITS OWN PLUGIN. It lives in gst-plugin-audiofx, so + # a -p naming the element would fail before anything built (1.14.174). + _rsb="$(grep -A26 '^git gst-plugins-rs' "$_sw")" + printf '%s' "$_rsb" | grep -q 'gst-plugin-audiofx' \ + || _p="$_p;audiornnoise is asked for without the plugin that carries it" + printf '%s' "$_rsb" | grep -q -- '-p gst-plugin-audiornnoise' \ + && _p="$_p;audiornnoise is named as a plugin, which it is not" + # every element in the probe must be asked for by some -p + for _el in webrtcsink audiornnoise; do + printf '%s' "$_rsb" | grep -q "$_el" \ + || _p="$_p;the probe checks $_el and nothing builds it" + done + # and the build and install lines must ask for the SAME set: a plugin + # built and not installed is the most confusing outcome available + _b="$(printf '%s' "$_rsb" | grep -o 'build="[^"]*"' | grep -o -- '-p [a-z0-9-]*' | sort)" + _i="$(printf '%s' "$_rsb" | grep -o 'install="[^"]*"' | grep -o -- '-p [a-z0-9-]*' | sort)" + [ "$_b" = "$_i" ] \ + || _p="$_p;the build and install lines ask for different plugins" + grep -B20 '^git gst-plugins-rs' "$_sw" | grep -q 'not a pin' \ + || _p="$_p;an unpinned ref is not called out as unpinned" + grep -B20 '^git gst-plugins-rs' "$_sw" | grep -q 'downloads crates' \ + || _p="$_p;nothing says the rust build needs the network" + grep -q 'gst-plugins-rs is not built (optional)' \ + "$(dirname "$LFS_TOOL")/stacks/webrtc.sh" \ + || _p="$_p;a missing optional plugin set fails the chain report" +fi + +# CARGO NAMES THE DESTINATION IN BACKTICKS, on the line above the reason: +# failed to copy `...a` to `/usr/lib/gstreamer-1.0/...a` +# Caused by: +# Permission denied (os error 13) +# No quotes, and no "Permission denied" on the same line -- so the repair +# extracted nothing, ran zero rounds, and still printed "grant access and +# retry" (1.14.173). +_hl5="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl5" ]; then + grep -q 'os error 13' "$_hl5" \ + || _p="$_p;cargo's permission failure is invisible to the repair" + printf '%s\n' \ + 'Error: CliError { error: Some(failed to copy `/src/libgstrsrtp.a` to `/usr/lib/gstreamer-1.0/libgstrsrtp.a`' \ + 'Caused by:' \ + ' Permission denied (os error 13)), exit_code: 101 }' > "$T/cargo.log" + _got="$(grep -aoE 'to `/[^`]+`' "$T/cargo.log" \ + | sed 's/^to `//; s/`$//' | sed 's|/[^/]*$||' | sort -u)" + [ "$_got" = /usr/lib/gstreamer-1.0 ] \ + || _p="$_p;the cargo destination parses to '$_got', not the directory" +fi + +# the chain report: builds nothing, and names the rebuild command +_wr="$(dirname "$LFS_TOOL")/stacks/webrtc.sh" +if [ -f "$_wr" ]; then + bash -n "$_wr" || _p="$_p;webrtc.sh does not parse" + grep -qE '^check +webrtc' "$_sw" || _p="$_p;the chain report is not in the stack" + grep -q 'gst-inspect-1.0' "$_wr" \ + || _p="$_p;it checks for files instead of registered elements" + grep -qE '^[[:space:]]*(install|cp|mv|rm|mkdir|chmod|chown|tee|ln|meson|ninja)[[:space:]]' "$_wr" \ + && _p="$_p;the chain report builds or writes something" + grep -q 'only if' "$_wr" \ + || _p="$_p;a missing element does not say which library it needed" +else + _p="$_p;there is no report of the WebRTC chain" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a library goes in before its consumers, expect= proves the feature, and the chain is reportable" +fi + +# ---- privilege is a group, not a person (1.14.154) ------------------------- # +# The reference machine delegated halt/poweroff/reboot/pm-suspend to a group +# by hand: 4550 root:<prefix>_power, plus a boot script re-applying the modes +# on /sys/power/state because /sys does not survive a reboot. That is the +# same shape as seatd-<user>, and it was nowhere in the stacks. +# +# Two pieces of it could never have worked, and both looked like working +# configuration: +# SUBSYSTEM=="power", KERNEL=="state", MODE="4550" -- /sys/power/state has +# no uevent, so udev is never asked; and 550 has no write bit, so a group +# member could not write "mem" to it even if it had applied. +# ACTION=="add", KERNEL=="bootlog", OWNER="..." -- /run/bootlog is a +# file the bootscripts write, not a device. No account is needed. +_p="" +_pw="$(dirname "$LFS_TOOL")/stacks/power.sh" +_sv="$(dirname "$LFS_TOOL")/stacks/services.stack" +if [ -f "$_pw" ]; then + bash -n "$_pw" || _p="$_p;power.sh does not parse" + grep -qE '^cfg +power' "$_sv" 2>/dev/null \ + || _p="$_p;the power step is not in the services stack" + grep -q '4550' "$_pw" || _p="$_p;the power binaries are not setuid root" + grep -q 'chown "root:\$grp"' "$_pw" \ + || _p="$_p;a setuid binary is left owned by something other than root" + # THE GROUP NAME IS READ, NOT DEFAULTED. Reporting a built-in default as + # the configured value is a bug this project has already paid for. + grep -q 'collector_prefix' "$_pw" \ + || _p="$_p;the group name ignores the collector prefix" + grep -q 'POWER_GROUP' "$_pw" || _p="$_p;the group name cannot be overridden" + grep -q 'no collector_prefix in' "$_pw" \ + || _p="$_p;a missing prefix silently becomes a default group name" + # 0660 on the sysfs attribute: writing to it IS the suspend + grep -q 'chmod 0660' "$_pw" \ + || _p="$_p;the power interface is not writable by the group" + grep -qE 'chmod 0?550 .*power/state' "$_pw" \ + && _p="$_p;the power interface is set read-only" + # ONE file, symlinked -- the machine had a second copy under rc3.d, which + # is a second place to edit + grep -q 'ln -sf ../init.d/power-perms' "$_pw" \ + || _p="$_p;the boot script is copied into the runlevel instead of linked" + [ "$(grep -c 'cat > "\$init"' "$_pw")" = 1 ] \ + || _p="$_p;the boot script is written in more than one place" + # usermod has returned 0 and changed nothing on this project, three times + grep -q 'id -nG' "$_pw" || _p="$_p;the group membership is not read back" + # and the two rules that cannot fire are named + grep -q 'KERNEL==\"state\"' "$_pw" \ + || _p="$_p;a udev rule that cannot fire is left unexplained" + grep -q 'KERNEL==\"bootlog\"' "$_pw" \ + || _p="$_p;the bootlog rule is left looking like something that works" + grep -q 'rc.site' "$_pw" \ + || _p="$_p;it does not say where the boot log IS configured" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "power is delegated to a group, by the step, with the dead rules named" +fi + +# ---- something has to fetch an address (1.14.153) -------------------------- # +# The wireless link came up and `wlp3s0` would not go anywhere: the network +# stack had wpa_supplicant, libnl, iptables and net-tools, and no DHCP +# client at all -- while `lfs config network dhcp` writes no static +# interface files precisely because a client from BLFS is supposed to follow. +# Nothing followed. +_p="" +_ns="$(dirname "$LFS_TOOL")/stacks/network.stack" +if [ -f "$_ns" ]; then + grep -qE '^(book|tar|git) +dhcpcd\b' "$_ns" \ + || _p="$_p;the network stack still has no DHCP client" + # A PACKAGE IS NOT A SERVICE. dhcpcd installs no /lib/services/dhcpcd; + # that script comes from blfs-bootscripts' install-service-dhcpcd, and + # without it SERVICE=dhcpcd (or a wpa hand-off to it) has nothing to run. + # Adding the package was half the job (1.14.155). + grep -q 'install-service-dhcpcd' "$_sv" 2>/dev/null \ + || _p="$_p;the dhcpcd service script is never installed" + grep -q 'install-service-wpa' "$_sv" 2>/dev/null \ + || _p="$_p;the wpa service script is never installed" + # and the interview's promise must name something that exists in it + _dh="$(grep -oE 'dhcpcd' "$(dirname "$LFS_TOOL")/lfs" | head -n1)" + [ -n "$_dh" ] && ! grep -q 'dhcpcd' "$_ns" \ + && _p="$_p;lfs promises dhcpcd from BLFS and no stack provides it" +fi +_nw="$(dirname "$LFS_TOOL")/stacks/net-wifi.sh" +if [ -f "$_nw" ]; then + bash -n "$_nw" || _p="$_p;net-wifi.sh does not parse" + grep -qE '^check +net-wifi' "$_ns" 2>/dev/null \ + || _p="$_p;the wifi check is not in the stack that installs wifi" + # THE SERVICE NAMES COME OFF THE DISK. Which variable the wpa service + # chains through differs between bootscript releases, and a remembered + # one is how you get a joined link with no address. + grep -q 'SERVICE' "$_nw" || _p="$_p;the ifconfig SERVICE= is never checked" + grep -q 'services' "$_nw" || _p="$_p;it does not look at the installed services" + # predictable interface names: a file called ifconfig.wifi0 does nothing + # for wlp3s0 + grep -q '/sys/class/net' "$_nw" \ + || _p="$_p;it assumes an interface name instead of reading the kernel's" + grep -q 'phy80211' "$_nw" \ + || _p="$_p;wireless interfaces are identified by name rather than by sysfs" + # IT MUST WRITE NOTHING: the files hold the user's SSID and key. + # (First version of this assertion matched `>/dev/null` and a `>` inside + # a quoted suggestion -- text about a command is not the command, which + # is a rule this suite has caught five filters and one injector on.) + grep -qE '^[[:space:]]*(install|cp|mv|rm|mkdir|chmod|chown|tee|ln)[[:space:]]' "$_nw" \ + && _p="$_p;a check writes to the system" + # a check must be quiet about what it cannot know and loud about files + grep -q 'chmod 600' "$_nw" \ + || _p="$_p;a world-readable psk file is not called out" + # A STEP MUST NOT FAIL FOR WORK IT WAS NEVER GOING TO DO. This check + # stopped the stack at 8 of 9 because no /etc/sysconfig/ifconfig.wlan0 + # existed -- a file that holds the user's SSID. Two levels now: !! is + # the stack's and fails, -> is yours and does not. + grep -q '^todo()' "$_nw" \ + || _p="$_p;the user's own files still fail the step" + # GIT MUST NEVER ASK A QUESTION IN A BUILD: on a terminal that hangs, + # and in a log it reads like a credentials problem on a public repo. + _ph="$(dirname "$LFS_TOOL")/lfs-phases" + if [ -f "$_ph" ]; then + grep -q 'GIT_TERMINAL_PROMPT=0' "$_ph" \ + || _p="$_p;a clone can still stop and ask for a password" + grep -q 'PATH IS WRONG' "$_ph" \ + || _p="$_p;a 401 on a public repository is not explained" + grep -q 'git ls-remote \$link' "$_ph" \ + || _p="$_p;the failure does not say how to test the URL" + # AN EMPTY LIST UNDER A HEADING IS A RIDDLE. A fork with no tags + # printed "Recent tags:" and then nothing, and the branch list + # offered `origin`, which is not a ref anyone can check out. + grep -q 'none at all' "$_ph" \ + || _p="$_p;a repository with no tags prints an empty list" + grep -q 'HEAD' "$_ph" \ + || _p="$_p;the branch list offers the remote name and HEAD as refs" + fi + grep -q 'todo "write \$f' "$_nw" \ + || _p="$_p;a missing ifconfig file is still the stack's failure" + # THE INTERFACE FILE HOLDS NO SECRET, so there is no reason to make the + # reader work out its syntax: print it, with the interface that exists + # and the hand-off variable this machine's wpa service reads (1.14.185). + grep -q 'this is the whole file' "$_nw" \ + || _p="$_p;it asks for a file without saying what goes in it" + grep -q 'IFACE=' "$_nw" || _p="$_p;the printed file names no interface" + grep -q '_chain_var' "$_nw" \ + || _p="$_p;the printed file hard-codes the hand-off variable" + # and a file with no key cannot be a leaked key: it said both, two lines + # apart, about one file + grep -q 'it holds a key and is world-readable' "$_nw" \ + || _p="$_p;the permission warning is not tied to actually holding a key" + grep -q '__\*__' "$_nw" \ + || _p="$_p;a placeholder template is reported as a broken config" +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the network stack can actually get an address, and says what is missing" +fi + +# ---- firmware is answerable in a chroot, and fixable without a kernel build - # +# FIRMWARE_ONLY="amdgpu amd-ucode amd intel" installed intel/, and the AX200 +# still had no firmware: +# Direct firmware load for iwlwifi-cc-a0-77.ucode failed with error -2 +# no suitable firmware found! +# The driver asks for a FLAT path, so a directory named `intel` never +# satisfies it -- and there was no way to install one blob without building a +# whole kernel. The running kernel printed the exact path it wanted; the +# checkout's git index knows where that file is. Ask both. +_p="" +_k4="$(dirname "$LFS_TOOL")/lfs-kernel" +if [ -f "$_k4" ]; then + bash -n "$_k4" || _p="$_p;lfs-kernel does not parse" + grep -q '^ firmware)' "$_k4" \ + || _p="$_p;there is no firmware subcommand" + grep -q 'lfs-kernel firmware' "$_k4" \ + || _p="$_p;the firmware subcommand is not in the help" + # --help must not silently drop it: the slice used to be a line range + bash "$_k4" --help 2>/dev/null | grep -q 'firmware --missing' \ + || _p="$_p;--help does not mention firmware --missing" + # the functions must be defined BEFORE the build runs, or a firmware-only + # run would have to build a kernel to reach them + _defl="$(grep -n '^_install_firmware()' "$_k4" | cut -d: -f1)" + _bldl="$(grep -n '^echo "# kernel \$VERSION -> \$ESP' "$_k4" | cut -d: -f1)" + [ -n "$_defl" ] && [ -n "$_bldl" ] && [ "$_defl" -lt "$_bldl" ] \ + || _p="$_p;the firmware code is only reachable after a kernel build" + # it reads the failures out of dmesg, and strips the trailing punctuation + _rq="$(_slice_fn "$_k4" _fw_requests)" + [ -n "$_rq" ] || _p="$_p;nothing reads what the kernel asked for" + _got="$( eval "$_rq" + dmesg() { printf '%s\n' \ + "[ 4.051584] iwlwifi 0000:03:00.0: Direct firmware load for iwlwifi-cc-a0-77.ucode failed with error -2" \ + "[ 1.698606] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2" \ + "[ 4.073516] Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi"; } + _fw_requests )" + case "$_got" in + *iwlwifi-cc-a0-77.ucode*) ;; + *) _p="$_p;the wireless firmware request is not extracted from dmesg" ;; + esac + case "$_got" in + *regulatory.db*) ;; + *) _p="$_p;the regulatory database request is not extracted" ;; + esac + # a firmware that LOADED is not a missing one + case "$_got" in + *ibt-20-1-3*) _p="$_p;a successful firmware load is reported as missing" ;; + esac + # the index, not the working tree: a sparse checkout has removed the very + # file being looked for + _ms="$(_slice_fn "$_k4" _firmware_missing)" + printf '%s' "$_ms" | grep -q '_fw_index' \ + || _p="$_p;it searches the checkout instead of asking the index" + # what it installs is the path the DRIVER asked for + _pl="$(_slice_fn "$_k4" _fw_place)" + printf '%s' "$_pl" | grep -q '/lib/firmware/\$req' \ + || _p="$_p;the blob is not placed at the path the driver asked for" + # a file in the index but not in the sparse checkout must still be + # installable -- and by ONE blob. The first version widened the sparse + # checkout and fell back to `sparse-checkout disable`, which in cone mode + # would have materialised all ten gigabytes to fix one 2 MB blob. + printf '%s' "$_pl" | grep -q "show 'HEAD:" \ + || _p="$_p;a file in the index but not on disk cannot be installed" + printf '%s' "$_pl" | grep -q 'sparse-checkout disable' \ + && _p="$_p;the fallback would check out the whole of linux-firmware" + printf '%s' "$_pl" | grep -q 'stat -c %s' \ + || _p="$_p;the copy is not confirmed by reading the file back" + # ONE `local` PER DERIVED VALUE. `local a="$2" d="/lib/firmware/$a"` + # reads $a before this local assigned it, and set -u killed the function + # at the first --fix on a real machine. + printf '%s' "$_pl" | grep -qE '^ *local [a-z_]+="\$1".*dst=' \ + && _p="$_p;dst is derived inside the local that assigns its source" + # git's own message, not a guess: "does not exist in HEAD" and a missing + # blob in a partial clone want different answers + printf '%s' "$_pl" | grep -q '_gerr' \ + || _p="$_p;git's error is discarded and replaced with a guess" + # THE REPOSITORY IS THE FIRMWARE ACCOUNT'S AND THIS RUNS AS ROOT, which + # git refuses: the empty result was reported as "linux-firmware does not + # have a file by that name at all", about a file that was right there. + _ix="$(_slice_fn "$_k4" _fw_index)" + [ -n "$_ix" ] || _p="$_p;there is no single place that asks the index" + printf '%s' "$_ix" | grep -q 'safe.directory' \ + || _p="$_p;git will refuse the account's repository when run as root" + # ls-TREE, not ls-files: with a sparse index git collapses an excluded + # directory into one entry, so every nested blob looked absent -- 56 + # files reported as "no file by that name" while they were all there. + printf '%s' "$_ix" | grep -q 'ls-tree -r' \ + || _p="$_p;a sparse index hides nested blobs from the lookup" + printf '%s' "$_ix" | grep -q '_FW_INDEX_ERR' \ + || _p="$_p;a failed index lookup cannot be told from an absent file" + printf '%s' "$_ms" | grep -q 'UNKNOWN -- not absent' \ + || _p="$_p;a failed lookup is still reported as a fact" + # the scratch file must not land in someone else's checkout + printf '%s' "$_ix" | grep -q 'mktemp' \ + || _p="$_p;the index listing is written inside the repository" + # A CHROOT'S dmesg IS THE HOST'S -- and a chroot is where this work + # happens, so refusing to answer there is no answer. The tree can be + # asked directly: its modules declare their firmware, and the hardware is + # in /sys. Every line says which source named it. + _wt="$(_slice_fn "$_k4" _fw_wanted)" + [ -n "$_wt" ] || _p="$_p;the list comes from the running boot alone" + printf '%s' "$_wt" | grep -q 'modinfo' \ + || _p="$_p;the tree's own modules are never asked what they need" + printf '%s' "$_wt" | grep -q '/sys/devices' \ + || _p="$_p;every driver's firmware is listed, not this machine's" + printf '%s' "$_wt" | grep -q 'modprobe -S' \ + || _p="$_p;aliases are resolved against the running kernel's modules" + printf '%s' "$_wt" | grep -q 'CONFIG_EXTRA_FIRMWARE' \ + || _p="$_p;firmware the config embeds is not counted" + # ONE dmesg PARSER: _fw_wanted must not re-implement _fw_requests + printf '%s' "$_wt" | grep -q '_fw_requests' \ + || _p="$_p;there are two dmesg parsers to keep in step" + printf '%s' "$_ms" | grep -q 'the HOST kernel' \ + || _p="$_p;a chroot does not say whose boot the boot lines describe" + printf '%s' "$_ms" | grep -q 'modules scanned' \ + || _p="$_p;the report does not say which kernel's modules it read" + # A DECLARATION IS NOT A FAILURE. `modinfo -F firmware amdgpu` lists + # every chip amdgpu supports, so the first modules scan printed 56 + # MISSING lines -- Vega, Aldebaran, five iwlwifi RF families this laptop + # does not have -- each with its own remedy, and concluded "56 missing". + printf '%s' "$_ms" | grep -q 'CANDIDATES, not failures' \ + || _p="$_p;a module's declarations are reported as failures" + printf '%s' "$_wt" | grep -q 'modules:\$m' \ + || _p="$_p;a declared file does not say which module declared it" + printf '%s' "$_ms" | grep -q -- '--fix --all' \ + || _p="$_p;there is no way to install the declared files anyway" + grep -q -- '--all) _FWALL=1' "$_k4" \ + || _p="$_p;the suggested --all is not a flag firmware accepts" + # WHAT IS THERE, NEXT TO WHAT IS NOT. A missing blob on a machine whose + # wifi worked under the previous kernel is either the same file under + # another path (FIRMWARE_ONLY entries keep the repository's, the loader + # wants its own) or an older firmware API the new driver refuses. Both + # change what you do next, and the report said neither. + _nb="$(_slice_fn "$_k4" _fw_neighbours)" + [ -n "$_nb" ] || _p="$_p;nothing looks for what IS installed nearby" + printf '%s' "$_nb" | grep -q 'find /lib/firmware -name' \ + || _p="$_p;the same file under another path is not noticed" + printf '%s' "$_nb" | grep -q 'older' \ + || _p="$_p;an older firmware API on disk is not reported" + # the path is returned in a variable, not on stdout: the caller's $(...) + # swallowed the whole explanation with it + printf '%s' "$_nb" | grep -q '_FW_LOCAL=' \ + || _p="$_p;the explanation is captured instead of printed" + printf '%s' "$_ms" | grep -q '_fw_neighbours "\$req"' \ + || _p="$_p;the neighbour check is never called" + # SAY WHAT IS SATISFIED. A boot line whose file is now on disk was + # skipped in silence, so a run with everything in place looked exactly + # like one that had not looked -- and in a chroot that line is the + # reassurance the reader came for. + printf '%s' "$_ms" | grep -q 'installed now' \ + || _p="$_p;a request that is now satisfied is passed over silently" + # ...and a green run is still not a promise: the driver derives the + # firmware name from ids it reads off the card at run time + printf '%s' "$_ms" | grep -q 'Only a boot of THIS kernel settles it' \ + || _p="$_p;the report reads as a verdict on a kernel that has not run" + # a chroot host can be another distribution on other hardware + printf '%s' "$_ms" | grep -q 'neither confirm nor deny' \ + || _p="$_p;the host's boot log is presented as evidence about this tree" + # WHICH KERNEL: uname -r is the host's in a chroot + _mv="$(_slice_fn "$_k4" _fw_modver)" + printf '%s' "$_mv" | grep -q 'VERSION' \ + || _p="$_p;the modules of the configured kernel are not preferred" + # regulatory.db is NOT in linux-firmware, so "update the checkout" is + # advice that can never work + _el="$(_slice_fn "$_k4" _fw_elsewhere)" + printf '%s' "$_el" | grep -q 'regulatory.db|wireless-regdb' \ + || _p="$_p;regulatory.db is looked for in the wrong project" + # FIRMWARE_ONLY is copied to the same relative path, so it can only + # satisfy a request whose path matches the repository's + printf '%s' "$_ms" | grep -q 'no FIRMWARE_ONLY entry can express this' \ + || _p="$_p;a nested blob with a flat request gets advice that cannot work" + _if="$(_slice_fn "$_k4" _install_firmware)" + printf '%s' "$_if" | grep -q 'cp -a --parents' \ + || _p="$_p;a nested FIRMWARE_ONLY entry is installed with its path dropped" + # and an unchanged FIRMWARE_ONLY is not advice + printf '%s' "$_ms" | grep -q 'if \[ -n "\$sugg" \]' \ + || _p="$_p;it prints the unchanged FIRMWARE_ONLY as though it were a fix" + # every flag the messages suggest is one the command accepts + printf '%s' "$_ms" | grep -q 'firmware --missing --fix' \ + && { grep -q -- '--fix)' "$_k4" \ + || _p="$_p;the suggested --fix is not a flag firmware accepts"; } + # THE REGULATORY DATABASE IS A DIFFERENT PROJECT. Naming it in + # FIRMWARE_ONLY failed the entire firmware install, because a pattern + # matching nothing in the repository is an error -- so the shipped config + # must not name it, and an entry someone else already copied must be + # skipped with a note rather than taken as a failure. + _rg="$(_slice_fn "$_k4" _install_regdb)" + [ -n "$_rg" ] || _p="$_p;nothing installs the regulatory database" + printf '%s' "$_rg" | grep -q 'sort -V | tail -n1' \ + || _p="$_p;the wireless-regdb version is hard-coded instead of asked for" + printf '%s' "$_rg" | grep -q 'stat -c %s /lib/firmware/regulatory.db' \ + || _p="$_p;the regulatory database install is not confirmed" + printf '%s' "$_rg" | grep -q 'REGDB:-yes' \ + || _p="$_p;the regulatory database cannot be turned off" + printf '%s' "$_if" | grep -q '_install_regdb' \ + || _p="$_p;a normal firmware run does not install the database" + printf '%s' "$_if" | grep -q 'is not in linux-firmware' \ + || _p="$_p;a FIRMWARE_ONLY entry for it still fails the whole run" + # in a chroot the boot list is NOT the reliable one, and the advice used + # to say it was, two lines under a note saying the opposite + printf '%s' "$_ms" | grep -q 'boot THIS kernel and run this' \ + || _p="$_p;in a chroot it still points at the host's boot list as evidence" + # AN OPTION IN THE WRONG SECTION FAILS THE WHOLE BUILD. `alsa` reads + # perfectly well under [gst10-plugins-bad] and is a gst-plugins-BASE + # option; meson answers "Unknown option: alsa" and stops (1.14.200). + _wu="$(_slice_def "$pm_src" _warn_unknown_meson_opts)" + [ -n "$_wu" ] || _p="$_p;a machine.conf key is passed without checking it exists" + printf '%s' "$_wu" | grep -q 'NOT passed' \ + || _p="$_p;an unknown option is passed to meson anyway" + printf '%s' "$_wu" | grep -q 'it does have' \ + || _p="$_p;it does not offer the near misses" + printf '%s' "$(_slice_def "$pm_src" _meson_option_names)" | grep -q 'meson.options' \ + || _p="$_p;the option names are not read from the package's own source" + # builtins are not in meson.options and must still be allowed + printf '%s' "$_wu" | grep -q 'wrap_mode' \ + || _p="$_p;a meson built-in like wrap_mode would be dropped" + # and the shipped machine.conf must put alsa where it belongs + grep -A2 '^\[gst10-plugins-base\]' "$_mc5" 2>/dev/null | grep -q 'alsa' \ + || _p="$_p;alsa is not asked for where the option exists" + grep -A4 '^\[gst10-plugins-bad\]' "$_mc5" 2>/dev/null | grep -q '^alsa' \ + && _p="$_p;alsa is still under the section that has no such option" + + # A HARDWARE-ONLY MESA HAS NOTHING TO FALL BACK TO. `gallium-drivers = + # radeonsi` alone gave "libEGL warning: egl: failed to create dri2 + # screen" and then nothing. The book, twice: add softpipe or llvmpipe + # as a backup gallium driver, and swrast as a backup vulkan one + # (1.14.215). + grep -A3 '^\[mesa\]' "$_mc5" 2>/dev/null | grep -q 'gallium-drivers.*llvmpipe\|gallium-drivers.*softpipe' \ + || _p="$_p;mesa is built with no software gallium driver" + grep -A4 '^\[mesa\]' "$_mc5" 2>/dev/null | grep -q 'vulkan-drivers.*swrast' \ + || _p="$_p;mesa is built with no software vulkan driver" + grep -A2 -E '^book +mesa' "$_sw2" | grep -q 'swrast_dri.so' \ + || _p="$_p;nothing proves the software renderer was built" + # the probe must pass when EITHER file is there -- `ls a b` needs both + grep -A2 -E '^book +mesa' "$_sw2" | grep -q -- '-o -e' \ + || _p="$_p;the mesa probe demands both driver files, not either" + # llvmpipe installs as swrast_dri.so, and the comment has to say so or + # the next reader looks for llvmpipe_dri.so and finds nothing + grep -B6 -E '^book +mesa' "$_sw2" | grep -q 'named after the interface' \ + || _p="$_p;nothing explains why the file is not called llvmpipe" + # and the kernel option the book ties to it + _ks2="$(dirname "$LFS_TOOL")/stacks/kernel-sway.sh" + if [ -f "$_ks2" ]; then + bash -n "$_ks2" || _p="$_p;kernel-sway.sh does not parse" + grep -q 'CONFIG_DRM_VGEM' "$_ks2" \ + || _p="$_p;the kernel option llvmpipe needs is not checked" + # reported, not fatal: a machine that never falls back does not need it + grep -q 'need CONFIG_DRM_VGEM' "$_ks2" \ + && _p="$_p;a missing VGEM fails the stack, which is too strong" + fi + + # USE_GSTREAMER_WEBRTC=ON MAKES webkit WANT librice. webkitgtk 2.52 + # takes its ICE from librice rather than libnice when the GStreamer + # WebRTC backend is on, and the BLFS page never hits this because it + # does not enable that backend -- librice is not in the 13.1 book at + # all. libnice IS installed and is what the rest of the chain uses + # (1.14.232). + # USE_GSTREAMER_WEBRTC=ON MAKES webkit WANT librice, which is not in the + # book at all. The option and the package must agree: the option + # without the package fails configure, the package without the option + # wastes the build (1.14.233). + _wk_rice="$(grep -A8 '^\[webkitgtk\]' "$_mc5" 2>/dev/null \ + | grep -oE 'USE_LIBRICE *= *(ON|OFF)' | grep -oE '(ON|OFF)')" + case "$_wk_rice" in + ON) + grep -qE '^git +librice' "$_ms" \ + || _p="$_p;USE_LIBRICE=ON and nothing builds librice" + _ro="$(python3 - "$pm_src" "$_ms" <<'PYRICE' +import sys, types +mod = types.ModuleType("pm"); mod.__file__ = sys.argv[1] +exec(compile(open(sys.argv[1]).read(), sys.argv[1], "exec"), mod.__dict__) +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +print("OK" if "librice" in o and o.index("librice") < o.index("webkitgtk") + else "librice is missing or comes after webkitgtk") +PYRICE +)" + [ "$_ro" = OK ] || _p="$_p;$_ro" + grep -A8 -E '^git +librice' "$_ms" | grep -q 'rice-proto' \ + || _p="$_p;nothing proves librice installed its pkg-config files" + # AND THE URL AND CRATE NAMES WERE CHECKED, after a first + # attempt that guessed the host, the ref and both package + # names and got all four wrong. + grep -A6 -E '^git +librice' "$_ms" | grep -q 'ystreet/librice' \ + || _p="$_p;librice points at a host that answers 401" + grep -A8 -E '^git +librice' "$_ms" | grep -q -- '-p rice-proto -p rice-io' \ + || _p="$_p;the librice crate names are the guessed librice-* ones" + grep -B24 -E '^git +librice' "$_ms" | grep -q 'CHECKED against the repository' \ + || _p="$_p;nothing records that the librice entry was verified" + ;; + OFF) + grep -qE '^git +librice' "$_ms" \ + && _p="$_p;librice is built and USE_LIBRICE is OFF" + ;; + *) _p="$_p;USE_LIBRICE is not set either way for webkitgtk" ;; + esac + + # x264enc EXISTS ONLY WITH -D gpl=enabled, in the book's own words, so + # the shipped machine.conf has to ask for it + _mc5="$(dirname "$LFS_TOOL")/stacks/machine.conf" + grep -A2 '^\[gst10-plugins-ugly\]' "$_mc5" 2>/dev/null | grep -q 'gpl *= *enabled' \ + || _p="$_p;x264enc will not be built: gpl is not enabled for -ugly" + grep -B4 '^\[gst10-plugins-ugly\]' "$_mc5" 2>/dev/null | grep -q 'A)GPL' \ + || _p="$_p;nothing says why that switch matters" + # the chain report must name the encoders, not just the decoder + for _el in x264enc openh264enc; do + grep -q "$_el" "$(dirname "$LFS_TOOL")/stacks/webrtc.sh" \ + || _p="$_p;$_el is not in the chain report" + done + # A GITHUB ARCHIVE URL ENDS IN THE TAG, NOT THE PROJECT. wget saved + # v2.6.0.tar.gz, the unpack looked for the declared + # pkg=openh264-2.6.0.tar.gz, and the run said "nowhere to fetch it from" + # one line under "'v2.6.0.tar.gz' saved [60302243]" (1.14.201). + _ph2="$(dirname "$LFS_TOOL")/lfs-phases" + if [ -f "$_ph2" ]; then + _fe="$(_slice_fn "$_ph2" _pp_fetch)" + printf '%s' "$_fe" | grep -q -- '-O "$out"' \ + || _p="$_p;a download does not land under the name pkg= declares" + # -O leaves an empty file on a 404 and -c would resume it for ever + printf '%s' "$_fe" | grep -q 'rm -f "$out"' \ + || _p="$_p;a failed download leaves an empty file that -c resumes" + grep -q 'what IS here, newest first' "$_ph2" \ + || _p="$_p;a name mismatch still reads as a failed download" + # prove it lands under pkg= + _fx="$T/fx"; rm -rf "$_fx"; mkdir -p "$_fx" + ( cd "$_fx" + _pp_say() { :; } + _pp_mirrors_for() { printf '%s\n' "$1"; } + wget() { _o=""; while [ $# -gt 0 ]; do case "$1" in + -O) _o="$2"; shift 2 ;; -4|-c) shift ;; *) shift ;; esac; done + echo body > "${_o:-wrong-name.tar.gz}"; } + eval "$_fe" + pkg="openh264-2.6.0.tar.gz" + _pp_fetch "https://github.com/cisco/openh264/archive/refs/tags/v2.6.0.tar.gz" "$pkg" + [ -f "$pkg" ] || exit 1 ) \ + || _p="$_p;the fetch still writes the URL's basename, not pkg=" + fi + + # THE BOOK PASSES -D USE_WOFF2=OFF, so a .woff2 icon font renders as a + # blank square with no error anywhere. woff2 IS a book page, so it goes + # in the stack before webkitgtk, and the switch goes in machine.conf -- + # either alone changes nothing (1.14.203). + _ord3="$(python3 - "$pm_src" "$_sw2" <<'PYW' +import sys, types +mod = types.ModuleType("pm") +exec(compile(open(sys.argv[1]).read(), "pm", "exec"), mod.__dict__) +o = [r["target"] for r in mod._parse_stack_file(sys.argv[2])] +print("OK" if "woff2" in o and "webkitgtk" in o + and o.index("woff2") < o.index("webkitgtk") else "BAD") +PYW +)" + [ "$_ord3" = OK ] \ + || _p="$_p;woff2 is missing from the stack or comes after webkitgtk" + grep -A6 '^\[webkitgtk\]' "$_mc5" 2>/dev/null | grep -q 'USE_WOFF2 *= *ON' \ + || _p="$_p;USE_WOFF2 is not turned on, so installing woff2 changes nothing" + grep -A2 -E '^book +webkitgtk' "$_sw2" | grep -q 'woff2' \ + || _p="$_p;nothing proves webkitgtk actually linked woff2" + + # and openh264 is ours: the book does not have it + grep -B8 -E '^tar +openh264' "$_sw2" | grep -q 'NOT in the BLFS book' \ + || _p="$_p;openh264 is not marked as an out-of-book entry" + grep -B8 -E '^tar +openh264' "$_sw2" | grep -qi 'meson path was NOT built' \ + || _p="$_p;the unverified part of the openh264 entry is not said to be unverified" + + # and the shipped config covers this machine's wireless + _kc4="$(dirname "$LFS_TOOL")/kernel.conf" + if [ -f "$_kc4" ]; then + grep -q 'iwlwifi-\*\.ucode' "$_kc4" \ + || _p="$_p;FIRMWARE_ONLY does not cover the flat iwlwifi blobs" + grep -qE '^FIRMWARE_ONLY=.*regulatory' "$_kc4" \ + && _p="$_p;FIRMWARE_ONLY names regulatory.db, which fails the run" + grep -q '^REGDB=' "$_kc4" \ + || _p="$_p;the config cannot ask for the regulatory database" + fi +fi +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "missing firmware is derived from the tree, not only from dmesg, and installed without a kernel build" +fi + # ---- fix them together, not one round each (1.14.83) ----------------------- # # fcron names ONE file at a time. First /etc/fcron.conf must be owned by # root:fcron; fixed that, and the next run said /etc/fcron.allow: Permission @@ -19234,7 +23651,7 @@ fi # what is on PATH; the ENTRY keeps its name because p_minibrowser is the # account already on the user's disk. _p="" -_ssZ="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ssZ="$_DESKTOP_STACK" if [ -f "$_ssZ" ]; then grep -qE '^git +wlroots .*have=/usr/lib/pkgconfig/wlroots-0\.19\.pc' "$_ssZ" \ || _p="$_p;wlroots' have= does not match the version it pins" @@ -20130,7 +24547,7 @@ fi # drm when it is absent, mentions it in a summary line, and builds happily. # The cost of that silence is a full desktop that cannot open a screen. _p="" -_ss6="$(dirname "$LFS_TOOL")/stacks/sway.stack" +_ss6="$_DESKTOP_STACK" _pmDR="$(dirname "$LFS_TOOL")/packagemanager" grep -qE '^book +libdisplay-info' "$_ss6" \ || _p="$_p;libdisplay-info is not in the sway stack" @@ -21150,6 +25567,605 @@ else ok "slow steps announce themselves, and headers install as their own package" fi +# ---- a library something still links against is not an orphan (1.14.304) --- # +# `update --clean` removes what the OLD version installed and the new one did +# not rewrite. webkitgtk rebuilt from the GTK4 port to the GTK3 one leaves +# libwebkitgtk-6.0.so.4 exactly there: same package, same owner, old mtime, +# in the old manifest -- an orphan by every test, and deleting it breaks every +# program built against it with no warning and no way back but a rebuild. +# +# Built for real: a library with a soname, a consumer that records it, and the +# consumer owned by ANOTHER account (a package cleans its own files freely). +_p="" +if command -v gcc >/dev/null 2>&1 && command -v readelf >/dev/null 2>&1; then + _g="$T/rdep"; rm -rf "$_g"; mkdir -p "$_g/bin" "$_g/lib" + printf 'int oldfn(void){return 42;}\n' > "$_g/old.c" + printf 'extern int oldfn(void); int main(void){return oldfn();}\n' > "$_g/user.c" + if gcc -shared -fPIC -Wl,-soname,libzzold.so.4 -o "$_g/lib/libzzold.so.4.13.8" \ + "$_g/old.c" 2>/dev/null \ + && ln -sf libzzold.so.4.13.8 "$_g/lib/libzzold.so.4" \ + && gcc -o "$_g/bin/consumer" "$_g/user.c" "$_g/lib/libzzold.so.4.13.8" 2>/dev/null; then + # linked against the FILE, not -lzzold: that needs a libzzold.so dev + # symlink, and DT_NEEDED records the soname either way + # another owner, when this suite runs as root; otherwise the guard's + # "a package cleans its own files" rule hides the case + chown daemon "$_g/bin/consumer" 2>/dev/null || true + _rd="$(python3 - "$pm_src" "$_g" <<'PYRD' +import importlib.machinery as m, os, sys, time +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +g = sys.argv[2] +lib, link = g + "/lib/libzzold.so.4.13.8", g + "/lib/libzzold.so.4" +pm._ELF_SCAN_DIRS = (g + "/bin", g + "/lib") +gone = [] +pm.remove_paths_as_user = lambda u, paths: gone.extend(paths) +pm.printInfo = pm.printWarning = lambda *a, **k: None +own = os.stat(lib).st_uid +import pwd +user = pwd.getpwuid(own).pw_name +pm._orphan_sweep(user, [lib, link], time.time() + 1000) +print("KEPT" if not gone else "REMOVED") +os.rename(g + "/bin/consumer", g + "/consumer.parked") +gone2 = [] +pm.remove_paths_as_user = lambda u, paths: gone2.extend(paths) +pm._orphan_sweep(user, [lib, link], time.time() + 1000) +print("SWEPT" if len(gone2) == 2 else "STUCK") +PYRD +)" + case "$_rd" in + *KEPT*) ;; + *) _p="$_p;--clean would delete a library another package still links against" ;; + esac + case "$_rd" in + *SWEPT*) ;; + *) _p="$_p;a library nothing links against is no longer cleaned up" ;; + esac + else + _p="$_p;the fixture would not build" + fi + # and the question must be answerable on its own, not only as a side + # effect of --clean + grep -q '^def cmd_used_by' "$pm_src" \ + || _p="$_p;there is no way to ask what links against a library" + grep -q 'SONAME' "$pm_src" \ + || _p="$_p;the soname is guessed from the file name rather than read" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "--clean keeps a library that is still linked against, and sweeps one that is not" + fi +else + skip "no gcc/readelf here: the library-still-in-use guard cannot be built" +fi + +# ---- turning the machine off is a membership (1.14.309) -------------------- # +# halt/poweroff/reboot are 4550 root:<collector>_power: setuid root so they +# can do it, executable only by the group so not everyone may. That is two +# modes a rebuild or a chown destroys silently, and `cfg power` had no +# expect=, so once it was cached "done" nothing ever looked again. +_p="" +_ss="$(dirname "$LFS_TOOL")/stacks/services.stack" +_pw="$(dirname "$LFS_TOOL")/stacks/power.sh" +if [ -f "$_ss" ] && [ -f "$_pw" ]; then + _pex="$(awk "/^cfg[[:space:]]+power/,/^\\\$/" "$_ss" | grep -c "expect=")" + [ "$_pex" -ge 1 ] || _p="$_p;cfg power has no expect=, so a lost setuid bit is never noticed" + grep -q "4550 root:" "$_ss" \ + || _p="$_p;the expect does not check the mode and owner of halt" + grep -q "power-perms" "$_ss" \ + || _p="$_p;the expect does not check the boot script" + # the step itself must still set both halves and re-apply /sys at boot + grep -q "chmod 4550" "$_pw" || _p="$_p;power.sh no longer sets 4550" + grep -q "sys/power/state" "$_pw" \ + || _p="$_p;nothing re-applies the group to /sys/power/state at boot" + # and the expect must be a runnable shell expression + _pexpr="$(sed -n "/^cfg[[:space:]]\+power/,+2p" "$_ss" | sed -n "s/^ *expect='\(.*\)'$/\1/p")" + if [ -n "$_pexpr" ]; then + bash -n -c "$_pexpr" 2>/dev/null || _p="$_p;the power expect= is not valid shell" + else + _p="$_p;the power expect= could not be read back out of the stack file" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "poweroff is a group membership, and the stack re-checks the modes that make it one" + fi +else + skip "no services.stack or power.sh here" +fi + +# ---- chown destroys a setuid bit (1.14.308) -------------------------------- # +# Ownership is attributed BY TIME WINDOW: every file newer than the build +# stamp. A webkitgtk build runs for hours, and anything that touched +# /usr/bin/su in that window -- `lfs-helper setid`, a shadow reinstall, `cfg +# shadow-pam` -- made su "newer than the stamp". It is root-owned, which the +# sweep treated as fair game, so it was chowned to p_webkitgtk: +# -rwx--x--x 1 p_webkitgtk p_webkitgtk /bin/su +# The kernel clears setuid/setgid on chown, so ONE command took the file and +# removed the bit that makes su work. Twice, on this machine. +_p="" +_hl="$(dirname "$LFS_TOOL")/lfs-helper" +if [ -f "$_hl" ]; then + grep -q 'it is setuid/setgid' "$_hl" \ + || _p="$_p;a setuid file can still be taken by the ownership sweep" + grep -q '\[ "\$_root_ok" != 1 \] && \[ "\$cur" = root \]' "$_hl" \ + || _p="$_p;the chown pass does not apply the root-owned rule the manifest does" + # the mode test itself, against real files + _sd="$T/setid"; rm -rf "$_sd"; mkdir -p "$_sd" + _bad="" + for _m in 4711 2755 6755 0711 0755; do + : > "$_sd/f$_m"; chmod "$_m" "$_sd/f$_m" + case "$(stat -c %a "$_sd/f$_m" 2>/dev/null)" in + [1-7][0-7][0-7][0-7]) _hit=skip ;; + *) _hit=take ;; + esac + case "$_m:$_hit" in + 4711:skip|2755:skip|6755:skip|0711:take|0755:take) ;; + *) _bad="$_bad $_m($_hit)" ;; + esac + done + [ -z "$_bad" ] || _p="$_p;the setuid test judges these wrongly:$_bad" + # and the mode that matters is declared, so a repair knows what to restore + _si="$(dirname "$LFS_TOOL")/stacks/setid.conf" + if [ -f "$_si" ]; then + grep -qE '^/usr/bin/su[[:space:]]*\|[[:space:]]*4711' "$_si" \ + || _p="$_p;setid.conf does not declare su as 4711 root" + fi + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the ownership sweep never takes a setuid file, nor a root file a package user could not have written" + fi +else + skip "no lfs-helper here: the ownership sweep cannot be checked" +fi + +# ---- who INSTALLED it, not who holds it (1.14.310) ------------------------- # +# The setuid damage came back a third time, and with it six root-owned +# symlinks: +# lrwxrwxrwx 1 p_webkitgtk p_webkitgtk /usr/sbin/iptables -> xtables-nft-multi +# made by `cfg iptables-nft` on a different day, and recorded in +# /usr/src/pkgusr/p_webkitgtk/pkg.lst. Nothing in the sweep or the lease +# asked WHO INSTALLS a file -- only who owns it now, which is exactly the +# thing that had gone wrong. Two facts that do not depend on current +# ownership: setid.conf names the package behind each setuid file, and a +# symlink belongs to whoever owns its target. +_p="" +_hl3="$(dirname "$LFS_TOOL")/lfs-helper" +_si3="$(dirname "$LFS_TOOL")/stacks/setid.conf" +if [ -f "$_hl3" ] && [ -f "$_si3" ]; then + grep -qE '^/usr/bin/su[[:space:]]*\|[^|]*\|[^|]*\|[^|]*\|[[:space:]]*shadow[[:space:]]*$' "$_si3" \ + || _p="$_p;setid.conf does not name shadow as the package that installs su" + _nocol="$(awk -F'|' '/^[[:space:]]*(#|$)/ { next } + { n = split($0, c, "|"); k = (n >= 5 ? c[5] : "") + gsub(/[[:space:]]/, "", k) + if (k == "") { f = c[1]; gsub(/[[:space:]]/, "", f); print f } }' "$_si3")" + [ -z "$_nocol" ] || _p="$_p;setid.conf lines with no package (leased to nobody): $(echo $_nocol)" + # the reader itself + ( eval "$(_slice_fn "$_hl3" _setid_conf)"; eval "$(_slice_fn "$_hl3" _setid_pkg_for)" + warn() { :; } + SETID_CONF="$_si3" _setid_pkg_for /usr/bin/su | grep -qx shadow ) \ + || _p="$_p;_setid_pkg_for does not read the package column" + # ...and the old three-column reader still sees the group + ( eval "$(_slice_fn "$_hl3" _setid_conf)"; eval "$(_slice_fn "$_hl3" _setid_policy_for)" + SETID_CONF="$_si3" _setid_policy_for /usr/libexec/dbus-daemon-launch-helper \ + | grep -qx '4750|root|messagebus' ) \ + || _p="$_p;the 5th column broke _setid_policy_for's group" + grep -q 'not taking \$p -- setid.conf says' "$_hl3" \ + || _p="$_p;the ownership sweep does not consult setid.conf" + grep -q 'not taking \$p -- it points at' "$_hl3" \ + || _p="$_p;the ownership sweep takes a symlink into another package" + grep -q 'readlink "\$dir/files/\$n"' "$_hl3" \ + || _p="$_p;lease_settle still compares symlinks with cmp" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "provenance beats current ownership: setid.conf's package column and a symlink's target" + fi +else + skip "no lfs-helper/setid.conf here: provenance cannot be checked" +fi + +# ---- disown: taking back what a package cannot have installed (1.14.310) --- # +# The repair for a machine that already went through those doors. `reclaim` +# pushes ownership towards pkg.lst; this one distrusts pkg.lst, because the +# record is what went wrong. Real files, real accounts, real chown. +_p="" +# self-sufficient: a subset that does not include an earlier block must not +# inherit its variables (the block runs alone under `--only`) +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ "$(id -u)" != 0 ]; then + skip "disown (needs root for chown)" +else + _dw="$(python3 - "$pm_src" <<'PYDIS' +import os, sys, tempfile, pwd, importlib.machinery as m + +R = tempfile.mkdtemp(prefix="lfsdisown.", dir="/var/tmp") +# two real accounts stand in for package users; the prefix is turned off so +# the account name IS the package name +accts = [l.pw_name for l in pwd.getpwall() if 0 < l.pw_uid < 1000][:2] +if len(accts) < 2: + print("SKIP no two system accounts"); raise SystemExit +A, B = accts + +os.makedirs(f"{R}/base/pkgusr/{A}") +os.makedirs(f"{R}/base/pkgusr/{B}") # the package the symlink really is +os.makedirs(f"{R}/stacks"); os.makedirs(f"{R}/sbin") +open(f"{R}/conf", "w").write("pkgusr_prefix=\ncollector_prefix=\n") + +# 1. a setuid file setid.conf gives to someone else, already chowned (so its +# setuid bit is gone -- that is what chown does) +open(f"{R}/sbin/su", "w").write("x") +os.chmod(f"{R}/sbin/su", 0o711) +os.chown(f"{R}/sbin/su", pwd.getpwnam(A).pw_uid, -1) +# 2. a symlink whose target belongs to another package +open(f"{R}/sbin/xtables-nft-multi", "w").write("x") +os.chown(f"{R}/sbin/xtables-nft-multi", pwd.getpwnam(B).pw_uid, -1) +os.symlink("xtables-nft-multi", f"{R}/sbin/iptables") +os.lchown(f"{R}/sbin/iptables", pwd.getpwnam(A).pw_uid, -1) +# 3. a file that really is this package's, which must be left alone +open(f"{R}/sbin/mine", "w").write("x") +os.chown(f"{R}/sbin/mine", pwd.getpwnam(A).pw_uid, -1) + +open(f"{R}/stacks/setid.conf", "w").write( + f"{R}/sbin/su | 4711 | root | - | shadow\n") +open(f"{R}/base/pkgusr/{A}/pkg.lst", "w").write( + f"{R}/sbin/su\n{R}/sbin/iptables\n{R}/sbin/mine\n") +os.makedirs(f"{R}/state/manifests") +open(f"{R}/state/manifests/{A}.files", "w").write( + f"{R}/sbin/su\n{R}/sbin/iptables\n{R}/sbin/mine\n") + +os.environ.update(PKGUSR_BASE=f"{R}/base", PKGUSR_CONFIG=f"{R}/conf", + PM_STACK_DIR=f"{R}/stacks", LFS_PKGUSR_DIR=f"{R}/state") +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() + +class Args: + packages = [A]; all = False; run = False +pm.cmd_disown(Args()) +print("DRYRUN-KEPT", pwd.getpwuid(os.lstat(f"{R}/sbin/su").st_uid).pw_name == A) + +Args.run = True +pm.cmd_disown(Args()) +st = os.lstat(f"{R}/sbin/su") +print("SU-ROOT", st.st_uid == 0) +print("SU-MODE", oct(st.st_mode & 0o7777) == "0o4711") +print("LINK-BACK", + pwd.getpwuid(os.lstat(f"{R}/sbin/iptables").st_uid).pw_name == B) +print("KEPT-MINE", + pwd.getpwuid(os.lstat(f"{R}/sbin/mine").st_uid).pw_name == A) +left = open(f"{R}/base/pkgusr/{A}/pkg.lst").read().split() +print("LST", sorted(left) == [f"{R}/sbin/mine"]) +man = open(f"{R}/state/manifests/{A}.files").read().split() +print("MAN", sorted(man) == [f"{R}/sbin/mine"]) +import shutil; shutil.rmtree(R, ignore_errors=True) +PYDIS +)" + case "$_dw" in + *"SKIP"*) skip "disown (no two system accounts to stand in for packages)" ;; + *) + case "$_dw" in *"DRYRUN-KEPT True"*) ;; *) _p="$_p;a dry run changed ownership" ;; esac + case "$_dw" in *"SU-ROOT True"*) ;; *) _p="$_p;a setuid file setid.conf gives to another package was not returned to root" ;; esac + case "$_dw" in *"SU-MODE True"*) ;; *) _p="$_p;the mode from setid.conf was not put back (su stays broken)" ;; esac + case "$_dw" in *"LINK-BACK True"*) ;; *) _p="$_p;a symlink was not given to the owner of its target" ;; esac + case "$_dw" in *"KEPT-MINE True"*) ;; *) _p="$_p;a file the package really installed was taken away" ;; esac + case "$_dw" in *"LST True"*) ;; *) _p="$_p;pkg.lst still records the paths -- --clean would delete them" ;; esac + case "$_dw" in *"MAN True"*) ;; *) _p="$_p;the build manifest still records them -- the next lease re-arms" ;; esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "disown hands back what a package cannot have installed, and clears the record" + fi ;; + esac +fi + +# ---- --clean: park, reinstall, THEN judge (1.14.311) ----------------------- # +# `update webkitgtk --clean --run` ended green and left +# jsc/jsc.h:29: fatal error: jsc/JSCOptions.h: No such file or directory +# CMake skips a file whose content is unchanged, so the jsc headers kept their +# old mtime and --clean called them orphans. The reapply meant to repair that +# ran `bash ~/install_<account>` -- a file 1.14.16 stopped writing -- with its +# output discarded and its exit status ignored. It had not worked for +# ~300 releases. +_p="" +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +_cl="$(python3 - "$pm_src" <<'PYCL' +import importlib.machinery as m, os, sys, tempfile, time +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +R = tempfile.mkdtemp(prefix="lfsclean.") +pm.TMP_DIR = R + "/pmtmp" +pm.printInfo = pm.printWarning = pm.printError = lambda *a, **k: None +pm.write_pkg_list = lambda *a, **k: None +pm.remove_paths_as_user = lambda u, ps: [os.unlink(p) for p in ps + if os.path.lexists(p) and not os.path.isdir(p)] +user = __import__("pwd").getpwuid(os.getuid()).pw_name +same, orphan = R + "/JSCOptions.h", R + "/old-only.h" +def fixture(): + for p in (same, orphan): + open(p, "w").write("old") + os.utime(p, (1, 1)) +since = time.time() + +# 1. the install step re-creates the unchanged header, not the orphan +fixture() +pm._reapply_install = lambda u: (open(same, "w").write("cur"), True)[1] +pm._clean_after_install(user, [same, orphan], since) +print("KEPT", os.path.exists(same)) +print("ORPHAN-GONE", not os.path.exists(orphan)) +print("NO-LITTER", not any(f.endswith(pm._PARK_SUFFIX) for f in os.listdir(R))) + +# 2. the install step FAILS: everything is put back, nothing deleted +fixture() +pm._reapply_install = lambda u: False +pm._clean_after_install(user, [same, orphan], since) +print("RESTORED", os.path.exists(same) and os.path.exists(orphan) + and open(same).read() == "old") +print("NO-LEDGER", not os.path.exists(pm._park_ledger(user))) +import shutil; shutil.rmtree(R, ignore_errors=True) +PYCL +)" +for _k in KEPT ORPHAN-GONE NO-LITTER RESTORED NO-LEDGER; do + case "$_cl" in *"$_k True"*) ;; *) _p="$_p;--clean: $_k failed" ;; esac +done +_ra="$(python3 - "$pm_src" <<'PYRA' +import sys, re +s = open(sys.argv[1]).read() +f = s[s.index("def _reapply_install("):] +f = f[:f.index("\ndef ", 1)] +print("DEVNULL" if "DEVNULL" in f else "LOGGED") +print("CHECKED" if "return True" in f and "returncode" in f else "UNCHECKED") +print("VERSIONED" if "_versioned_script_in" in f else "STALE-NAME") +PYRA +)" +case "$_ra" in *LOGGED*) ;; *) _p="$_p;the reapply still throws its output away" ;; esac +case "$_ra" in *CHECKED*) ;; *) _p="$_p;the reapply's exit status is still ignored" ;; esac +case "$_ra" in *VERSIONED*) ;; *) _p="$_p;the reapply still runs install_<account>, which no longer exists" ;; esac +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "--clean deletes only what a reinstall does not put back, and undoes itself when the reinstall fails" +fi + +# ---- the shared hub is a system path, not a home (1.14.312) --------------- # +# The links to every shared directory lived in the main user's ~/Shared, so +# nothing but that one login could find them by a fixed path, and every +# login enumerated ITS OWN $HOME/Shared for the SHARED_* variables -- which +# in an app account is its data directory. Now: one hub, and ~/Shared is a +# link to it. The move keeps anything that is not ours. +_p="" +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +if [ "$(id -u)" != 0 ]; then + skip "the shared hub move (needs root for chown)" +else + _hb="$(python3 - "$pm_src" <<'PYHUB' +import importlib.machinery as m, os, sys, tempfile, pwd, shutil +R = tempfile.mkdtemp(prefix="lfshub.", dir="/var/tmp") +open(R + "/conf", "w").write("shared_hub=%s/hub\nmain_user=root\n" % R) +os.environ["PKGUSR_CONFIG"] = R + "/conf" +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +pm.printInfo = pm.printWarning = lambda *a, **k: None +home = R + "/home" +pm._home_of = lambda u: home +print("CONFIGURED", pm.shared_hub() == R + "/hub") +print("ENV", pm._system_env_values().get("PKGUSR_SYS_SHARED") == R + "/hub") + +# an old layout: ~/Shared holding one of our links +os.makedirs(home + "/Shared"); os.makedirs(R + "/u_claude/Shared") +os.symlink(R + "/u_claude/Shared", home + "/Shared/Claude") +pm._ensure_shared_hub("root") +print("MOVED", os.path.islink(R + "/hub/Claude") + and os.readlink(R + "/hub/Claude") == R + "/u_claude/Shared") +print("HOMELINK", os.path.islink(home + "/Shared") + and os.path.realpath(home + "/Shared") == os.path.realpath(R + "/hub")) +print("ROOTOWNED", os.stat(R + "/hub").st_uid == 0) +# twice is the same as once +pm._ensure_shared_hub("root") +print("IDEMPOTENT", sorted(os.listdir(R + "/hub")) == ["Claude"]) + +# a ~/Shared with a file of YOURS in it stays a directory +os.unlink(home + "/Shared"); os.makedirs(home + "/Shared") +open(home + "/Shared/notes.txt", "w").write("mine") +os.symlink("/nowhere-x", home + "/Shared/Other") +pm._ensure_shared_hub("root") +print("KEPT-MINE", os.path.isdir(home + "/Shared") and not os.path.islink(home + "/Shared") + and os.path.exists(home + "/Shared/notes.txt")) +print("MOVED-OURS", os.path.islink(R + "/hub/Other") and not os.path.lexists(home + "/Shared/Other")) +shutil.rmtree(R, ignore_errors=True) +PYHUB +)" + for _k in CONFIGURED ENV MOVED HOMELINK ROOTOWNED IDEMPOTENT KEPT-MINE MOVED-OURS; do + case "$_hb" in *"$_k True"*) ;; *) _p="$_p;shared hub: $_k failed" ;; esac + done + _sv="$(dirname "$LFS_TOOL")/stacks/services.stack" + grep -q '\[ -L "\$h/Shared" \]' "$_sv" \ + || _p="$_p;a machine with the old ~/Shared never re-runs cfg app-users" + # 1.14.315: the DATA lives in /home/Shared, the homes are 0700. 0750 + # let the main user -- a member of every account's group, for the + # shared directory -- read the account's profile and cookies. + grep -q '^SHARED_HUB_DEFAULT = "/home/Shared"$' "$pm_src" \ + || _p="$_p;the shared directories do not default to /home/Shared" + _sau="$(_slice_def "$pm_src" _setup_application_user)" + printf '%s' "$_sau" | grep -q '"chmod", "0700", home' \ + || _p="$_p;an application home is still readable by its group" + printf '%s' "$_sau" | grep -q '"chmod", "0750", home' \ + && _p="$_p;an application home is still made 0750 somewhere" + printf '%s' "$_sau" | grep -q 'ln", "-s", share, _in_home' \ + || _p="$_p;the account's ~/Shared is not linked to its hub directory" + printf '%s' "$_sau" | grep -q 'moved (that is yours to do)' \ + || _p="$_p;a ~/Shared with files in it could be replaced" + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "the shared links live in one system hub; ~/Shared is a link to it, and nothing of yours is moved" + fi +fi + +# ---- needs= speaks the package's name; the book files it by anchor (1.14.317) +# `needs=libjpeg-turbo` on grim: the book's anchor is `libjpeg`, so the +# stack tried to INSTALL an installed package and stopped on "not found". +_p="" +pm_src="$(dirname "$LFS_TOOL")/packagemanager" +_rn="$(python3 - "$pm_src" <<'PYRN' +import importlib.machinery as m, sys +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +pm.printInfo = lambda *a, **k: None +out = (" [package] libjpeg libjpeg-turbo-3.2.0 installed libjpeg-turbo-3.2.0\n" + " [package] openjpeg2 OpenJPEG-2.5.4\n" + " [package] a1 foo-1.0\n [package] a2 foo-2.0\n") +pm.run_blfs = lambda a: (0, out, "") +print("MAP", pm._resolve_need("libjpeg-turbo") == "libjpeg") +print("ANCHOR", pm._resolve_need("libjpeg") == "libjpeg") +print("AMBIG", pm._resolve_need("foo") == "foo") +print("NONE", pm._resolve_need("nothing") == "nothing") +PYRN +)" +for _k in MAP ANCHOR AMBIG NONE; do + case "$_rn" in *"$_k True"*) ;; *) _p="$_p;needs= resolution: $_k failed" ;; esac +done +grep -q '_resolve_need(x.strip())' "$pm_src" \ + || _p="$_p;the stack's needs= loop does not resolve names" +grep -E '^git +grim ' "$(dirname "$LFS_TOOL")/stacks/sway.stack" | grep -q 'libjpeg-turbo' \ + && _p="$_p;sway.stack still names libjpeg-turbo instead of the anchor" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "needs= accepts a package's own name when the book files it under another anchor" +fi + +# ---- the desktop is three stacks, each entry written once (1.14.320) ------ # +# sway.stack had grown to 477 lines answering four questions. Split: +# wayland (the base), media (unchanged), sway (compositor + tools + your +# programs), apps (viewers), tools (CLI). `stack sway` must still be the +# whole desktop in one command, and nothing may be written twice. +_p="" +_sd="$(dirname "$LFS_TOOL")/stacks" +grep -nE '^include +(wayland|media)$' "$_sd/sway.stack" | cut -d: -f2 | tr '\n' ' ' \ + | grep -q '^include wayland include media $' \ + || _p="$_p;sway.stack does not include wayland, then media" +for _f in wayland apps tools; do + [ -f "$_sd/$_f.stack" ] || _p="$_p;no $_f.stack" +done +_dup="$(for _f in wayland media sway apps tools vision diag dev network services; do + grep -hoE '^(book|git|tar|cfg|check) +[^ ]+' "$_sd/$_f.stack" 2>/dev/null + done | awk '{print $2}' | sort | uniq -d | tr '\n' ' ')" +[ -z "$_dup" ] || _p="$_p;written in more than one stack: $_dup" +for _e in "wayland mesa" "wayland wlroots" "wayland gtk4" "sway sway" "sway foot" \ + "sway grim" "apps zathura" "apps imv" "tools jq" "tools translate-shell"; do + set -- $_e + grep -qE "^(book|git|tar) +$2( |\$)" "$_sd/$1.stack" \ + || _p="$_p;$2 is not in $1.stack" +done +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "the desktop is wayland + media + sway in one command, apps and tools beside it, each entry once" +fi + +# ---- "phased" means it ends in the runner (1.14.306) ----------------------- # +# `update <git-pkg>` warned "git script isn't phased" for a perfectly current +# script, because it tested for `unpack_pkg` -- the RARE OVERRIDE that replaces +# the built-in unpack. A correct script does not define it: the runner's own +# unpack is what clones git. The advice that followed (`template --git`) +# rewrites the file and takes any hand-tuned build_pkg with it -- on this +# machine that is imv's `-D c_link_args=-licuuc`. +_p="" +_ok="$(python3 - "$pm_src" <<'PYPH' +import importlib.machinery as m, sys +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +cur = ('#!/bin/bash\n### phases: all\nname="imv"\nis_git="1"\n' + 'build_pkg() { meson setup build; }\n' + '. "${PKGUSR_LIB:-lfs-phases}" && pkgusr_run "$@"\n') +old = ('#!/bin/bash\nname=imv\ngit clone https://x/y.git\n' + 'cd y && meson setup build && ninja -C build install\n') +print("CUR", pm._is_phased_script(cur)) +print("OLD", pm._is_phased_script(old)) +print("NOUNPACK", "unpack_pkg" not in cur) +PYPH +)" +case "$_ok" in *"CUR True"*) ;; *) _p="$_p;a current phased script is not recognised as phased" ;; esac +case "$_ok" in *"OLD False"*) ;; *) _p="$_p;an unphased script passes as phased" ;; esac +case "$_ok" in *"NOUNPACK True"*) ;; *) _p="$_p;the fixture does not reproduce the case" ;; esac +grep -q 'unpack_pkg" not in open' "$pm_src" \ + && _p="$_p;the phased test still asks for the rare unpack_pkg override" +if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done +else + ok "a script is phased when it ends in the runner, not when it overrides unpack" +fi + +# ---- a branch is not a point in time (1.14.305) ---------------------------- # +# `git checkout master` on an ALREADY CLONED tree says "Already on 'master'" +# and stays exactly where it was -- so a package pinned to a branch fetched +# the new commits and rebuilt the old ones, for ever, while reporting that it +# had updated. A tag must do the opposite and never move. +_p="" +_ph="$(dirname "$LFS_TOOL")/lfs-phases" +if [ -f "$_ph" ] && command -v git >/dev/null 2>&1; then + grep -q 'checkout -B "$_pp_ref" "origin/$_pp_ref"' "$_ph" \ + || _p="$_p;a branch ref is not moved to the remote on a rebuild" + grep -q 'checkout --detach' "$_ph" \ + || _p="$_p;a tag or commit is not checked out detached" + grep -q 'symbolic-ref --short refs/remotes/origin/HEAD' "$_ph" \ + || _p="$_p;with no ref at all nothing follows the default branch" + # and the behaviour itself, against a real repository + _gr="$T/gitref"; rm -rf "$_gr"; mkdir -p "$_gr/up" + ( cd "$_gr/up" && git init -q -b master . \ + && git config user.email t@t && git config user.name t \ + && echo one > V && git add -A && git commit -qm one && git tag v1 ) >/dev/null 2>&1 + git clone -q "$_gr/up" "$_gr/clone" >/dev/null 2>&1 + ( cd "$_gr/up" && echo two > V && git add -A && git commit -qm two ) >/dev/null 2>&1 + ( cd "$_gr/clone" && git fetch -q --all --tags --prune ) >/dev/null 2>&1 + _was="$(git -C "$_gr/clone" rev-parse --short HEAD 2>/dev/null)" + git -C "$_gr/clone" checkout -q master >/dev/null 2>&1 + _plain="$(git -C "$_gr/clone" rev-parse --short HEAD 2>/dev/null)" + git -C "$_gr/clone" checkout -q -B master origin/master >/dev/null 2>&1 + _moved="$(git -C "$_gr/clone" rev-parse --short HEAD 2>/dev/null)" + [ "$_plain" = "$_was" ] \ + || _p="$_p;this git does not reproduce the bug (plain checkout moved)" + [ -n "$_moved" ] && [ "$_moved" != "$_was" ] \ + || _p="$_p;checkout -B did not move the branch to the remote" + # the reporting side: branch vs tag, and what is newer + _gs="$(python3 - "$pm_src" <<'PYGS' +import importlib.machinery as m, sys +pm = m.SourceFileLoader("pm", sys.argv[1]).load_module() +print("VSORT", pm._vsort_key("1.9.0") < pm._vsort_key("1.10.0")) +print("VAR", pm._script_var('link="https://x/y"\ngit_ref="2.0.0"\n', "git_ref")) +print("CMD", hasattr(pm, "cmd_git_status"), hasattr(pm, "cmd_git_ref")) +PYGS +)" + case "$_gs" in *"VSORT True"*) ;; *) _p="$_p;1.10.0 does not sort after 1.9.0" ;; esac + case "$_gs" in *"VAR 2.0.0"*) ;; *) _p="$_p;the ref cannot be read out of an install script" ;; esac + case "$_gs" in *"CMD True True"*) ;; *) _p="$_p;there is no way to see or pick a git ref" ;; esac + if [ -n "$_p" ]; then + printf '%s\n' "${_p#;}" | tr ';' '\n' | while IFS= read -r m; do + [ -n "$m" ] && bad "$m" + done + else + ok "a branch ref follows the remote on rebuild; a tag stays put and newer ones are offered" + fi +else + skip "no lfs-phases or no git here: the branch-ref check cannot run" +fi + +#@@ SUMMARY @@ -- everything below runs whichever blocks were selected echo -echo " PASS: $(_pass_total) FAIL: $(_fail_total) (final)" +echo " PASS: $(_pass_total) FAIL: $(_fail_total) SKIP: $(_skip_total) (final)" +[ "$(_skip_total)" -eq 0 ] || echo " (skipped checks need something this machine has not got)" [ "$(_fail_total)" -eq 0 ]